Not an easy situation. But remember this is the man you chose to spend the rest of your life with after 'being alone for a long period'. His attitude to the grandchildren and the disruption they cause is not a fault that needs correction. There's doubtless an element in truth in his 'do no wrong' comment, but being an indulgent grandmother is not a fault either. Yes, you can visit them on your own. Perhaps he could drive you over, stay for an hour, hand out the half-crowns (£5 notes now, I suppose) then amiably say "Right, I'm off to be a grumpy old man now! Have a good time! See you in a few days2. Isn't that a good idea? And of COURSE it would still be 'a marriage'.
whats the oldest you can hire a car?
Do you get emotionally attached to the plants in your garden ?
Only 50% of middle age adult manage more than 1 brisk 10 min walk a month.

