Grandparents looking after a toddler one day a week is not comparable to professional childcare, If you send your child to Day care, no one there takes them out without your permission and a whole sheaf of paperwork.
There is the world of difference between a grandparent caring for a child in a known safe and contained environment (ie, his own home) and taking him out by car or on foot.
My PIL would be quite capable of looking after my son at home, mine or theirs (though there is a lot more to break at theirs). But I would not trust their driving, and they are not active and quick enough on their feet to manage a young child outside. They can't catch him. They can't stop him running off. It doesn't matter how experienced you are, or how many children you have raised. If you are getting older and slower, you may no longer be able to care for small children safely. That is sad, but it is reality. It's the job of the parents to put the child's safety first.
It can be quite difficult to tell your parent that they are no longer physically capable of taking a child out safely, but that might well be the issue.
OP has basically two choices. Follow their rules, or not get alone time with her grandson. That's how it is.