I was in a sort of similar position a few years ago. When asked to do the Nursery run for my sons little boy, I had to say ‘No’, because I was already committed to school-runs for my older 2 grandchildren plus I was working part-time. I think my son and d.I.l.were quite miffed at first, but gradually understood my position. I assured them that I would help with babysitting, if the wanted to go out etc. And I still do. I am glad I made that decision . I would have absolutely no time for myself.
So, you should put your foot down. You and your husband want time for yourselves.
And you have done enough for them, until now. Your son and d.I.l. will eventually understand.