Providing childcare

As previous posters have said he’s controlling you both.

Say when you are available and leave it at that.

Your dd needs a solicitor and proper contact agreements.

This man needs to be brought to heel.

What mcem (and others) said - he can't be allowed to continue to call the tune in this way. Contact should be about the children, about meeting their needs. Children need routines, they benefit from knowing which day they'll see daddy. Only knowing the day before or minutes before leaves them in limbo.
Is this a pattern that was set in their relationship, that is he makes the decisions about what happens and when it happens. If so, it may help if they get some mediation to help them discuss his contact to the children in a neutral environment.
It's also totally unfair to expect you to dance to his tune. I'd suggest as mcem does, that you set a day and a time and stick to it. It can only be re-negotiated when its impossible to avoid.

You (and DD) must not let him exert this amount of control or it will extend into other areas and will get worse.

Your time is being given so it has to be on your terms.
He wouldn't get away if they were employing a professional childminder.

Suggest that Sunday afternoon is diary time and offer hours that suit all 3 of you and times when you are not available.
With some discussion that becomes the plan for the week and is only to be changed in an emergency!

It's difficult Bell1 but you have to be firm as Farmor15 said. Tell the parents what times you are available and that outside of those times you have other commitments.
You need to have a life too.

I agree with M0nica I would also start to keep a diary as there will come a time when contact is arranged and he will not turn up, or has agreed to have them back by a certain time and does not.

The father is playing power games with you all

Your daughter needs to talk to a solicitor about getting a formal agreed contact plan. This kind of behaviour is the classic behaviour of a man trying to exercise power over his wife and family after they separated.

Did the marriage break-up because he was controlling, either overtly, or covertly by being unreliable.

Amicable break-ups are fine as long as both play by the rules, but when one half of the amicable split starts acting like this, regrettably, solicitors need to be involved.

No experience of this, but thought I'd reply to move your post up. Based on other threads here, the only way you can deal with the situation is to be only available for the hours you agree with daughter and ex, and be firm about it.