If you were to go before the courts as a parent who missed payments, access would be considered an entirely different issue. In this day and age, when people are struggling with the cost of living and the fact that he is unemployed, I think that to take away access is cruel.
Let's face it, this is not about not having enough money. If it was, your child being fed and watered by somebody else for the weekend would save you money. If the situation continues, I'd advise you fund an access application for fixed access for your son to see his daughter. It doesn't have to be an expensive exercise. We had to do it for our son and the courts were very understanding.
Of course, if your son is out of work, he can apply to have his maintenance payment reduced whilst he looks for work. Yes, he has responsibility to maintain his child but if you do not have the money, you can't magic it up out of thin air.