I myself would never refuse to look after a sick grandchild, but then I am not at all concerned about catching things, and both my DH and I are healthy, touch wood.
However, if you are seriously worried about catching illnesses, or have a condition that makes even a cold potentially more dangerous for you than for me, then I think you should explain kindly when the grandchildren are all well to their parents, that looking after sick kiddies is not, and never has been your thing, or that you because of your own state of health just cannot help out.
Your children won't be happy about it, as getting sick children looked after is well-nigh impossible, and employers are not amenable to parents taking time off work because the kindergarten or creche, sensibly refuses to take in sick children, but that cannot be helped, if your really cannot help them out.