SunshineDad21, if it you and your wife think it is unreasonable, you are in full agreement with each other and are ready to take on the world together, you are not being unreasonable. As important as family is, it is your job as parents to protect your family now and if the wider family don't like it, that is their problem.
That said, try to be a little empathetic for their disappointment. I'd be inclined to be a bit like the Royal Family and put out a statement which says how disappointed you are that they cannot visit in a way that keeps your precious child safe within the confines of a World Pandemic. Thank them for their understanding that they wish to keep their Grandchild safe and that that he/she is looking forward to seeing them when the rules are relaxed. I'd probably also say that I was going to maintain a period of radio silence in order to concentrate on your new family.
The way you handle this at this point is so important. Your family have to realise that they do not call the shots and that this is your family and your rules. They have a choice which means they can play the game appropriately or withdraw. They way you handle your family is also important in the context of your marriage too. Your wife needs to know that you have her back because this sort of thing colours perception when the mistakes add up further down the line. Hormones, tiredness and gilt being well and truly being chipped of the gingerbread make everything feel 10 times worse.
Congratulations to you both on the birth of your baby.