Ultimately, its up to you whether you choose to go with your husband or stay behind!
I can understand your concerns, but I can see this turning nasty, with you on one side and your husband and son on the other! Even when lockdown ends (if ever) there are bound to be lasting resentful feelings on the part of your husband and son, and in the long term this could seriously affect your relationship with not only your husband, but also your son, his wife and even your grandchildren! Is that something you are prepared for?
Also, your husband and son are not small children, who can be controlled or dictated to! It is entirely their choice as to whether they are prepared to break the lockdown "Stay Home" restrictions, risk your husband being stopped by the police and being fined. A journey of one and a half hours away isn't really very far, and unless your husband is blatantly flouting lockdown restrictions, then the chances of him getting stopped and fined are fairly remote (although conceivably it could happen)!
Lockdown restrictions mean different things to different people, and maybe, just maybe, at 78 years old, your husband is utterly fed-up of government forced incarceration (as many elderly people are) and he actually wants to see his son and grandchildren while he is still physically able!
None of us know what is around the corner! So if he is prepared to take his chances, let him, and if you don't want to accompany him, then you stay home! Whatever choice you make, you have to live with the consequences of it, but please do not allow this to degenerate into a major family argument, which it has the real potential to do!
I know that most people here are on your side, and what I am saying probably is not what you want to hear! However, in early 2016, I had a major disagreement with my daughter, who lives over 50 miles away. My husband took her side and made excuses for her appalling behaviour! This has resulted in the subsequent deterioration of my relationship with my husband of 37 years to such an extent that I am now considering leaving him! Neither he or my daughter are ever going to admit they were wrong, nor are they going to apologise for the terrible things that they have said to me in anger! This has seriously damaged my mental health! In the end, you have to do what you feel is best for you - being a spoilsport or being unreasonable really doesn't come into it! Good luck!