Well, it's dh's home, too, and certainly not the op's brother's. It's hard to have to choose between one's db and one's dh, but I think where the first loyalty lays is obvious. If dh were just being controlling or trying to keep her brother away, for some reason, that would be different. But his health is poor, he tires easily, etc. So, imo, his concerns are real.
Besides, what is this with people inviting themselves to stay in someone else' home? How about waiting for an invitation?
But, of course, you can't say that to db, glen. So I'm with those who say to let him know that you'd love to see him and his family when they are there, but can't put them up. If they offer to help, just decline and let them know that "won't work." Either they'll find accommodations elsewhere or they'll stay home. The latter may be best, at this time, though I know you'll miss them.
Luckgirl - I also hope your unwelcome relative "takes the 'ump" and stays away.