DaiseyMae, it sounds to me as if you have reached the point where giving it up would be the right thing for you.
How does your husband feel about it?
If he too honestly prefers being at home, sleeping in his own bed etc. then yes, the time has come to give it up.
Perhaps if you don't have the strain and stress of a holiday away from home, you and he can go out for a day occasionally.
If on the other hand, he still wants to go away, for a change or for your sake, the matter is more complicated.
In either case I think you need to tell him honestly that it is getting too much for you, and see what you and he can work out together.
Right now, DH is talking gleefully of a holiday in Budapest or Prague next year, but unless he is far fitter and stronger then than he is now (unlikely) I don't see it happening, but am biding my time not wanting to put a wet blanket on his dreams. So I sympathise with you both.