None of the faff is necessary. It has all gradually built up around the day until it seems that it is essential. It is not.
If you are a churchgoer (even just an occasional one) then a carol service is a lovely way to spend an hour and remember the joy that any birth brings to a family, and if it is held at midnight and the bells ring out the good news, it can bring a tear to the driest eye.
If you are not, it can be a day for a family meal. No need for special food, or decorations, or party games, just a chance to get together with family members you may not see very often.
If you don't even want to do that, then you don't have to, There is no obligation. Nothing is wrong with you, it is just not your thing.
Going away is a good move but you could just stay home and do whatever you would do on an ordinary day - why not?