I would grin and bear it. Someday, or may be already, someone has felt like that about the other guests or some other reason that meant that they really didn't want to accept an invitation from you. If they avoided it, you may well have been hurt by their absence, or their presence made your day.
We cannot have life all our way all the time, and sometimes it is grinning and bearing it on occasions like this, is that makes us the person other people like to know.
It won'tkill you, so go. Put on a good face and internally think what you like of your fellow guests, you might even think hard and very gently wind them up a bit!