I would depending on the company, either say "I need to wee", or, "I'm going to the toilet".
In very formal company, I would just say, "Excuse me a minute, please." and leave the room.
At school we either said, "May I leave the room, please?" or "May I be excused, please?" so I might even use some version of one of these.
I think it is only in England that "lavatory" is considered acceptable. It wasn't in my childhood in Scotland, where the polite word was "toilet", although you could say, as most Americans do that you needed the bathroom.
I friend of mine who grew up in Pakistan, says she is going to the wash-room.