I suspect women might worry more, as biologically intact heterosexual males in spaces where women are undressed are able to do far more harm to them than (most) women can, or are inclined to do to (most) men. As, of course, are gay or bisexual men, but most male on female sex crimes are committed by straight ones.
Also, as has been said, much depends on what you mean by 'transwomen' and transmen'. If it's someone who says they are the opposite sex, then anyone qualifies. Including men who have no desire to 'live as' women (whatever that means) but just want an excuse to be in spaces where they can make women feel uncomfortable and show us who is in charge, or to find vulnerable victims.
Many men might feel embarrassed about women seeing them pee in a urinal, but that bears no comparison, as far as I can see.