I think it’s because connection and acceptance regardless of status is coded as feminine. So most men don’t want to be the feminine one in the men’s room.

The ladies room is everything the author said and more. That’s why we tend to go there together. lol.