The story
Originally in reference to any chamber or suite reserved for the use of women, whether public or private. Popularized by its use as a clipped form of ladies' waiting room in reference to gender-separated facilities at train stations, etc. Its use in reference to lavatories replaced other euphemisms including cloak room, lavatory room, toilet room, private room, etc.
Etymology from Wiktionary, CC BY-SA 4.0.