The story
From the mostly archaic sense of office as a "duty" or "function" and hence unmentionable "bodily functions".
Etymology from Wiktionary, CC BY-SA 4.0.
house of office
Today it means: An outhouse: an outbuilding used as a lavatory. Full definition →