The story
From an early 20th century practice of reserving short trousers only for small children.
Etymology from Wiktionary, CC BY-SA 4.0.
in short trousers
Today it means: In childhood; as a child; when quite young (including nonliterally, e.g., as a young adult). Full definition →