The story
From vest + pocket, referring to the pockets of a suit vest, which are only sized to hold watches, handkerchiefs, wallets, and similarly small items.
Etymology from Wiktionary, CC BY-SA 4.0.
vest-pocket
Today it means: Small enough to be carried in a vest pocket or other small container. Full definition →