The story
From the idea of another referent included in "we," besides the speaker, being invisible.
Etymology from Wiktionary, CC BY-SA 4.0.
you got a mouse in your pocket
Today it means: Used as a response to statements starting with "we," usually indicating the speaker does not want to be involved in the interlocutor's statements. Full definition →