The story
From the “point thumb over shoulder” motion prescribed in many sports as the referee signal to indicate that a player or coach has been ejected.
Etymology from Wiktionary, CC BY-SA 4.0.
get the thumb
Today it means: To be ejected from a game or match. Full definition →