The story
From the game of cricket, where a day's play is terminated by the umpire drawing the stumps out of the ground.
Etymology from Wiktionary, CC BY-SA 4.0.
draw stumps
Today it means: To declare an end to the day's play, and remove the bails and sometimes the stumps. Full definition →