The story
First attested in 20th century American English. One possible origin for the phrase is from an Ancient Greek practice of using different-coloured beans for voting, where spilling the jar containing them would unwantedly reveal the results, though this might be folk etymology.
Etymology from Wiktionary, CC BY-SA 4.0.