The story
A specialization of a sense of potato meaning a thing of little value, and a comment on the low value of a small potato in particular, appearing in its modern form in North America in the mid-19th century (see 1855 quotation).
Etymology from Wiktionary, CC BY-SA 4.0.