The story
From the British currency (one penny being one hundredth of a pound or, prior to decimalisation, one 240th of a pound).
Etymology from Wiktionary, CC BY-SA 4.0.
penny wise and pound foolish
Today it means: Prudent and thrifty with small amounts of money, but wasteful with large amounts. Full definition →