The story
From pencil + out (possibly based on or influenced by work out), the idea being that rough calculations are temporarily written in pencil rather than permanently written in pen.
Etymology from Wiktionary, CC BY-SA 4.0.
pencil out
Today it means: Of an investment, to make sense financially or to be expected to generate the desired returns. Full definition →