The story
In the Middle Ages, gingerbread cakes were decorated with a thin layer of gold leaf. Without this casing, they were considered to be humble offerings, generally being little more than flavoured but stale bread.
Etymology from Wiktionary, CC BY-SA 4.0.