The story
Created by analogy with similar expressions such a glint in your father's eye, but implying the child was conceived adulterously; milkmen, visiting housewives while their husbands would have been at work, are often associated with adultery.
Etymology from Wiktionary, CC BY-SA 4.0.