The story
Variant of earlier (18th century) Lord love you, God love us, etc (present subjunctive), used as exclamation of exasperation or surprise. The significance of the duck is unknown. Attested from the late 19th century.
Etymology from Wiktionary, CC BY-SA 4.0.