The story
Derived from bugger (idiomatic, obsolete, UK, Ireland, Commonwealth) “sodomite”, (idiomatic, vulgar, UK, Ireland, Commonwealth) “man”, especially in the colloquialism (idiomatic, dismissal, UK, Ireland, Commonwealth) old bugger for “an old man”.
Etymology from Wiktionary, CC BY-SA 4.0.