The story
From a likening of the mechanics of anal sex, which may involve pushing "dirt" (i.e., fecal matter) back into the rectum, by analogy to a gardener moving soil uphill.
Etymology from Wiktionary, CC BY-SA 4.0.
uphill gardener
Today it means: A homosexual man. Full definition →