The story
Originally a British shortening of "come off the grass!", an older (originally American) phrase. Come off the grass!, which is roughly indicative of the speaker's disbelief, or that the speaker believes that the one being spoken to needs to face reality, is ultimately a play on the oft-seen phrase on signs in places such as public parks: "keep off the grass".
Etymology from Wiktionary, CC BY-SA 4.0.