The story
When riding at maximum speed, a racer often perches on the front tip of the saddle, where the shell of an old-style leather saddle would be attached to the saddle frame with a rivet.
Etymology from Wiktionary, CC BY-SA 4.0.
on the rivet
Today it means: Riding at maximum speed. Full definition →