The story
Perhaps from baseball or other games played with a ball, alluding to unpredictable bounces; but compare also bounce off the walls.
Etymology from Wiktionary, CC BY-SA 4.0.
off-the-wall
Today it means: Wildly unconventional; bizarre; absurd. Full definition →