The story
This saying has been attributed to originating in boxing as a minor blow taking no skin off. See: The American Heritage Dictionary Of Idioms, Second Edition Paperback – February 26, 2013 by Christine Ammer and Random House Dictionary of America's Popular Proverbs and Sayings: Second Edition Paperback – April 18, 2000 by Gregory Titelman.
Etymology from Wiktionary, CC BY-SA 4.0.