The story
One possible etymology comes from the sport of boxing, where competitors fistfight wearing padded gloves; removing the gloves during a fight could result in serious physical injury. Another comes from the trend of gentlemen wearing gloves as part of their outfit; these gloves would often be taken off before engaging in rough behavior. Compare throw down the gauntlet.
Etymology from Wiktionary, CC BY-SA 4.0.