The story
The origin of the term is in the wool and cotton mills of Northern England where the noise of the machinery and build-up of cotton dust made it difficult to hear people speaking. The earliest example in print is "I wish you'd listen. Have you got cloth ears?" from Carnival, a novel by Compton Mackenzie (1912).
Etymology from Wiktionary, CC BY-SA 4.0.