The story
From the Wikipedia article for Teman (Edom): Teman (Hebrew: תימן), was the name of an Edomite clan and of its eponym, according to the Bible, and an ancient biblical town of Arabia Petraea. The term is also traditionally used in Biblical Hebrew as the synonym of the direction South and was applied to being used as the Hebrew name of Yemen (whose Arabic name is "Yaman") due to its location in the Southern end of the Arabian Peninsula, thus making Yemenite Jews being called "Temanim" in Hebrew.
Etymology from Wiktionary, CC BY-SA 4.0.