The story
After the golden-yellow skin tone of some multiracial people. "High" is usually considered a reference to a social class system in which skin color (and associated ancestries) is a major factor, placing those of lighter skin (with more white ancestry) at the top and those of darker skin at the bottom.
Etymology from Wiktionary, CC BY-SA 4.0.