The story
From hard + wood. The term refers to the seed coat of the tree's seeds; it does not refer to the degree of hardness of the tree's wood, but it seems to do so and is usually assumed to do so by speakers not well informed about trees and timber.
Etymology from Wiktionary, CC BY-SA 4.0.