The story
From farm + hand. Regarding the baseball sense, farm team § Etymology explains further, and calling players farm hands also (by the same token) played into the era's notion of slighting someone by dismissing them as a hayseed (country bumpkin).
Etymology from Wiktionary, CC BY-SA 4.0.