The story
Anglicization of caravanserai, influenced by -ary, -ery (suffix denoting ‘place of’), as if meaning “place of caravans” by reanalysis as caravan + -s- (interfix) + -ary, -ery. Cognates * German Karawanserei
Etymology from Wiktionary, CC BY-SA 4.0.