The story
The word is associated with sitting under a lime tree, or having nothing more demanding to do than squeezing limes. It is also thought to originate from "limey", a slang term meaning a British serviceman in Trinidad and Tobago during World War II (noted for hanging around bars and drinking). The term has since been adopted by other countries in the English-speaking Caribbean.
Etymology from Wiktionary, CC BY-SA 4.0.