The story
According to one theory, Lammas Day (1 August) being an important day for settling accounts, latter Lammas signified the last day for settling accounts: in other words, judgement day. Thus, in worldly terms, at latter Lammas means never.
Etymology from Wiktionary, CC BY-SA 4.0.