Skip to content

tokenizer

Today it means: A system that parses an input stream into its component tokens. Full definition →

Ask AI about its historyChatGPT Claude Perplexity Google AI Grok

The story

From tokenize + -er.

Etymology from Wiktionary, CC BY-SA 4.0.