AI Glossary
Tokenisation
What is Tokenisation?
The process of breaking text into smaller units (tokens) that an LLM can process. Tokens can be words, subword units, or individual characters depending on the tokeniser. Most English words map to 1–2 tokens; context window limits are measured in tokens, not words — typically around 0.75 words per token.
Example in practice
A product manager calculating API costs for a summarization feature would tokenise their typical input document first — discovering a 5,000-word report maps to roughly 6,700 tokens, directly affecting per-request pricing.
See also
Learn more
See Tokenisation applied in a professional context through this free course.
AI Fundamentals for Professionals →