AI Glossary
Latency
What is Latency?
The time delay between sending a prompt to an AI model and receiving the first token of its response. Critical for real-time applications. Measured as time-to-first-token (TTFT) or total generation time. Larger models tend to have higher latency.
Example in practice
A product team building an AI live chat feature would reject a model with 8-second time-to-first-token latency in favour of a smaller, faster model — users on chat expect near-instant responses.
See also