AI Glossary
Throughput
What is Throughput?
The volume of requests or tokens an AI system can process per unit of time. Throughput is a key operational metric for production AI deployments alongside latency. High-throughput AI infrastructure requires careful engineering of batching, caching, and model serving — a concern primarily for engineering teams building AI-powered products.
Example in practice
An engineering team that discovers their AI document processing pipeline takes 14 hours to handle 10,000 documents overnight would investigate throughput optimisations — batching requests, adding serving replicas, or switching to a faster model.
See also