The Sequence Opinion - Issue 914: From Prompt to Token: How AI Inference Really Works
A field guide to prefill, decode, KV caches, and the systems that turn model weights into a responsive product.
TL;DR
- Inference is a complex system, not just a single forward pass.
- Inference systems manage asynchronous traffic, diverse prompt lengths, and user demands for speed and cost-efficiency.
- A modern inference system functions like a miniature operating system around a token factory.
- Key functions include context assembly, tokenization, request routing, GPU scheduling, memory management, kernel execution, output sampling, and text streaming.
- Serving thousands of users simultaneously requires managing different stages of requests.