LLM Serving Infrastructure and Capacity Planning: SLOs, Parallelism, and Autoscaling

Staffing the kitchen before exam season

Tanvi runs the platform team at VidyaAI, an AI tutor for school students. Board exams start in eight weeks, and traffic will jump tenfold. VidyaAI hosts its own 70-billion-parameter model, and Tanvi has to tell finance how many GPUs to rent. (Tanvi, VidyaAI and every number here are made up for teaching.)

LLM serving and inference cost explains the mechanics: prefill versus decode, the KV cache, continuous batching, quantisation and cost per token. That is how a cook prepares a dish. This guide is about running the kitchen: what waiting time to promise customers, how many cooks and stoves to staff, how to split a dish too big for one stove, and what to do at the dinner rush.

Tanvi's eight weeks, as questions:

1. "Is it even working?": latency targets that mean something

Answers stream out token by token, so one latency number isn't enough.

sequenceDiagram
    participant S as Student
    participant Q as Queue
    participant G as GPU
    S->>Q: question sent (t = 0)
    Q->>G: admitted after waiting
    G->>G: prefill: read the whole prompt
    G-->>S: first token (TTFT)
    G-->>S: token 2 (TPOT later)
    G-->>S: token 3 (TPOT later)
    G-->>S: ... last token (end-to-end latency)

Reading the diagram. It's one request's timeline, top to bottom.

• Before the first token, everything is waiting: queue time plus prefill, which means reading the whole prompt. That total is the TTFT. A long prompt or a long queue makes it worse, and it's what students feel as "is it working?"
• After the first token, each new token arrives one TPOT later. That is what makes streaming feel smooth or jerky.