LLM Serving and Inference Cost: What Actually Determines Latency and Price

1. A demo that works for one person can fail for a thousand

Our library assistant, now answering grounded questions with retrieval and completing tasks as an agent, works beautifully in a demo with one librarian typing one question at a time. Then the library network rolls it out to fifty branches. Answers that took two seconds now take twenty. The monthly bill arrives, and it is far larger than anyone expected.

Nothing about the assistant's reasoning changed. What changed is serving: how the requests are executed on real hardware, how many run at once, and how that work is measured and paid for. A Solutions Architect needs a working model of this layer, because decisions made elsewhere — context length, tool-call count, model choice — turn directly into latency and cost here.

All figures in this guide are invented teaching examples unless a source is linked; real numbers depend heavily on hardware, provider, and model version.

2. The idea in simple words

Think of a busy restaurant kitchen. Latency is how long one diner waits for their plate. Throughput is how many plates the kitchen produces per hour across all diners. A kitchen can be fast for one order and still collapse under fifty simultaneous ones if it has no way to work on several at once.

LLM serving has the same tension. A single request's latency and the system's overall throughput are both real constraints, and the techniques that improve one can affect the other.

3. Two phases with different costs

A request is not one uniform block of work. It has a prefill phase, which processes the whole input at once and is heavily parallelizable, and a decode phase, which produces output tokens one at a time because each token depends on every token before it — this sequential dependency is why decode cannot simply be sped up by adding more parallel compute the way prefill can.

flowchart TD
    P@{ icon: "algomi:svc-spark", label: "Prompt arrives", pos: "b", h: 52 }
    PF@{ icon: "algomi:async-gear", label: "Prefill: process full input in parallel", pos: "b", h: 52 }
    P --> PF
    KV@{ icon: "algomi:data-bolt", label: "KV cache holds intermediate values", pos: "b", h: 52 }
    PF --> KV
    DC@{ icon: "algomi:async-gear", label: "Decode: generate one token at a time", pos: "b", h: 52 }
    KV --> DC
    DC -->|Each new token reads the cache| KV
    OUT@{ icon: "algomi:svc-cube", label: "Completed response", pos: "b", h: 52 }
    DC --> OUT

Reading the diagram. Read it as one request's life in the kitchen. Prefill is reading the whole order at once, fast because every line can be read in parallel. Its result is kept in the KV cache, the cook's notes. Decode then plates the answer one token at a time, and the loop arrow shows why it is slow: every new token goes back to read the notes before the next one can start. The notes grow with every token, which is why long answers cost memory, not just time.

The KV cache exists so decode does not have to reprocess the entire growing sequence for every new token; it stores the reusable intermediate values from all prior tokens. That cache grows with context length and must live in GPU memory for as long as the request is active, which is why long conversations, large retrieved chunk sets, and long agent traces all raise the cost and memory pressure of serving them, not just the token count billed.

The two phases as movement: