Module 02 — Inside the Transformer: How Inference Actually Works

Goal of this module: Open up the LLM and see the machinery that produces tokens — at a level deep enough to reason about performance, but with no scary math. The crown jewel here is the KV cache: understand it and you understand why LLM serving is hard and why every optimization in later modules exists.

This is the most important conceptual module in the tutorial. Read it twice if needed. Everything in Modules 04–07 is, at heart, a clever response to something explained here.

2.1 Why we need to look inside

In Module 00 you learned the model generates tokens one at a time. In Module 01 you learned decode is memory-bound. Now we connect them: why is decode memory-bound, and what exactly is being stored and moved? The answer lives in the architecture called the Transformer, which nearly every modern LLM is built on.

We will not derive the Transformer mathematically. We will build just the parts that explain serving behavior.

2.2 The Transformer in plain language

A Transformer-based LLM is a tall stack of identical layers (e.g. 32 layers, 80 layers). A token’s representation flows up through the stack, getting refined at each layer, until at the top the model predicts the next token.

Each layer has two main parts:

Attention: the mechanism that lets each token “look at” the other tokens and decide which ones are relevant. This is where the model figures out, for example, that in “The trophy didn’t fit in the suitcase because it was too big,” the word it refers to the trophy.
A feed-forward network (FFN/MLP): a step that processes each token’s information further. This part holds a large share of the parameters and does a lot of the raw “knowledge” computation.

Stacked many times, plus some supporting pieces (normalization, etc.), these produce the model’s remarkable behavior. For serving, attention is the part with the performance-defining twist, so we zoom in there.

2.3 Attention, gently

Here is attention without equations.

For every token, the model creates three things:

A Query (Q): “what am I looking for?”
A Key (K): “what do I offer / what am I about?”
A Value (V): “the actual information I’ll hand over if you attend to me.”

To process a token, the model takes that token’s Query and compares it against the Keys of all the tokens so far. Where the Query matches a Key strongly, the model pulls in more of that token’s Value. So each token builds its new representation as a weighted blend of the Values of the tokens it found relevant.

The crucial structural fact:

To process the current token, the model needs the Keys and Values of every previous token in the sequence.

Read that again, because it is the seed of the entire KV cache.

2.4 The naïve approach is wasteful — enter the KV cache

Recall the decode loop: at each step we have generated text so far and we want the next token. To compute attention for the newest token, we need the Keys and Values of all tokens before it.

The naïve approach would be: at every single step, recompute the K and V for every token from scratch. But that is enormously wasteful — the Keys and Values of the earlier tokens do not change when we add a new token. The trophy’s “Key” is the same whether the sentence is 5 words or 50 words long.

So instead we do the obvious smart thing:

Compute each token’s Key and Value once, then store them. Reuse them for every future step.

This store is the KV cache (Key-Value cache). It holds the K and V vectors for every token processed so far. With it, generating a new token requires computing K and V for only the one new token, then reading the cached K and V of all previous tokens out of memory.

This is a massive speedup — without it, generation would get quadratically slower as text grows. The KV cache is not optional; it is what makes autoregressive generation practical.

But the KV cache is also the source of most serving pain

The KV cache buys speed at the cost of memory. And it has three nasty properties:

It grows with every token. Longer conversations / longer outputs → bigger cache. It grows during a request.
There is one per request. Every concurrent user has their own KV cache. 100 users = 100 caches sitting in GPU memory at once.
It must be read every single decode step. This is why decode is memory-bound (Module 01). Each new token requires reading the whole KV cache (plus the weights) out of memory while doing little compute.

So the KV cache simultaneously (a) makes generation fast, (b) eats GPU memory, and (c) defines the memory-bound nature of decode. Three of the biggest themes in serving, all from one data structure. This is why we said: understand the KV cache and you understand serving.

2.5 How big is the KV cache? (the formula worth knowing)

You can estimate KV cache size, and you should, because it competes with the weights for VRAM. The size depends on:

KV cache size ≈
   (number of tokens)
 × (number of layers)
 × (hidden size related to KV heads)
 × 2            ← one for K, one for V
 × (bytes per number, e.g. 2 for FP16)

You do not need to compute this by hand often, but the shape of the formula teaches the levers:

More tokens → linearly more cache. This is why long contexts and long outputs are memory-expensive. A 100K-token conversation can have a KV cache as large as, or larger than, the model’s weights.
More layers / bigger model → more cache per token.
Lower precision for the cache → smaller cache. This is exactly KV-cache quantization (Module 04) — store K and V in 8-bit instead of 16-bit and halve this whole budget.
Architectural tricks that share Keys/Values across attention heads (called MQA — Multi-Query Attention, and GQA — Grouped-Query Attention) shrink the KV cache dramatically. Most modern efficient models use GQA specifically to make the KV cache cheaper. Now you know why that design choice exists.

Concrete intuition: For a typical 13B model, the KV cache can cost on the rough order of hundreds of kilobytes to ~1 MB per token depending on architecture and precision. Multiply by thousands of tokens and dozens of simultaneous users and you are quickly talking about many gigabytes — competing directly with the weights for space. Managing this memory well is the central job of a serving engine like vLLM, as we will see.

2.6 Re-examining prefill vs decode, now with the KV cache

We can now give the crisp, hardware-grounded version of the two phases from Module 00.

Prefill (processing the prompt)

All input tokens are known up front, so the model computes Q, K, V for all of them at once, in parallel.
It fills the KV cache with the K and V for every prompt token.
Because it processes many tokens simultaneously, it does a lot of math relative to memory moved → compute-bound, and it uses the GPU’s parallel power well.
Cost grows with prompt length (and grows faster than linearly for the attention part). Long prompts make prefill — and therefore TTFT — slower.

Decode (generating the answer)

One new token at a time. For each, compute Q/K/V for just that one token, append its K/V to the cache, then attend over the entire cache.
Per step, the GPU must read all the weights and the whole KV cache but does little arithmetic → memory-bound, and it underuses the GPU’s parallel compute.
Cost per token is roughly constant-ish (slowly rising as the cache grows). Each step contributes to TPOT.

PREFILL                         DECODE
─────────────────────           ─────────────────────
all prompt tokens at once       one new token per step
parallel, GPU busy              sequential, GPU often idle
compute-bound                   memory-bound
fills the KV cache              reads + grows the KV cache
sets TTFT                       sets TPOT

Almost every optimization technique you will learn targets one column of that table. When you meet a new trick, ask yourself: “Is this attacking prefill or decode? Is it saving compute or memory?” That question alone will let you place any technique correctly.

2.7 Why serving many users at once is the whole game

Here is the economic heart of the field, now that you have the pieces.

A single user generating tokens leaves the GPU’s compute mostly idle (decode is memory-bound — the cooks are waiting). That is wasteful: you are paying for an expensive GPU that is barely calculating.

But notice: when the GPU reads the model weights from memory to generate one user’s token, those same weights could be used to generate a token for many other users at the same time, for almost no extra memory traffic. The weights are already loaded; use them for everyone.

This is batching — processing many requests together. It converts wasted, idle compute into useful work, dramatically increasing throughput (tokens served per second across all users) without proportionally hurting each user’s latency. Batching is the single highest-leverage idea in LLM serving, and it works because decode is memory-bound.

But batching has a catch: every batched request needs its KV cache resident in GPU memory simultaneously. So the number of users you can batch is limited by how much KV-cache memory you have and how efficiently you use it. And that is precisely the problem vLLM’s signature invention — PagedAttention — was built to solve (Module 05/06).

Follow the chain you just built:

decode is memory-bound (Mod 01)
        │
        ▼
so batching many users is nearly "free" compute → huge throughput win
        │
        ▼
but each user needs a KV cache in memory (Mod 02)
        │
        ▼
so KV-cache memory efficiency limits how many you can batch
        │
        ▼
so a smart KV-memory manager (PagedAttention) unlocks more batching
        │
        ▼
        → that is why vLLM exists (Mod 06)

You have just reasoned your way from raw hardware to the design of the most popular serving engine in the world. That is what understanding the KV cache gives you.

2.8 A few architectural terms you’ll meet (so they’re not scary later)

Attention heads: attention is done in several parallel “heads,” each looking for different kinds of relationships. More heads = richer attention but more KV cache.
MHA / MQA / GQA: Multi-Head Attention (the original, biggest KV cache), Multi-Query Attention (all heads share one set of K/V, smallest cache, slight quality cost), Grouped-Query Attention (a middle ground — groups of heads share K/V; the modern default). All three are just different trade-offs of KV-cache size vs quality.
Hidden size / embedding dimension: how wide each token’s internal representation is. Bigger = more capacity, more memory, more compute.
Context length / context window: the maximum number of tokens (prompt + output) the model can handle at once. Bigger context = potentially much bigger KV cache.
Mixture of Experts (MoE): an architecture where each token only uses a fraction of the model’s parameters (a few “experts”) per step, even though the model has many. This makes a huge-parameter model cheaper to run per token — important for the largest modern models, and a wrinkle for memory planning (all experts must still be stored, even if not all are used each step).

You will see these in model cards and config files. Now they read as design trade-offs, not mystery words.

2.9 Putting the lifecycle back together (with internals)

Returning to the request lifecycle from Module 00, you can now annotate it like an expert:

Tokenize the prompt.
Prefill: run all prompt tokens through every layer in parallel; build the KV cache; emit the first token. (Compute-bound; sets TTFT.)
Decode loop: for each step, run one token through every layer, read+extend the KV cache, sample the next token, stream it out. (Memory-bound; sets TPOT; the KV cache grows each step.)
Stop at the end-of-sequence token or a length limit; free that request’s KV cache so its memory can serve someone else.

Notice step 4’s quiet importance: when a request finishes, its KV cache memory must be freed and reused for the next request. Doing this efficiently — without fragmentation or waste — is, again, exactly what PagedAttention manages. Every theme keeps returning to KV memory. That is not an accident; it is the nature of the problem.

Check your understanding

What two things (Q vs K vs V) does the model compare to compute attention, and what is the third used for? (2.3)
What does the KV cache store, and what expensive work does it let us avoid at each decode step? (2.4)
Give two reasons the KV cache is a serving headache, despite the speedup it provides. (2.4)
Name two ways to make the KV cache smaller. (2.5)
Explain, using the KV cache, why decode is memory-bound and prefill is compute-bound. (2.6)
Why does batching many users together work so well, and what memory resource limits how far you can push it? (2.7)

Key terms introduced

Transformer · layer · attention · feed-forward network / MLP · Query / Key / Value (Q/K/V) · KV cache · attention head · MHA / MQA / GQA · hidden size · context length · Mixture of Experts (MoE) · batching · throughput

Next: Module 03 — Model Serving Fundamentals, where we turn all this into the metrics and goals that define a serving system: latency, throughput, TTFT, TPOT, and SLOs.