Module 09 — Benchmarking, Cost & Capacity Planning

Goal of this module: Learn to measure serving performance honestly, turn those measurements into cost per token/request, and do capacity planning — how many GPUs you need for a given workload and SLO. This is where engineering meets the budget, and it’s the skill that gets you taken seriously in planning meetings.

Earlier modules gave you techniques and metrics. This one teaches you to produce numbers you can defend.

9.1 Why honest benchmarking is hard (and most benchmarks lie)

Performance numbers are easy to fake — usually by accident. The most common ways LLM benchmarks mislead:

Single-request testing. Sending one request at a time measures idle latency and ignores vLLM’s entire reason for existing (batching, Module 05). It tells you nothing about throughput or behavior under load. Real systems run concurrent traffic.
Ignoring input/output length. A benchmark with 10-token prompts and 10-token outputs behaves completely differently from one with 4,000-token prompts and 500-token outputs (prefill vs decode balance, KV-cache size — Modules 02/03). A number with no length context is meaningless.
Reporting averages, not percentiles. Averages hide the tail (Module 3.4). Always report p50/p95/p99.
Warm-up effects. The first requests pay startup costs (CUDA graph capture, cache cold). Discard warm-up; measure steady state.
Unrealistic arrival patterns. Real traffic arrives randomly/bursty, not in a perfectly uniform stream. Closed-loop (“always exactly N in flight”) vs open-loop (requests arrive at a rate, queue if busy) give different, both-useful pictures.
Apples-to-oranges quality. A quantized or smaller model may be faster but worse. Speed numbers are only comparable at equal quality (Module 04).

The honesty rule: a performance number is only meaningful alongside its conditions — model, precision, hardware, input length, output length, concurrency level, and percentile. Report all of them, every time. A bare “we get 5,000 tokens/sec” is not an engineering claim; it’s a vibe.

9.2 What to measure (the benchmark spec)

A proper benchmark fixes these variables and reports these outputs.

Inputs you must specify:

Model + precision/quantization (e.g. Llama-3.1-8B, FP8).
Hardware (GPU type, count, parallelism layout).
Input length distribution (e.g. ~512 prompt tokens).
Output length distribution (e.g. ~256 generated tokens).
Concurrency / request rate (e.g. 64 concurrent, or 30 req/s open-loop).

Outputs you report (each at p50/p95/p99):

TTFT, TPOT, end-to-end latency.
Throughput (total output tokens/sec, and requests/sec).
Goodput (throughput within your SLO — the number that really matters).

The deliverable is usually a latency-throughput curve: sweep concurrency from low to high and plot throughput (x) against p99 latency (y). This single chart captures the Module 3.3 trade-off and shows your best operating point.

9.3 The latency-throughput curve (your most important chart)

Here’s the shape you’ll produce and how to read it:

 p99
latency
  │                                   ╱  ← past here, latency explodes:
  │                                 ╱      the system is overloaded,
  │                              ╱         queues grow unboundedly
  │                          ╱
  │     SLO ─────────────╳──────────────  (your latency limit)
  │                  ╱   ↑
  │             ╱        the "knee": best throughput
  │        ╱             while still meeting the SLO
  │   ╱                  → THIS is your operating point
  └────────────────────────────────────► throughput (tokens/s)

How to use it:

Below the knee: you have spare capacity; latency is comfortably under SLO. You could push more traffic (more batching) for more throughput “for free.”
At the knee: maximum throughput that still meets your latency SLO. This is your target operating point and the basis for capacity planning (Section 9.5). It’s essentially your goodput maximum.
Past the knee: latency shoots up as queues build; you’re overloaded. Autoscaling (Module 08) should add capacity before you reach here.

Producing this curve for your model/hardware/workload, then operating at the knee, is the core quantitative skill of serving. It replaces guessing with a picture.

9.4 Tools and how to run a benchmark

You don’t have to build benchmarking from scratch:

vLLM’s built-in benchmarking scripts (benchmark_serving, benchmark_throughput, etc.) drive load against a server and report latency/throughput. Great starting point and well-aligned with the metrics above.
Load generators like locust, k6, or community LLM-bench tools for open-loop, rate-based traffic.
Realistic datasets: use prompts/outputs that match your actual workload lengths (e.g. sample from your real traffic, or a public dataset with similar shape). Synthetic fixed-length tests are fine for controlled comparisons but verify against realistic data before trusting numbers for capacity.

A minimal sound procedure:

Fix model, precision, hardware, and input/output lengths.
Warm up; discard the first batch of results.
Sweep concurrency: e.g. 1, 4, 16, 32, 64, 128 concurrent requests (or rising request rates).
At each level, run long enough for steady state; record TTFT/TPOT/latency percentiles and throughput.
Plot the latency-throughput curve; find the knee under your SLO.
Change one variable (a flag from Module 06, or quantization from Module 04), repeat, compare. One change at a time — otherwise you can’t attribute the difference.

9.5 Capacity planning: from workload to GPU count

Now the money question: “How many GPUs do I need?” Here’s a clean method.

Step 1 — Characterize the workload. From Module 3.9: requests/sec (average and peak), average input length, average output length, and your latency SLO.

Step 2 — Find single-replica capacity from your benchmark (Section 9.3): the throughput at the knee for one replica (one GPU or one TP group) at your workload’s input/output lengths while meeting the SLO. Call it C (e.g. requests/sec per replica, or output tokens/sec per replica).

Step 3 — Compute replicas needed:

replicas needed (peak) ≈  ceil( PEAK demand / C )  ×  safety factor

Plan for peak, not average, traffic (or rely on autoscaling to cover the gap — but remember scale-up is slow, Module 8.4, so keep headroom).
A safety factor (e.g. 1.2–1.5×) covers variance, traffic spikes, and the tail. Don’t plan to run at 100% — you’ll violate SLOs the moment traffic wobbles.

Step 4 — Multiply by GPUs per replica. If each replica is a TP group of 4 GPUs (Module 07), total GPUs = replicas × 4.

Step 5 — Sanity-check memory. Confirm each replica’s GPUs hold weights + KV cache for the target concurrency at your context length (Modules 01/02). If not, quantize (Module 04) or add parallelism (Module 07).

Worked sketch:

Peak demand = 120 req/s. Benchmark says one replica handles 30 req/s at p99 within SLO for your length profile. → 120/30 = 4 replicas; ×1.3 safety ≈ 6 replicas. If each replica is 1 GPU → ~6 GPUs at peak. Autoscale down off-peak to save cost.

That five-step method, backed by a real benchmark, is exactly how capacity is planned in practice. You can now do it.

9.6 Cost: turning performance into dollars

The metric leadership cares about is cost per token (or per request, or per user).

cost per 1M output tokens ≈
   (GPU $ per hour)  ÷  (output tokens per hour at your operating point)  × 1,000,000

Where “output tokens per hour at your operating point” comes straight from your benchmark throughput at the knee (Section 9.3). Two implications:

Higher throughput per GPU = lower cost per token. This is why every optimization in Modules 04–07 matters financially: continuous batching, PagedAttention, quantization, etc. each raise tokens/hour per GPU, directly cutting cost per token. You can now put a dollar figure on, say, enabling prefix caching.
Utilization matters enormously. A GPU you’re paying for but running at 20% of its knee throughput costs ~5× per token versus one near the knee. So autoscaling and good batching aren’t just performance features — they’re the main cost levers.

Other cost factors to fold in: idle/over-provisioned capacity, redundancy for reliability, networking/storage, and the build-vs-buy choice (self-hosted GPUs vs managed API endpoints). For low or spiky volume, a managed per-token API is often cheaper and simpler; for high, steady volume, self-hosting on optimized infra (this whole tutorial) usually wins on cost per token — which is exactly why companies invest in serving engineering.

9.7 Comparing options like an engineer

When you evaluate a change (new model, quantization, new flag, different GPU), compare on all relevant axes at once, not just speed:

Axis	Question
Quality	Did output quality hold on my task? (Module 04)
TTFT / TPOT	Did user-facing latency stay within SLO? (Module 03)
Throughput / GPU	More tokens/hour per GPU? (this module)
Cost per token	Cheaper to run? (9.6)
Memory / fit	Does it still fit with headroom? (Modules 01/02)
Complexity / risk	More moving parts, harder to operate? (Modules 07/08)

A change that’s 20% faster but noticeably lower quality, or that adds fragile multi-node complexity, may be a bad trade. Experts evaluate the whole vector, validate quality on real tasks, and pick the option that maximizes goodput per dollar within the SLO at acceptable risk. That phrase — goodput per dollar within SLO — is a fair one-line summary of the entire job.

Check your understanding

List three common ways an LLM benchmark misleads, and the fix for each. (9.1)
What variables must accompany any throughput number for it to be meaningful? (9.1, 9.2)
On a latency-throughput curve, what is the “knee” and why is it your target operating point? (9.3)
Walk through the five-step method to estimate how many GPUs a workload needs. (9.5)
Write the cost-per-token relationship and explain why optimization techniques reduce it. (9.6)
Why is “20% faster” not automatically a good change? (9.7)

Key terms introduced

closed-loop vs open-loop load · warm-up · latency-throughput curve · the knee / operating point · benchmark_serving / benchmark_throughput · load generator (locust/k6) · single-replica capacity · safety factor / headroom · capacity planning · cost per token · utilization-driven cost · build vs buy · goodput per dollar

Next: Module 10 — Mastery: Putting It All Together, where everything converges into a full worked design, a decision framework, and an expert’s checklist.