Applied AI / LLM Infrastructure / Module 06 · vLLM Deep Dive
Module 06

vLLM Deep Dive

Building Systems LLM Infrastructure

Module 06 — vLLM Deep Dive

Goal of this module: Make everything concrete in vLLM, the most widely used open-source LLM inference engine. You’ll learn what vLLM is and why it exists, its architecture, how to install and run it both offline (as a Python library) and online (as an API server), the configuration flags that matter most and what they do (mapped to the concepts from earlier modules), and how to operate it sanely. This is the hands-on heart of the tutorial.

Everything here builds on Modules 01–05. When a flag appears, we connect it back to the concept it controls, so you’re configuring with understanding, not copy-pasting.

A note on versions: vLLM moves fast. Exact flag names, defaults, and features change between releases. The concepts below are stable; for the precise current syntax of any command, check vllm --help, vllm serve --help, and the official docs at https://docs.vllm.ai for the version you install. Treat the commands here as accurate-in-spirit templates, and verify specifics against your installed version.

6.1 What vLLM is and why it exists

vLLM is an open-source library for fast LLM inference and serving. It was created at UC Berkeley and is now a large community project. Its claim to fame is the technique you already understand deeply: PagedAttention (Module 5.3), which it introduced to slash KV-cache waste and thereby deliver far higher throughput than prior systems.

What vLLM gives you, out of the box, is essentially the entire toolbox from Module 05 already implemented and tuned:

  • PagedAttention KV-cache management (its signature).
  • Continuous batching of incoming requests.
  • FlashAttention-style optimized kernels.
  • Prefix caching, chunked prefill, speculative decoding, CUDA graphs, quantization support (GPTQ/AWQ/FP8/INT8, KV-cache quant), multi-LoRA, and multi-GPU parallelism.
  • An OpenAI-compatible API server, so existing OpenAI-client code can point at your vLLM instance with almost no changes.
  • Support for a huge range of model architectures from the Hugging Face ecosystem.

In short: vLLM is where Modules 02–05 stop being theory. You configure the ideas; vLLM executes them.

Where vLLM sits among alternatives (orientation, not tribalism): you’ll also hear about TGI (Hugging Face Text Generation Inference), TensorRT-LLM (NVIDIA’s highly optimized, NVIDIA-specific engine), SGLang (strong at structured/programmatic generation and prefix sharing), llama.cpp/Ollama (CPU/consumer/local, GGUF world from Module 04), and managed cloud endpoints. They share most concepts from this tutorial. vLLM is the most popular general-purpose open option and an excellent default; the skills you build here transfer directly to the others.

6.2 vLLM’s architecture (a mental model)

You don’t need internals to use vLLM, but a mental model helps you debug and tune. Roughly:

        Incoming requests (HTTP / Python calls)
                        │
                        ▼
              ┌───────────────────┐
              │     Scheduler     │  decides each step which requests run;
              │ (continuous batch)│  does continuous batching + chunked prefill
              └───────────────────┘
                        │
                        ▼
              ┌───────────────────┐
              │  KV Cache Manager │  PagedAttention: hands out KV blocks,
              │   (PagedAttention)│  tracks block tables, prefix sharing
              └───────────────────┘
                        │
                        ▼
              ┌───────────────────┐
              │   Model Executor  │  runs the model on the GPU(s) with
              │  (optimized kernels)│ FlashAttention, CUDA graphs, etc.
              └───────────────────┘
                        │
                        ▼
         GPU(s)  ──  weights + KV cache live here

The scheduler is the brain (what runs now, who gets batched), the KV cache manager is the memory boss (PagedAttention), and the executor does the math on the GPU. Newer vLLM (the “V1” engine rewrite) restructured these for lower overhead and cleaner async behavior, but the conceptual roles are the same.

A key startup concept: when vLLM launches, it figures out how much GPU memory to devote to the KV cache (governed by gpu-memory-utilization, below), pre-allocates that as PagedAttention blocks, and may “capture CUDA graphs” (Module 5.9) — which is why startup takes a little while and prints memory/graph messages.

6.3 Installation and prerequisites

What you need:

  • An NVIDIA GPU (for the standard build) with a recent enough CUDA-capable driver. (vLLM also has builds/support for some other hardware — AMD ROCm, etc. — but NVIDIA is the well-trodden path.)
  • A matching CUDA/driver stack. Version mismatches are the classic install headache (Module 08 revisits this).
  • Python (a recent version) and ideally a fresh virtual environment.

Typical install (verify current instructions in the docs):

# In a clean virtual environment
pip install vllm

That pulls in vLLM plus PyTorch and the CUDA bits it needs. For exotic hardware or pinned CUDA versions, follow the official install matrix rather than guessing. A quick sanity check:

python -c "import vllm; print(vllm.__version__)"
vllm --help

Common gotcha: “it imported fine but won’t run on the GPU” almost always means a driver/CUDA/PyTorch mismatch. Check nvidia-smi works, and that your driver supports the CUDA version vLLM was built against. We’ll treat this properly in Module 08; for now, know that most first-run failures are environment mismatches, not vLLM bugs.

6.4 Offline inference — vLLM as a Python library

The simplest way to use vLLM: process a list of prompts at maximum throughput, no server. This is the offline mode from Module 3.8 — perfect for batch jobs (classify a dataset, generate outputs, evaluate). It’s also the best way to learn.

from vllm import LLM, SamplingParams

# Load a model (downloaded from Hugging Face by name, or a local path).
# Start small if your GPU is small — e.g. a 1B–8B model.
llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct")

# Sampling settings map directly to Module 00's concepts:
sampling_params = SamplingParams(
    temperature=0.7,   # randomness (Module 0.3)
    top_p=0.9,         # nucleus sampling (Module 0.3)
    max_tokens=256,    # cap on OUTPUT tokens (decode length)
)

prompts = [
    "Explain what a GPU is in one paragraph.",
    "Write a haiku about memory bandwidth.",
]

# vLLM batches these together automatically (continuous batching, Module 5.2).
outputs = llm.generate(prompts, sampling_params)

for out in outputs:
    print("PROMPT:", out.prompt)
    print("OUTPUT:", out.outputs[0].text)
    print("---")

What’s happening under the hood, in terms you now own: vLLM tokenizes the prompts, schedules them with continuous batching, manages their KV caches with PagedAttention, runs optimized kernels on the GPU, and returns the decoded text. You wrote ten lines; Modules 02–05 are all running for you.

Tip for offline/batch jobs: pass all your prompts at once (a big list) rather than looping one at a time. vLLM’s whole advantage is batching them together — feeding them one-by-one throws away the throughput win (the offline mistake warned about in Module 3.8).

6.5 Online serving — the API server

For live traffic you run vLLM as a server exposing an OpenAI-compatible API (Module 3.1). Then any client that speaks the OpenAI API can talk to it.

Start the server:

vllm serve meta-llama/Llama-3.1-8B-Instruct
# (older syntax: python -m vllm.entrypoints.openai.api_server --model ...)

By default it listens on http://localhost:8000. Now call it exactly like the OpenAI API:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Explain the KV cache simply."}],
    "max_tokens": 200,
    "temperature": 0.7
  }'

Or from Python using the standard OpenAI client, just pointed at your server:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed-locally")

resp = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Explain the KV cache simply."}],
    max_tokens=200,
    stream=True,          # stream tokens as they generate (Module 00's decode loop)
)
for chunk in resp:
    delta = chunk.choices[0].delta.content or ""
    print(delta, end="", flush=True)

stream=True makes the server send tokens as they’re produced — this is the decode loop from Module 00 made visible, and it’s what gives users that “typing” feel with low perceived latency (good TTFT, then steady TPOT). The endpoints you’ll use most are /v1/chat/completions, /v1/completions, and often /v1/embeddings. There’s also a /metrics endpoint (Prometheus format) that exposes the serving metrics from Module 03 — we use it in Module 08.

6.6 The configuration flags that actually matter

This is the part to study. Each flag maps to a concept you already understand. (Names/defaults can shift between versions — confirm with vllm serve --help.)

Memory & KV cache

  • --gpu-memory-utilization (e.g. 0.90): the fraction of GPU memory vLLM may use. Higher = bigger KV cache = more concurrent requests / longer contexts (more batching, Module 02/03), but less headroom — push too high and you risk out-of-memory under load. The single most important throughput/stability knob. Start around 0.9 and adjust.
  • --max-model-len: the maximum context length (prompt + output) you’ll allow. Directly bounds KV-cache size per request (Module 2.5). Setting it lower than the model’s max can let you fit more concurrent requests. Setting it as high as the model supports costs KV memory.
  • --kv-cache-dtype (e.g. fp8): quantize the KV cache (Module 4.7). Roughly halves KV memory → more concurrency / longer context, with usually small quality impact.
  • --block-size: the PagedAttention block (page) size (Module 5.3). Usually leave at default.
  • --swap-space / CPU offload options: allow spilling KV cache to CPU (Module 5.9) when GPU memory is tight.

Batching & scheduling

  • --max-num-seqs: max number of requests batched concurrently. Higher = more throughput, but more KV memory and potentially higher per-request latency (the Module 3.3 trade-off, in a single knob).
  • --max-num-batched-tokens: max tokens processed per step across the batch; interacts with chunked prefill. Tuning this balances prefill vs decode work and tail latency (Module 5.6).
  • --enable-chunked-prefill: turn on chunked prefill (Module 5.6) to keep long prompts from stalling everyone’s decode. Often on by default in recent versions.

Reuse & speculation

  • --enable-prefix-caching: turn on automatic prefix caching (Module 5.4). Big TTFT win for shared system prompts / RAG / multi-turn chat. Cheap to enable; enable it if your workload shares prefixes.
  • Speculative decoding flags (e.g. specifying a draft model / method): enable speculative decoding (Module 5.7) for lower single-stream latency at low concurrency. Remember the caveat: not helpful under heavy batching.

Quantization

  • --quantization (e.g. awq, gptq, fp8): tell vLLM the weight quantization method of the model you’re loading (Module 04). Must match how the model was quantized. Smaller weights → fits bigger models, more KV room, often faster decode.
  • --dtype (e.g. auto, bfloat16, float16): the compute precision for an un-quantized model. auto/bfloat16 is usually right.

Multi-GPU (full treatment in Module 07)

  • --tensor-parallel-size (e.g. 2, 4, 8): split the model across N GPUs (tensor parallelism, Module 07) so a model too big for one GPU fits, and to add memory bandwidth. Set this to shard a 70B+ model across several GPUs on one machine.
  • --pipeline-parallel-size: split the model by layers across GPUs/nodes (pipeline parallelism, Module 07), often for multi-node.

Serving / ops

  • --port, --host: where the server listens.
  • --api-key: require a key for access (basic auth; Module 08 covers real security).
  • --served-model-name: the name clients use in the "model" field (handy aliasing).
  • --max-num-seqs, --max-model-len, --gpu-memory-utilization together are your primary tuning triangle for the latency/throughput/fit balance.

How to actually tune (the workflow): Don’t twiddle randomly. (1) Get it running with defaults on a small model. (2) Identify your bottleneck via metrics (Module 03/09): is it TTFT, TPOT, throughput, or fit? (3) Change one relevant flag, re-benchmark under realistic concurrency (Module 09), keep it if it helped. Tuning is measure → change one thing → measure, not guess-and-pray.

6.7 A realistic example: tuning for a chatbot vs a batch job

Putting flags to work with the Module 3.9 framing.

Scenario A — interactive chatbot (online; long shared system prompt; users want snappy responses):

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \          # shared system prompt → big TTFT win (5.4)
  --enable-chunked-prefill \         # keep long prompts from stalling TPOT (5.6)
  --max-model-len 8192 \             # bound KV per request to fit more users
  --max-num-seqs 64                  # moderate concurrency for good latency
# (optionally add speculative decoding since interactive + likely modest concurrency)

Scenario B — overnight batch classification of 5M documents (offline; throughput is everything; latency irrelevant):

Use the offline Python API (Section 6.4), feed all prompts at once, crank memory utilization and concurrency, skip latency-oriented tricks:

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    gpu_memory_utilization=0.95,   # squeeze every byte for max batching
    max_model_len=2048,            # short docs → small KV → huge concurrency
    # quantize weights/KV if it lets you batch even more without quality loss
)
# pass the entire list of prompts in one generate() call

Same engine, opposite tuning, because the goal differs — exactly the Module 03 lesson that you optimize throughput-subject-to-latency, and the latency constraint changes everything.

6.8 Reading vLLM’s logs and metrics

When running, vLLM logs the numbers you now understand: things like running/waiting request counts, GPU KV-cache usage %, prefix-cache hit rate, and throughput. Learn to read these:

  • KV-cache usage near 100% + a growing waiting queue → you’re memory-bound on concurrency. Levers: raise gpu-memory-utilization, lower max-model-len, quantize the KV cache, or add GPUs.
  • High prefix-cache hit rate → prefix caching is earning its keep (good for shared-prompt workloads).
  • Long queue / high TTFT → you’re capacity-limited; scale out (Module 08) or accept higher latency.

The /metrics endpoint exposes these in Prometheus format for dashboards (Module 08). Watching these while you change flags is how tuning actually happens.

6.9 Common pitfalls (so you skip the pain)

  • Out-of-memory at high load but fine when idle. KV cache grows with concurrency; you set gpu-memory-utilization too high or max-model-len/max-num-seqs too high for the hardware. Lower one, or quantize the KV cache. (Memory is dynamic — Module 02.)
  • Benchmarking with a single request and declaring it “slow.” Single-stream ignores vLLM’s whole point (batching). Always benchmark under realistic concurrency (Module 09).
  • Quantization flag mismatch. Pointing --quantization gptq at an AWQ model (or vice versa) fails or misbehaves. Match the flag to how the model was actually quantized (Module 04).
  • Expecting speculative decoding to help a fully-batched server. It won’t — there’s no spare compute (Module 5.7).
  • Driver/CUDA mismatch on first run. Environment issue, not a vLLM bug (Section 6.3, Module 08).
  • Forgetting prefix caching on a system-prompt-heavy workload. Free TTFT left on the table (Module 5.4).

Each pitfall is just a concept from an earlier module showing up in practice — which is the whole reason we built the concepts first.

Check your understanding

  1. Which famous technique did vLLM introduce, and what problem from Module 02 does it solve? (6.1)
  2. In vLLM’s architecture, what are the jobs of the scheduler vs the KV-cache manager vs the executor? (6.2)
  3. When would you use vLLM’s offline Python API instead of the API server, and what’s the #1 tip for feeding it prompts? (6.4, 6.7)
  4. What does --gpu-memory-utilization control, and what’s the risk of setting it too high? (6.6)
  5. For a chatbot with a long shared system prompt, which two flags give the biggest wins and why? (6.6, 6.7)
  6. You see KV-cache usage at ~100% with a growing waiting queue. Name two flags you’d change and the trade-off involved. (6.8)

Key terms introduced

vLLM · V1 engine · scheduler / KV-cache manager / executor · offline vs online vLLM · OpenAI-compatible endpoints · streaming · gpu-memory-utilization · max-model-len · max-num-seqs · max-num-batched-tokens · enable-prefix-caching · enable-chunked-prefill · kv-cache-dtype · quantization flag · tensor-parallel-size · /metrics endpoint · TGI / TensorRT-LLM / SGLang / llama.cpp (alternatives)

Next: Module 07 — Distributed & Multi-GPU Serving, for when one GPU simply isn’t enough.

← Module 05: Inference Optimization Techniques