Glossary
Glossary
A plain-language reference for every important term in the LLM Infrastructure tutorial. Terms are grouped by theme, and the module where each is introduced is noted in (parentheses). When in doubt, read the one-line definition, then revisit the module for the full picture.
LLM (Large Language Model)
A program that has learned, from huge amounts of text, to predict the next chunk of text. All its abilities come from this one skill applied repeatedly.
Parameter / weight
One of the billions of tunable numbers inside a model that together encode what it “knows.” More parameters = more memory and compute needed.
Token
A chunk of text (a word, part of a word, or punctuation) — the actual unit LLMs read and produce. Roughly 1 token ≈ 0.75 English words ≈ 4 characters.
Vocabulary
The fixed list of all tokens a model knows (often tens to hundreds of thousands).
Tokenization
Splitting text into tokens (and detokenization is converting tokens back to text).
Context window / context length
The maximum number of tokens (prompt + output) a model can handle at once, e.g. “128K context.”
Input / prompt tokens
The text you send in. Output / generated / completion tokens — the text the model produces. They cost different amounts of work.
Autoregressive generation
Producing output one token at a time, each new token depending on all previous ones. The root cause of most serving challenges.
Sampling
Choosing the actual next token from the model’s predicted probabilities. Controlled by temperature, top-p, top-k.
Temperature
A knob (≈0–2) for randomness: low = focused/repetitive, high = creative/varied.
Top-p (nucleus) / Top-k sampling
Methods that restrict sampling to the most likely tokens (smallest set summing to probability p, or the top k).
Inference
Using a trained model to produce outputs (every request is an inference). The subject of this tutorial.
Training
The separate, earlier process of creating a model by tuning its parameters. Not covered here.
Prefill
The first phase: the model reads the whole prompt at once (in parallel), builds the KV cache, and produces the first token. Compute-bound; sets TTFT.
Decode
The second phase: generating output tokens one at a time, sequentially. Memory-bound; sets TPOT; the KV cache grows here.
CPU
A processor with a few very capable cores; great at complex sequential tasks. (The “master chefs.”)
GPU (Graphics Processing Unit)
A processor with thousands of simpler cores; great at massively parallel work like the matrix math in LLMs. (The “stadium of line cooks.”)
Core
An individual processing unit; GPUs have thousands.
Matrix multiplication
The core math operation in LLMs; millions of independent multiply-adds, ideal for parallel GPU hardware.
FLOPS / TFLOPS
Floating-point Operations Per Second; raw compute speed. TFLOPS = trillions of FLOPS.
VRAM / GPU memory
The GPU’s dedicated memory, separate from system RAM, measured in GB. Holds weights, KV cache, activations.
Memory capacity
How much data (GB) the GPU can hold; decides whether a model fits.
Memory bandwidth
How fast data moves between GPU memory and cores (GB/s or TB/s); largely decides decode speed.
Compute-bound
Limited by calculation speed (cores busy, data plentiful). Prefill is typically compute-bound.
Memory-bound
Limited by data-movement speed (cores idle, waiting for data). Decode is memory-bound — the single most important performance fact in serving.
Precision / bits
How many bits store each number. More bits = more accurate but bigger.
FP32 / FP16 / BF16 / FP8 / INT8 / INT4
Number formats of 32/16/16/8/8/4 bits (= 4/2/2/1/1/0.5 bytes per parameter). FP16/BF16 are common serving baselines; the rest are quantized.
HBM (High Bandwidth Memory)
The fast memory stacked next to data-center GPUs, giving them their high bandwidth.
Memory hierarchy
Tiers from tiny-fast (registers, SRAM/L1) to big-slow (VRAM), then off-GPU (system RAM, disk). Good kernels keep hot data in fast tiers.
CUDA
NVIDIA’s platform for running general computation on their GPUs. (ROCm is AMD’s equivalent.)
Kernel
A small program that runs one operation on the GPU. Optimized kernels make engines fast.
Tensor Cores
Special GPU units that do low-precision matrix math (FP16/INT8/FP8) very fast; a reason quantization speeds compute.
NVLink / NVSwitch
NVIDIA’s very fast GPU-to-GPU links inside a server; what makes tensor parallelism viable.
PCIe
A slower general-purpose bus; TP over PCIe-only is often too slow.
Transformer
The architecture nearly all modern LLMs use: a tall stack of identical layers.
Layer
One block in the stack; each has an attention part and a feed-forward part.
Attention
The mechanism letting each token “look at” relevant other tokens. Where the KV cache comes from.
Feed-forward network / MLP
The per-token processing part of a layer; holds many parameters.
Query / Key / Value (Q / K / V)
Per-token vectors: Query (“what I seek”), Key (“what I offer”), Value (“info I provide”). Attention matches Queries against Keys to blend Values.
KV cache (Key-Value cache)
Stored Keys and Values for all prior tokens, so they aren’t recomputed each step. Makes generation practical, but grows with tokens, exists per-request, and must be read every decode step — the central memory challenge in serving.
Attention head
One of several parallel attention computations; more heads = richer attention, bigger KV cache.
MHA / MQA / GQA
Multi-Head / Multi-Query / Grouped-Query Attention: design choices trading KV-cache size against quality. GQA (a middle ground) is the modern default for efficient KV cache.
Hidden size / embedding dimension
How wide each token’s internal representation is; bigger = more capacity, memory, compute.
Mixture of Experts (MoE)
An architecture where each token uses only a few of the model’s many “experts” per step — cheap to run per token, but all experts must still be stored.
Serving / inference server
Running a model as a continuous service that handles many users’ requests over an API.
OpenAI-compatible API
A de-facto standard API shape most engines (including vLLM) support, so clients can switch backends easily.
Latency
How long one request takes (a per-user measure).
Throughput
Total work across all users (e.g. total output tokens/sec or requests/sec); higher = lower cost per request.
TTFT (Time To First Token)
Time from sending a request to the first output token. Includes queue wait + prefill. The “pause before it starts typing.”
TPOT (Time Per Output Token) / ITL (Inter-Token Latency)
Average gap between subsequent output tokens. The “typing speed.” ≈ 1 / per-stream tokens-per-second.
Latency–throughput trade-off
Bigger batches raise throughput but can raise per-user latency. The central tension of serving; the goal is max throughput subject to a latency SLO.
SLO (Service Level Objective)
An internal performance target, stated at a percentile (e.g. “p99 TTFT < 800 ms”).
SLA (Service Level Agreement)
A contractual performance promise, usually looser than the SLO.
Percentiles (p50 / p95 / p99)
The value below which 50% / 95% / 99% of requests fall. Used instead of averages because averages hide bad experiences.
Tail latency
The slow end (p95/p99) of the latency distribution; where real user frustration lives.
Goodput
Throughput counting only requests that met their latency SLO. The best single summary of serving quality.
GPU utilization (and its trap)
A “GPU is busy” reading that can sit at 100% during memory-bound decode while compute is idle — so it’s a misleading efficiency signal for LLMs.
Online vs offline serving
Online = live users, latency-sensitive. Offline = batch jobs, throughput is everything.
Quantization
Storing a model’s numbers in fewer bits, trading a little accuracy for big memory savings and often more speed.
Weight-only vs weight+activation quantization
Compressing just the stored weights (big memory/bandwidth win, easy on quality) vs also quantizing the live activations for compute speedups (harder, due to outliers).
WxAy notation
Weights x-bit, activations y-bit (e.g. W4A16 = 4-bit weights, 16-bit activations).
Dequantization
Converting quantized numbers back to higher precision for computation; adds small overhead.
GPTQ
A careful, calibration-based weight-only 4-bit quantization method. Well supported by vLLM.
AWQ (Activation-aware Weight Quantization)
A 4-bit weight-only method that protects the most important weights. Good quality-for-size; well supported by vLLM.
GGUF
A model file format (from llama.cpp) with flexible quant levels (Q4_K_M, etc.); popular for CPU/consumer/local use.
SmoothQuant / LLM.int8()
INT8 methods that handle activation outliers so 8-bit quantization stays accurate.
FP8
Modern 8-bit float, natively accelerated on the newest GPUs; memory savings + real compute speedup with small quality loss.
Calibration dataset
A small set of representative texts used to set quantization scales well.
KV-cache quantization
Storing the KV cache in 8-bit (or lower) to roughly halve its memory, enabling more concurrency / longer context.
Pre-quantized model
An already-quantized model you download and serve directly.
Static batching
Naive batching that waits for all requests in a batch to finish; wastes GPU slots.
Continuous (in-flight / dynamic) batching
Refreshing the batch every token step, removing finished requests and adding new ones. A foundational throughput win.
PagedAttention
vLLM’s KV-cache manager that stores the cache in small fixed-size blocks (pages) allocated on demand via a block table, eliminating waste and fragmentation and enabling much bigger batches. Borrowed from OS virtual memory.
KV block / page / block table
The fixed-size unit of KV memory, and the per-request map from logical token positions to physical blocks.
Copy-on-write KV sharing
Letting requests share identical KV blocks (e.g. same prompt), copying only when they diverge.
Prefix caching (Automatic Prefix Caching, APC)
Reusing the cached KV of shared prefixes (system prompts, RAG context, chat history) to skip redundant prefill. Big TTFT win for shared-prefix workloads.
FlashAttention
An exact, memory-efficient attention kernel that avoids writing big intermediates to slow memory; speeds attention, especially for long contexts. No quality loss.
Chunked prefill
Breaking long prefills into chunks interleaved with decode steps, so big prompts don’t stall everyone’s TPOT.
Speculative decoding
Using a small draft model (or heuristic) to guess several tokens, then verifying them in one parallel pass of the big model. Lowers single-stream latency when there’s spare compute (low concurrency); unhelpful when already batched full. Output is identical to normal decoding. Variants: Medusa, EAGLE, n-gram lookup.
Disaggregated prefill/decode
Running prefill and decode on separate GPU pools so the two phases stop interfering; a large-scale technique.
CUDA graphs
Recording a whole step’s GPU operations and replaying them as one unit to cut launch overhead.
Kernel fusion
Merging several small operations into one kernel to reduce memory traffic.
KV offloading / swapping
Moving cold KV cache to CPU/disk when GPU memory is tight; trades speed for capacity.
LoRA / multi-LoRA serving
LoRA = small add-on weight patches that customize a base model. Multi-LoRA serving runs many such adapters on one shared base model, saving memory vs loading full separate models.
vLLM
The most popular open-source LLM inference/serving engine; introduced PagedAttention and bundles the whole optimization toolbox plus an OpenAI-compatible server.
V1 engine
vLLM’s re-architected core for lower overhead and cleaner async behavior.
Scheduler / KV-cache manager / executor
vLLM’s three conceptual roles: decides what runs (continuous batching), manages KV memory (PagedAttention), and runs the model on the GPU.
Offline vs online vLLM
The Python library (LLM.generate) for batch jobs vs the API server (vllm serve) for live traffic.
Streaming
Sending output tokens to the client as they’re generated (the decode loop made visible).
gpu-memory-utilization
Flag for the fraction of GPU memory vLLM may use; the key throughput/stability knob (bigger KV cache vs less headroom).
max-model-len
Flag bounding max context length, which bounds KV-cache size per request.
max-num-seqs
Flag for max concurrently batched requests; trades throughput against latency and memory.
max-num-batched-tokens
Flag for max tokens processed per step; balances prefill vs decode and tail latency.
enable-prefix-caching / enable-chunked-prefill / kv-cache-dtype / quantization
Flags turning on the corresponding techniques from Modules 04–05.
/metrics endpoint
vLLM’s Prometheus-format metrics for dashboards.
TGI / TensorRT-LLM / SGLang / llama.cpp / Ollama
Alternative engines: Hugging Face’s TGI, NVIDIA’s highly optimized TensorRT-LLM, SGLang (structured generation/prefix sharing), and the CPU/local llama.cpp + Ollama world.
Node
One physical server (often holding multiple GPUs).
Tensor parallelism (TP)
Splitting each layer’s math across GPUs; makes a big model fit and adds bandwidth, but needs constant fast communication — keep within one NVLink node. (vLLM: --tensor-parallel-size.)
Pipeline parallelism (PP)
Splitting the model by layers across GPUs/nodes, assembly-line style; less communication, so it can cross nodes. Risk: pipeline “bubbles.” (vLLM: --pipeline-parallel-size.)
Data parallelism (DP)
Running multiple full copies (replicas) of a fitting model behind a load balancer to add throughput. Scales nearly linearly.
Expert parallelism (EP)
Distributing a Mixture-of-Experts model’s experts across GPUs.
Replica
One complete serving instance (a single GPU or a TP/PP group); the unit you data-parallelize and autoscale.
Pipeline bubble
Idle gaps in pipeline parallelism; reduced by keeping many micro-batches in flight.
Micro-batch
A small chunk of work kept flowing through a pipeline to fill bubbles.
InfiniBand / RoCE / RDMA
Fast inter-node networking technologies that make multi-node serving practical.
Interconnect
Any GPU-to-GPU or node-to-node link; its speed dictates which parallelism you can use where (the governing rule: chatty parallelism → fastest link).
Container / Docker
A portable bundle of app + dependencies that runs identically everywhere; the fix for GPU environment (“works on my machine”) pain.
NVIDIA Container Toolkit
Lets containers access the host’s GPUs.
Kubernetes (K8s)
The standard orchestrator for running, scaling, and self-healing many containers across machines.
Pod / Deployment / ReplicaSet
K8s units: the running container(s), and the controllers that keep N replicas alive (operationalizing data parallelism).
Liveness vs readiness probe
Health checks: “is it alive?” (restart if not) vs “is it ready to serve?” (don’t route traffic until the model is loaded — crucial for slow-loading LLMs).
KServe / Ray Serve
LLM/ML-aware serving layers built on Kubernetes.
Autoscaling
Adjusting replica count to match traffic; for LLMs, scale on queue length / KV-cache utilization / TTFT-vs-SLO, not CPU. Complicated by slow model-load scale-up.
Warm pool
Pre-loaded spare capacity kept ready because LLM scale-up is slow.
Prometheus / Grafana
Tools to collect (scrape) and visualize metrics.
Observability
Metrics, logs, and traces that let you see what the system is doing.
Alerting
Automated paging when SLOs are at risk; alert on user-felt symptoms (latency, errors), not just machine stats.
Admission control / rate limiting
Capping the queue and per-user load; rejecting fast under overload instead of accepting unservable work.
Retry storm
Cascading overload caused by naive client retries; mitigated by backoff and circuit breakers.
Rolling / blue-green / canary deployment
Update strategies: replace replicas gradually / run new alongside old then switch / send a small traffic slice to the new version first. Canary is vital for LLMs because new models can change output quality.
Prompt injection
Malicious instructions hidden in input (especially dangerous with tools or untrusted RAG documents).
Guardrails / moderation
Input/output filtering to keep harmful or unsafe content in check.
Multi-tenant isolation
Ensuring shared infrastructure doesn’t leak one tenant’s data to another (watch shared prefix caches).
Closed-loop vs open-loop load
Keeping exactly N requests in flight vs sending requests at an arrival rate (queuing if busy). Both reveal different behavior.
Warm-up
Early requests that pay startup costs; discard them and measure steady state.
Latency-throughput curve
A plot of throughput vs p99 latency across concurrency levels; the most important serving chart.
The knee / operating point
The point of maximum throughput still within your latency SLO; your target operating point (≈ goodput maximum).
benchmark_serving / benchmark_throughput
vLLM’s built-in load/benchmark scripts.
Load generator (locust / k6)
Tools to drive realistic concurrent/rate-based traffic.
Single-replica capacity
The throughput one replica sustains within SLO at your length profile; the basis for sizing.
Safety factor / headroom
Extra capacity (e.g. 1.2–1.5×) above peak demand to absorb variance and tails.
Capacity planning
Estimating how many replicas/GPUs a workload needs: peak demand ÷ single-replica capacity × safety factor.
Cost per token
GPU $/hour ÷ tokens/hour at the operating point; the metric leadership cares about. Optimizations lower it by raising tokens/hour per GPU.
Build vs buy
Self-hosting GPUs (cheaper at high steady volume) vs managed per-token APIs (cheaper/simpler at low or spiky volume).
Goodput per dollar within SLO
The one-line summary of the serving engineer’s objective.
Dependency tree of the field
The map showing how every technique traces back to a few root facts (token-by-token generation; separate GPU compute/memory; memory-bound decode and the KV cache).
Decision framework
The repeatable nine-step procedure for designing any serving deployment (brief → fit → precision → parallelism → optimizations → config → benchmark/size → operate → cost).
Diagnostic checklist
The ordered set of checks for diagnosing slow or broken serving, each mapping to an earlier module’s concept.
Back to the tutorial index.