KV Cache

The memory buffer storing Key and Value vectors from all previous tokens during autoregressive (token-by-token) generation. In standard Multi-Head Attention, the KV cache grows linearly with sequence length and quadratically with model size (since each token must cache KV pairs for all previous tokens). The KV cache is often the memory bottleneck in LLM inference, not the model weights.

Cached attention Keys/Values for already-generated tokens; speeds generation but uses memory that grows with context and concurrency. (M14)

Stored Keys and Values for all prior tokens, so they aren't recomputed each step. Makes generation practical, but grows with tokens, exists per-request, and must be read every decode step — the central memory challenge in serving.