Speculative decoding
Using a small **draft model** (or heuristic) to guess several tokens, then verifying them in one parallel pass of the big model.
Using a small draft model (or heuristic) to guess several tokens, then verifying them in one parallel pass of the big model. Lowers single-stream latency when there's spare compute (low concurrency); unhelpful when already batched full. Output is identical to normal decoding. Variants: Medusa, EAGLE, n-gram lookup.