Inference

Appears in 2 papers · 3 tutorials

The process of running a trained model on new inputs to generate predictions.

The process of running a trained model on new inputs to generate predictions. For LLMs, inference means generating tokens one at a time (autoregressive decoding). Inference is where the KV cache problem manifests — your compute is bounded by memory bandwidth, not GPU compute power.

As used in Paper 23 — Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Model Parameters →

The process of running a trained model to produce outputs for new inputs. Inference is when the model "answers" — in contrast to training, where parameters are updated.

As used in Fine-Tuning & Model Customization →

Using a trained model to generate outputs (as opposed to training it). (M00)

As used in AI Production Engineering →

Using a trained model to get an answer (as opposed to training it). In production you do inference constantly. (Mod 0)

As used in LLM Infrastructure →

Using a trained model to produce outputs (every request is an inference). The subject of this tutorial.

Paper 18 — Mistral 7B → Paper 23 — Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Model Parameters → Fine-Tuning & Model Customization → AI Production Engineering → LLM Infrastructure →