Glossary

Plain-language definitions of every important term in this tutorial. Terms are grouped by theme, and an alphabetical index is at the bottom. The module where each term is introduced or used most is noted in parentheses.

AI (Artificial Intelligence)

Software that performs tasks once thought to require human intelligence, like understanding language or making decisions. (Mod 0)

Model

The trained “brain” of an AI system: a large mathematical function that makes predictions. You send it input and it returns output. (Mod 0)

LLM (Large Language Model)

A model trained to understand and generate human language. The thing behind chatbots. (Mod 0)

Inference

Using a trained model to get an answer (as opposed to training it). In production you do inference constantly. (Mod 0)

Training

Teaching a model by showing it lots of data. Done rarely; expensive. (Mod 0)

Fine-tuning

Further training an existing model on your own examples to specialize it (your domain, tone, or to make a small model good enough to replace a big one). (Mod 5, 10)

Token

The unit LLMs read and write in; roughly ¾ of a word in English. You pay per token, and models have token limits. The most important unit in AI production. (Mod 0, 5)

Prompt

The full text sent to an LLM: instructions + context + history + the user’s message. (Mod 0, 1)

System prompt / instructions

The standing orders in a prompt that set the AI’s behavior, rules, and tone. (Mod 1)

Context window

The maximum number of tokens a model can read at once (its short-term memory / desk space). Everything in the prompt counts against it. (Mod 1)

Temperature

A setting controlling randomness/creativity. Low = focused and repeatable; high = varied and creative. (Mod 1)

Max tokens

A cap on how long the model’s output can be. (Mod 1)

Hallucination

When an AI confidently states something false as if it were true. A core AI failure mode. (Mod 0, 8)

Deterministic

Same input always gives the same output (normal software). (Mod 0)

Probabilistic / non-deterministic / stochastic

Same input can give different outputs (AI). The root challenge of the field. (Mod 0)

Production

The live environment real users use, as opposed to a demo or test setup. (Mod 0)

Prompt engineering

The craft of writing effective instructions for an LLM. Important but only the start of production work. (Mod 1)

Orchestration

The code that assembles the prompt and manages the flow of a request (retrieval, history, model call, output handling). (Mod 1)

RAG (Retrieval-Augmented Generation)

Searching your own documents for relevant text and pasting it into the prompt so the model can answer with private/current data. Like an open-book exam. (Mod 1)

Retrieval

The “search” step of RAG that finds the most relevant chunks of text for a question. (Mod 1)

Vector database

A database that stores text as embeddings (lists of numbers capturing meaning) so you can search by meaning, not just keywords. (Mod 1)

Embedding

A list of numbers representing the meaning of a piece of text, so similar meanings land near each other. (Mod 1)

Tool / function calling

Giving the model the ability to call functions (look up an order, do math, search) and use the results. (Mod 1)

Agent

An LLM in a loop with tools and a goal: it thinks, calls a tool, sees the result, and repeats until done. Powerful but harder to run in production. (Mod 1)

Guardrail

A check around the model. Input guardrails filter incoming messages; output guardrails scan responses before they reach the user or take effect. (Mod 1, 6)

Gateway / proxy

A single chokepoint all model calls pass through, used to centralize caching, routing, rate limits, budgets, and logging. (Mod 1, 5)

Provider

The company running the model you call over the internet (e.g., OpenAI, Anthropic, Google). An external dependency you don’t control. (Mod 1, 9)

Open-weight model

A model whose weights you can download and run yourself, rather than calling a provider’s API. (Mod 1, 5)

Observability

The ability to understand what’s happening inside a system from the data it produces. The foundation pillar. (Mod 2)

Monitoring

Tracking whether known things are wrong (vs. observability, which lets you investigate why, including unforeseen problems). (Mod 2)

Log

A timestamped record of a single event (“what exactly happened in this one case”). (Mod 2)

Metric

A number measured and aggregated over time (“how is the system doing overall”). (Mod 2)

Trace

The full, timed story of one request through all its steps, built from spans. The key AI debugging tool. (Mod 2, 3)

Span

A single timed unit of work within a trace (e.g., “retrieval took 120 ms”), with attributes. Spans nest into a tree. (Mod 3)

Attribute

A key-value detail attached to a span (model name, token count, cost, etc.). (Mod 3)

Trace ID / correlation ID

The identifier that ties all the steps of one request together. (Mod 2, 3)

OpenTelemetry (OTel)

An open, vendor-neutral standard for creating traces, spans, and metrics, so you aren’t locked to one tool. Has emerging conventions for GenAI. (Mod 3)

Instrumentation

Adding the code that creates spans/logs/metrics. Can be automatic, via decorators, or manual. (Mod 3)

Context propagation

Passing trace context so each span knows its parent and the trace tree assembles correctly. Breaks cause “flat” or fragmented traces. (Mod 3)

Percentile (p50/p95/p99)

A way to summarize a distribution. p95 latency = 95% of requests are faster than this. Watch tails, not just averages. (Mod 2)

Cardinality

How many distinct values a label can have. High-cardinality labels (like user ID) belong in logs/traces, not metrics. (Mod 2)

Sampling

Keeping full detail for a representative subset of requests (plus all errors/flagged cases) to control storage cost at scale. (Mod 2)

Time to first token

How long until the first word of a streamed response appears. The latency users actually feel. (Mod 3, 8, 9)

Evaluation / eval

Systematically measuring AI output quality across many cases. The scorecard for everything else. (Mod 4)

Test set / dataset (for evals)

A curated collection of test inputs (and often expected outputs) you run your system against. Grows from production failures. (Mod 4)

Grader / scorer

Whatever decides how good an output is: code-based, human, or LLM-as-judge. (Mod 4)

Code-based grader

A deterministic rule check (valid JSON? contains required fact? correct label?). Cheap and repeatable. (Mod 4)

LLM-as-judge

Using another LLM call to grade an output against a rubric. Scales nuanced evaluation; must be validated against humans. (Mod 4)

Faithfulness / groundedness

Whether an answer sticks to its provided sources rather than making things up. A key RAG metric. (Mod 4)

Offline evaluation

Running your system against a fixed test set before shipping a change. Catches regressions. (Mod 4)

Online evaluation

Measuring quality on live production traffic via feedback and sampled LLM-judge scoring. (Mod 4)

A/B test

Running two versions side by side on comparable users to measure which is actually better. Online eval used to decide. (Mod 4, 10)

Flywheel

The improvement loop: production failures feed the eval set, which catches them before the next release, which improves the system. (Mod 4, 10)

Regression

When a change makes some previously-good cases worse. Caught by offline evals. (Mod 4, 10)

Cost management / FinOps for AI

Making AI cost visible, accountable, and continuously optimized. A first-class engineering concern. (Mod 5)

Input tokens / output tokens

Tokens you send vs. tokens the model generates. Output usually costs more per token. (Mod 5)

Cost per request / unit economics

The dollar cost of one request, and per-user/per-conversation/per-outcome cost. Tells you if the feature is sustainable. (Mod 5)

Cost attribution

Breaking spend down by feature, customer, model, or endpoint to see what/who drives cost. (Mod 5)

Caching

Reusing prior results to avoid paying again. Exact-match, semantic (by meaning), and prompt caching (provider caches a prompt prefix at a discount). (Mod 5)

Model routing / cascading

Sending each request to the cheapest model that can handle it; escalating to a bigger model only when needed. A major cost saver. (Mod 5)

Batch processing

Running non-urgent requests together at a discount. (Mod 5)

Budget / quota / circuit breaker (cost)

Limits and automatic stops that prevent runaway spending from bugs, spikes, or abuse. (Mod 5)

Cost-quality-latency triangle

The trade-off: you can’t maximize cheap, good, and fast all at once; you choose the balance per use case. (Mod 5)

Prompt injection

An attack where malicious instructions in the text the model reads hijack its behavior. The signature AI attack. (Mod 6)

Direct injection / jailbreaking

The user types malicious instructions to bypass the AI’s rules. (Mod 6)

Indirect injection

Malicious instructions hidden in content the model processes (a web page, document, email). Scarier, especially for agents. (Mod 6)

OWASP LLM Top 10

A standard list of the top security risks for LLM applications; a useful threat checklist. (Mod 6)

Insecure output handling

Trusting model output blindly and passing it into a database, shell, or page, enabling classic attacks. Treat output as untrusted input. (Mod 6)

Excessive agency

Giving an agent more power/tools than it needs, so one mistake or injection causes outsized damage. (Mod 6)

Least privilege / minimal agency

Giving the model and its tools the minimum power needed, to limit the blast radius of any failure. (Mod 6)

PII (Personally Identifiable Information)

Personal data (names, emails, etc.) that must be protected, redacted, and handled per privacy law. (Mod 6)

Red-teaming

Deliberately attacking your own system to find weaknesses before real attackers do. Security applied as evaluation. (Mod 6)

GDPR / CCPA / HIPAA / EU AI Act

Examples of data-protection and AI regulations imposing real obligations (consent, deletion, transparency, sometimes mandatory human oversight). (Mod 6)

Human-in-the-loop (HITL)

A human must approve before the AI’s action takes effect. For high-stakes, irreversible work. (Mod 7)

Human-on-the-loop (HOTL)

The AI acts autonomously but a human monitors and can intervene. For medium-stakes, high-volume work. (Mod 7)

Human-out-of-the-loop

Fully autonomous AI; humans only review aggregates and samples afterward. For low-stakes, reversible work. (Mod 7)

Review gate

A point where the flow pauses for human judgment, placed by stakes and uncertainty. (Mod 7)

Escalation / routing

Sending only the cases that need judgment (low confidence, flagged, high-stakes) to humans, and to the right human. (Mod 7)

Automation bias

The tendency of human reviewers to over-trust the AI and stop truly checking. A HITL failure mode to design against. (Mod 7)

Graduated autonomy

Moving a task from tighter to looser human oversight as the AI proves itself (and back if quality drops). (Mod 7)

AI UX

Designing experiences for an uncertain, fallible, probabilistic tool; also a safety mechanism against overreliance. (Mod 8)

Calibrated trust

Helping users trust the AI exactly as much as it deserves — avoiding both over-trust and under-trust. (Mod 8)

Streaming

Showing the answer as it’s generated, word by word, to slash perceived latency. The signature AI UX pattern. (Mod 8, 9)

Citations / sources

Linking the documents an answer is based on, so users can verify. A high-value trust pattern. (Mod 8)

Escape hatch

A clear way for the user to reach a human or bypass the AI when it isn’t working. (Mod 8, 7, 9)

Overreliance

Users trusting AI output (including hallucinations) too much and acting on it. A safety risk UX must counter. (Mod 8, 6)

Conversational vs. embedded UX

Open chat (flexible but a blank-box burden) vs. AI woven into a specific workflow (constrained, often easier to make reliable). (Mod 8)

Reliability

Whether the system keeps working under real load and failures. (Mod 9)

Latency

How long a response takes. Measured at percentiles; perceived latency (via streaming) matters as much as actual. (Mod 9)

SLO (Service Level Objective)

A committed target like “99.9% success” or “p95 < 3s.” Define one per feature. (Mod 9)

Timeout

A maximum wait time for a call; exceeding it triggers a fallback. (Mod 9)

Retry (with backoff and jitter)

Re-attempting a failed call, waiting progressively longer (backoff) with randomness (jitter), only for retryable errors, with a cap. (Mod 9)

Fallback

A worse-but-working alternative when the primary path fails (another model, a cache, a non-AI path, or a human). The key AI reliability pattern. (Mod 9)

Rate limiting

Capping request volume — the provider’s limits on you, and your limits on each user (protects availability, cost, and against abuse). (Mod 9, 5, 6)

Circuit breaker

Automatically stops sending requests to a clearly-failing dependency for a while, using the fallback instead. (Mod 9)

Graceful degradation

Designing the system to get slower/simpler/fall back under stress rather than failing outright. (Mod 9)

Load shedding

Politely rejecting or queuing low-priority traffic under extreme load to protect the whole system. (Mod 9)

CI/CD (Continuous Integration / Continuous Deployment)

Automated testing and deployment on every change; for AI, evals run here to gate changes. (Mod 10)

Versioning

Tracking changes to prompts, models, retrieval config, guardrails, and code so behavior is reproducible and rollback is possible. (Mod 10)

Feature flag

A switch to turn a change on/off instantly without redeploying. The basis of safe rollout. (Mod 10)

Canary release

Releasing a change to a small percentage of traffic first, watching metrics, then expanding. (Mod 10)

Shadow mode

Running a new version alongside production without showing its output to users, just logging/evaluating what it would have done. (Mod 10)

Rollback

Quickly reverting to the previous version when a change goes wrong. The most important deploy safety property. (Mod 10)

Drift

Behavior decaying over time: model drift (the provider changed the model), data drift (inputs changed), knowledge drift (your knowledge base went stale). Caught by scheduled evals and monitoring. (Mod 10)

MLOps / LLMOps

The practices and tooling for deploying, monitoring, and continuously improving ML/LLM systems in production. (Mod 10)

Maturity model

A scale (Level 0 “Vibes” to Level 4 “Optimized”) for assessing how production-ready an AI system is across all pillars. (Mod 0, 11)

A/B test · Agent · AI · AI UX · Attribute · Automation bias · Batch processing · Budget · Calibrated trust · Canary release · Caching · Cardinality · Cascading · CI/CD · Circuit breaker · Citations · Code-based grader · Context propagation · Context window · Conversational UX · Correlation ID · Cost attribution · Cost management · Cost per request · Cost-quality-latency triangle · Dataset · Deterministic · Direct injection · Drift · Embedding · Embedded UX · Escalation · Escape hatch · Eval · Evaluation · Excessive agency · Faithfulness · Fallback · Feature flag · Fine-tuning · Flywheel · FinOps · Function calling · Gateway · GDPR · Grader · Graceful degradation · Graduated autonomy · Groundedness · Guardrail · Hallucination · HIPAA · HITL · HOTL · Human-out-of-the-loop · Indirect injection · Inference · Input guardrail · Input tokens · Instrumentation · Insecure output handling · Jailbreaking · Knowledge drift · Latency · Least privilege · LLM · LLM-as-judge · Load shedding · Log · LLMOps · Max tokens · Maturity model · Metric · MLOps · Model · Model drift · Model routing · Monitoring · Observability · Offline evaluation · Online evaluation · Open-weight model · OpenTelemetry · Orchestration · Output guardrail · Output tokens · Overreliance · OWASP LLM Top 10 · Percentile · PII · Production · Prompt · Prompt caching · Prompt engineering · Prompt injection · Provider · Probabilistic · Proxy · Quota · RAG · Rate limiting · Red-teaming · Regression · Reliability · Retrieval · Retry · Review gate · Rollback · Sampling · Semantic caching · Shadow mode · SLO · Span · Streaming · System prompt · Temperature · Test set · Time to first token · Timeout · Token · Tool · Trace · Trace ID · Training · Unit economics · Vector database · Versioning