Module 02 — Observability Fundamentals

Goal of this module: Learn what observability is, why it is the foundation of every other pillar, the classic “three pillars” (logs, metrics, traces), and — crucially — what is different about observing AI systems versus normal software. By the end you will know exactly what to capture for every AI request and why.

2.1 What observability actually means

Observability is the ability to understand what is happening inside a system by looking at the data it produces from the outside.

A simpler way to say it: observability is how you answer questions about your system without having to add new code and wait for the problem to happen again. A well-observed system lets you ask, after the fact, “why was this user’s answer wrong/slow/expensive?” and actually find out.

There is a classic distinction worth learning:

Monitoring tells you whether something is wrong. (A dashboard light goes red.) It answers questions you knew to ask in advance.
Observability lets you figure out why it’s wrong, including for problems you never anticipated. It lets you ask new questions of data you already collected.

In normal software, observability is important. In AI software it is essential and non-negotiable, for one reason: AI fails quietly and probabilistically (Module 00). A normal app that breaks throws an error you can catch. An AI app that “breaks” returns a fluent, confident, wrong answer — and unless you recorded the prompt, the context, the retrieved documents, and the output, you have no way to ever know it happened or why. You cannot reproduce it, because the same input might behave differently next time.

The golden rule: If you did not log it, it did not happen — you simply cannot prove or investigate it. In AI, capture everything about every request.

2.2 Why observability comes first

Recall from Module 00 that observability is the foundation pillar. Here’s the dependency chain, made explicit:

You can’t evaluate quality (Module 04) without recording inputs and outputs to score.
You can’t manage cost (Module 05) without recording tokens per request.
You can’t investigate a security incident (Module 06) without a trace of what the attacker sent and what the model did.
You can’t improve UX (Module 08) without data on where users get bad answers or give up.
You can’t debug reliability (Module 09) without timing data on each step.

Every other pillar consumes observability data. Build this first, or build the rest blind.

2.3 The three pillars: logs, metrics, traces

The traditional foundation of observability has three kinds of data. Learn them — they are the vocabulary of the whole field.

Logs

A log is a timestamped record of a single event. “At 10:31:07, user 482 asked X and we replied Y.” Logs are detailed and specific. They answer “what exactly happened in this one case?” The downside: there are millions of them, so you need to be able to search and filter them.

For AI, a single log entry for one request should capture (this is the canonical “capture everything” list — memorize it):

A unique request ID (so you can find this exact request later).
User ID / session ID (who, and which conversation).
Timestamp.
The full prompt actually sent to the model — system instructions, assembled context, history, and user message. (Not just the user’s question! The assembled prompt is what the model actually saw.)
The retrieved documents (for RAG) — which chunks were pulled and their scores.
The model and version used, and the settings (temperature, max tokens).
The full model output, including any tool calls it made and tool results.
Token counts: input tokens, output tokens (and cached tokens if applicable).
Latency: how long the whole thing took, ideally broken down by step.
Cost in dollars (derived from tokens × price).
Outcome: success, error, guardrail-blocked, etc.
User feedback if available (thumbs up/down, did they retry, did they escalate).

Privacy note: Prompts and outputs often contain personal or sensitive data. Logging “everything” must be balanced with privacy and security — redact or encrypt sensitive fields, control who can read logs, and set retention limits. We cover this in Module 06. Capture richly, but handle the captured data responsibly.

Metrics

A metric is a number measured over time and aggregated. “Average response time this hour was 3.2 seconds.” “We served 12,000 requests today.” “2% of requests errored.” Metrics are cheap to store and great for spotting trends and triggering alerts, but they lose the detail of any individual case. They answer “how is the system doing overall, right now and over time?”

Key AI metrics you will almost always want (we expand these per pillar in later modules):

Volume: requests per minute/hour/day.
Latency: typically measured at percentiles — p50 (median), p95, p99. (More on percentiles in 2.5; they matter a lot.)
Error rate: fraction of requests that failed.
Token usage: input and output tokens over time.
Cost: dollars per hour/day, and cost per request (Module 05).
Quality scores: from evals (Module 04).
Guardrail triggers: how often inputs/outputs were blocked (Module 06).
Cache hit rate: how often you avoided a model call (Module 05).
User satisfaction: thumbs-up rate, escalation rate, retry rate.

Traces

A trace records the full journey of a single request through all the steps and components, with timing for each. If a log is a snapshot of one event and metrics are the aggregate trend, a trace is the story of one request from start to finish. It shows, for example: retrieval took 120 ms, the first model call took 2.1 s, a tool call took 300 ms, a second model call took 1.8 s, output guardrail took 50 ms.

Traces are the killer tool for AI debugging, because AI requests have so many steps (recall the journey in Module 01). They are so important they get their own module — Module 03 is a deep dive on tracing. For now, just hold the relationship: traces tie all the logs of one request together into a timed sequence.

The three pillars together: Metrics tell you something is wrong (“p95 latency spiked at 2pm”). Traces tell you which step is wrong (“retrieval is suddenly taking 8 seconds”). Logs tell you exactly what happened in a specific failing request (“this query returned 5,000 documents”). You use all three, moving from broad to specific.

2.4 What’s different about observing AI (vs. normal software)

This is the heart of the module. Normal observability tools and habits get you part of the way, but AI adds genuinely new requirements:

You must capture the content, not just the shape. Normal monitoring cares that a request succeeded and how long it took. For AI, the text of the prompt and response is the most valuable data you have — because the failure mode is usually “it returned a bad answer,” and you can’t see “bad” without the words. This makes AI observability data much larger and more sensitive.
“Success” is not binary. A request can return HTTP 200 (technical success) while the answer is wrong, biased, off-topic, or unhelpful. Your observability must capture quality signals, not just did-it-crash signals. This is why evaluation (Module 04) is tightly coupled to observability.
You need to observe the non-deterministic parts. Because the same input can yield different outputs, you often want to capture enough to understand the distribution of behavior, not just one example. You care about “how often does this go wrong,” which requires sampling and aggregating across many requests.
The interesting unit is the trace, not the log line. A single AI request is a multi-step journey (retrieval, model, tools, guardrails). Looking at one step in isolation rarely explains a failure. AI observability is trace-first.
Cost is a first-class observable. In normal software you rarely log the dollar cost of a request. In AI you must, because tokens make every request individually expensive enough to matter (Module 05).
Feedback closes the loop. Capturing user reactions (thumbs, edits, retries, escalations) turns observability into a continuous improvement engine. This human signal is gold and mostly unique to AI products.
You observe to evaluate, not just to alert. Much AI observability data exists to be scored later by evals and to build test sets, not only to fire alerts. Your logs become your test data.

2.5 A few concepts that trip people up

Percentiles (and why averages lie)

If you only look at the average latency, you can be badly misled. Imagine 99 requests take 1 second and one takes 100 seconds. The average is about 2 seconds — sounds fine! But one user waited 100 seconds and probably left. Percentiles fix this:

p50 (median): half of requests are faster than this. The “typical” experience.
p95: 95% are faster; 5% are slower. The “bad but not rare” experience.
p99: 99% are faster; 1% are slower. The “tail” — your unhappiest users.

In AI, tails are long and common (a complex agent request, a huge document, a slow provider moment). Always watch p95 and p99, not just the average. The tail is where users churn and where your worst stories come from.

Cardinality

Cardinality is how many distinct values a label can have. Tagging metrics by model_name (a handful of values) is cheap. Tagging by user_id (millions of values) can blow up your metrics storage and bill. Rule of thumb: use low-cardinality labels for metrics (model, endpoint, status), and put high-cardinality detail (user ID, full prompt text) in logs/traces, which are built to handle it.

Sampling

You may not be able to afford to store full detail for every single request at huge scale. Sampling means keeping full traces for a representative subset (say, 10%, plus 100% of errors and flagged cases). For AI, a common pattern is: keep metrics for 100% of traffic, keep full traces for a sample plus all failures/low-scores. Start by capturing everything; introduce sampling only when volume forces you to, and always over-sample the interesting cases.

2.6 Putting it together: the AI observability stack

A practical, vendor-neutral picture of how the data flows:

Your AI app
   │  emits structured events at every step (Module 01 journey)
   ▼
Instrumentation layer  ── often OpenTelemetry (Module 03) or an LLM-observability SDK
   │  standardizes and ships the data
   ▼
A collector / backend  ── stores logs, metrics, traces
   │
   ├──► Dashboards & alerts        (metrics: volume, latency, cost, errors)
   ├──► Trace explorer             (debug one request end to end — Module 03)
   ├──► Log search                 (find the exact failing case)
   └──► Eval & feedback pipeline    (score quality, build test sets — Module 04)

Concrete tool examples in this space (you do not need to learn a specific one to understand the concepts): Langfuse, LangSmith, Arize Phoenix, Helicone, Datadog, Grafana/Prometheus, OpenLLMetry. Many are built on the open standard OpenTelemetry, which we cover next module so your instrumentation isn’t locked to one vendor.

2.7 Common mistakes (learn from others)

Only logging the user’s question, not the full assembled prompt. Then you can’t tell whether a bad answer came from bad instructions, bad retrieval, or the model. Log what the model actually saw.
Logging only on errors. AI’s worst failures return “success.” You must capture successful-but-bad cases too.
No request ID tying steps together. Then you have a pile of disconnected log lines and can’t reconstruct the journey. Always carry a correlation ID through every step.
Watching averages, ignoring tails. See 2.5.
Capturing data but never looking at it. Observability is only valuable if someone reviews traces and feedback regularly. Build the habit, not just the pipeline.
Ignoring privacy. Dumping raw prompts (which may contain personal data) into a poorly secured log store is a breach waiting to happen (Module 06).

2.8 Try this

Design a log schema. For your imaginary support-assistant app, write out the exact list of fields you would record for every request, using 2.3 as a starting point. Add any app-specific fields.
Average vs. tail. Given these ten latencies (seconds): 1, 1, 1, 1, 1, 2, 2, 2, 3, 30 — compute the average and the p90. Notice how different the story is. Which one would you alert on?
Pick the pillar. For each question, decide whether logs, metrics, or traces answers it best: (a) “Is our error rate rising this week?” (b) “Why did this customer get a wrong answer at 3pm?” (c) “Which step is making requests slow today?“

2.9 Self-check

Define observability and contrast it with monitoring.
List the three pillars and what kind of question each answers.
Recite the “capture everything” list for an AI request.
Give three ways observing AI differs from observing normal software.
Explain why averages can mislead and what to watch instead.

Next: Module 03 — Tracing Deep Dive, where we zoom into the single most powerful AI debugging tool.