Module 09 — Reliability & Performance

Goal of this module: Learn to make AI systems dependable and fast under real-world load. Models are slow, flaky, and run on someone else’s servers — so reliability is a design problem you must engineer for. You’ll learn latency optimization, retries, timeouts, fallbacks, rate limits, graceful degradation, and how to handle the fact that your most important dependency is outside your control. This is classic production engineering, adapted for AI’s quirks.

9.1 Why AI reliability is hard

In Module 00 we said “the model is a dependency you don’t control.” This module is about living with that reality. AI systems are harder to keep reliable than normal software for specific reasons:

The model is slow. Responses take seconds, not milliseconds, and output time grows with answer length. This drags down everything built on it.
The model is variable. The same request can take 1 second or 20 seconds depending on load, length, and the provider’s moment-to-moment capacity. Tail latency (Module 02) is long and routine.
The provider can fail or throttle you. External providers have outages, rate limits, and capacity limits. When they’re down or slow, you’re down or slow — unless you’ve designed for it.
Agents compound everything. A multi-step agent (Module 01) chains many slow, flaky calls, so its total latency and failure probability are the sum/product of all the steps. More steps = slower and more fragile.
Demand spikes. A viral moment or a batch job can flood your system and blow past provider rate limits.

The core principle: Treat the model provider like any unreliable external dependency — assume it will be slow, throttle you, and go down, and design so your system degrades gracefully instead of collapsing. Hope is not a strategy; fallbacks are.

9.2 Measuring reliability and performance

You can’t improve what you don’t measure (Module 02). The key reliability/performance metrics:

Latency, at percentiles — p50, p95, p99 (not averages! recall Module 02). Plus time to first token for streaming (Module 08), which is what users actually feel.
Availability / uptime: the fraction of requests that succeed. Often framed as an SLO (Service Level Objective) — a target you commit to, like “99.9% of requests succeed” or “p95 latency under 3 seconds.”
Error rate, broken down by type: provider errors, timeouts, rate-limit rejections, guardrail blocks, your own bugs. The breakdown tells you what to fix.
Throughput: requests per second you can handle.
Rate-limit headroom: how close you are to your provider’s limits.
Fallback/retry rates: how often you’re relying on your safety nets (a rising rate is an early warning).

Define SLOs explicitly. Decide what “reliable enough” means for each feature (a real-time chat needs low latency; an overnight batch job doesn’t), write it down as an SLO, and alert when you’re at risk of breaching it. Without a target, “reliable” is just an opinion.

9.3 The reliability toolkit

These are the standard mechanisms for surviving an unreliable dependency. Layer them — defense in depth, like security (Module 06).

Timeouts

Never wait forever. Set a maximum time for each model/tool call. If it exceeds it, give up and trigger a fallback. Without timeouts, one slow provider moment can hang requests indefinitely and exhaust your resources.

Retries (with care)

If a call fails transiently (a momentary network blip, a temporary provider error), retry it. But retry intelligently:

Exponential backoff: wait a bit, then longer, then longer still between retries — don’t hammer a struggling provider, which makes things worse.
Add jitter (randomness) so all your retries don’t synchronize into a thundering herd.
Only retry retryable errors. Don’t retry a “bad request” or a content-policy refusal — that’ll just fail again and waste money. Retry timeouts and transient server errors.
Cap retries. Infinite retries amplify cost (Module 05) and latency. Limit them.
Beware double-charging and double-acting. A retried request still costs tokens, and if the first attempt did take an action (sent an email) before failing, retrying could do it twice. Use idempotency where actions are involved.

Fallbacks (the most important AI reliability pattern)

When the primary path fails, have a fallback — a worse-but-working alternative. Options, roughly in order:

Fall back to another model (a different provider, or a smaller/cheaper model) when the primary is down or throttled. Multi-provider setups are a common resilience strategy.
Fall back to a cached or default response for common queries.
Fall back to a simpler, non-AI path — a search results page, a templated answer, an FAQ.
Fall back to a human (Module 07) — “I’m having trouble; let me connect you to someone.”

The goal: the user gets something useful even when the ideal path is broken. A degraded answer beats an error page. Design every critical AI path with at least one fallback.

Rate limiting (yours and theirs)

Provider rate limits: providers cap how many requests/tokens you can send per minute. Exceed them and you get rejected. Manage this with request queuing, smoothing traffic, and spreading load across providers/keys.
Your own rate limits: cap how much any single user can consume, to protect availability for everyone and prevent abuse and runaway cost (this is the same lever as Module 05’s quotas and Module 06’s abuse prevention — one mechanism, three benefits).

Circuit breakers

If a dependency is clearly failing (lots of errors in a row), a circuit breaker “trips” and stops sending requests to it for a while, immediately using the fallback instead. This prevents you from piling requests onto a dead service and lets it recover. Once it’s healthy, the breaker closes again. Borrowed straight from classic distributed-systems engineering.

Load shedding and queuing

Under extreme load, it’s better to shed some load (politely reject or queue low-priority requests) than to let the whole system collapse. Prioritize the important traffic. For non-real-time work, queue it and process when capacity allows (also a cost win — Module 05’s batching).

Graceful degradation

The umbrella principle tying these together: when things go wrong, the system should degrade — get slower, simpler, or fall back — rather than fail outright. A slightly worse experience under stress is a success; a crashed experience is a failure.

9.4 Latency optimization (making it fast)

Reliability is “does it work”; performance is “is it fast enough.” Levers to reduce latency (many overlap with cost — Module 05 — because both come from doing less work):

Streaming (Module 08): doesn’t reduce total time but slashes perceived latency by showing output immediately. The highest-leverage, easiest win. Optimize time to first token.
Use smaller/faster models where quality allows (the routing lever from Module 05 helps latency too).
Shorten prompts and outputs. Less input to process and less output to generate = faster (and cheaper). Trim context; cap output length.
Parallelize independent work. If you need three retrievals or several independent model calls, do them at the same time, not one after another. This can collapse a long sequential trace (Module 03) into a much shorter one.
Cache (Module 05): a cached answer is near-instant and free. The best request is the one you don’t have to make.
Reduce agent steps. Fewer loop iterations = lower latency and cost. Design tasks to finish efficiently; cap iterations.
Pre-compute / pre-fetch where you can anticipate what’s needed.

Use your traces (Module 03) to find the latency. Open a slow trace, find the longest span, fix that. Don’t optimize by guessing — the trace shows you exactly where the seconds go.

9.5 Scaling and capacity

As traffic grows:

Plan for provider limits. Know your rate limits and request increases before you need them. Spread load across multiple providers/keys/regions for both capacity and resilience.
Smooth spiky traffic with queues so you don’t slam into limits during bursts.
Right-size with routing (Module 05): handling easy traffic on small models frees flagship-model capacity for the hard cases and reduces the chance of hitting limits.
Watch unit costs as you scale (Module 05): make sure cost-per-request stays sustainable as volume grows — scaling a money-losing feature just loses money faster.

9.6 How reliability connects to the other pillars

Observability (02–03): you can’t manage reliability without latency/error metrics and traces. Reliability lives on the data from Modules 02–03.
Cost (05): retries, fallbacks, and model size all affect cost; rate limits protect both availability and the bill. Many latency and cost levers are the same lever.
Security (06): rate limiting and load shedding defend against denial-of-service and abuse; circuit breakers contain cascading failures.
Human-in-the-loop (07): “fall back to a human” is both a reliability fallback and an escalation path.
UX (08): streaming, progress indicators, honest failure messages, and escape hatches are how reliability feels to the user. Graceful degradation is a UX design as much as an engineering one.
Evaluation (04): evaluate that your fallbacks still produce acceptable answers — a degraded path that gives wrong answers isn’t graceful, it’s just a quieter failure.

9.7 Common reliability mistakes

No timeouts, letting slow calls hang indefinitely.
Naive retries without backoff/jitter/caps, amplifying load, cost, and (for actions) double-execution.
No fallback, so any provider hiccup becomes a user-facing outage.
Single provider, single point of failure — when they go down, you go down.
Watching averages, missing the tail — p99 users suffer silently (Module 02).
No rate limiting, so one abuser or spike degrades everyone.
Optimizing real latency but ignoring perceived latency — a silent 4-second wait feels worse than a streamed 8-second one.
Untested fallbacks that turn out to be broken exactly when you need them. Test your failure paths deliberately (chaos testing).

9.8 Try this

Set SLOs. For your support assistant, write an availability SLO and a p95 latency SLO. Justify the numbers based on how users use it.
Design the fallback chain. The assistant’s primary model provider goes down. Write the full fallback chain (9.3) — what happens first, second, third — so the user still gets something useful. Where does a human come in?
Find the latency. Given a trace where three retrievals run one after another (0.3s each) before a 2s model call, total ~2.9s — what change cuts the most time, and what’s the new total?

9.9 Self-check

Why is the model provider a uniquely unreliable dependency, and what principle follows from that?
What is an SLO, and why define one per feature?
Explain timeouts, smart retries (backoff/jitter/caps), fallbacks, and circuit breakers — and when each fires.
Why is streaming the highest-leverage latency win even though it doesn’t reduce total time?
Give three ways reliability levers overlap with cost levers.
What is graceful degradation, and why is an untested fallback dangerous?

Next: Module 10 — Deployment & Lifecycle. We can run it well; now we learn to change it safely over time.