Module 05 — Cost Management

Goal of this module: Learn how AI systems cost money, why costs can explode silently, and the full toolkit for controlling them — without sacrificing quality. You’ll understand tokens and pricing deeply, learn every major optimization lever (caching, routing, prompt trimming, model selection), and how to budget, attribute, and alert on spend. This is “FinOps for AI,” and it’s a first-class engineering responsibility.

5.1 Why cost is an engineering problem in AI

In traditional software, the engineer writes code and someone else worries about the server bill. In AI, the engineer’s design choices directly and immediately determine cost, often dramatically. A single decision — “let’s include the whole document in every prompt” or “let the agent loop until it’s confident” — can multiply your bill by 10x. Cost is not a finance problem bolted on later; it’s a property of your architecture.

Three facts make AI cost uniquely tricky:

You pay per token, on every single request. Unlike a fixed server you’ve already paid for, each request has a real, variable marginal cost. More usage = proportionally more money, forever.
Costs are invisible until you measure them. A request that costs $0.03 feels free. A million of them is $30,000. The danger is the silent accumulation. (This is why Module 02’s cost-per-request logging exists.)
Quality and cost trade off constantly. Bigger models, longer context, more agent steps, and more retries all improve quality and raise cost. Cost management is really cost-quality optimization — and you need evals (Module 04) to do it safely.

5.2 Tokens, deeply

Everything starts with tokens. Recall from Module 00: a token is the unit models read and write, roughly ¾ of a word in English. Now the production details:

You pay for input tokens AND output tokens, usually at different prices. Output tokens are typically more expensive than input tokens (often several times more), because generating is more costly than reading. So a chatty model that writes long answers costs more than the token count alone suggests.
The whole prompt counts as input every time. Your system instructions, all the retrieved documents, the entire conversation history, and the user’s message — all of it is input tokens on every single call. In a long conversation or a big RAG context, the user’s actual question might be 20 tokens while the surrounding context is 5,000. You pay for all 5,020, every turn.
Conversation history grows. In a naive chatbot, each new turn re-sends the entire history. Turn 1 sends 100 tokens; turn 20 might re-send 10,000. Cost per message grows over the conversation unless you manage history.
Agents multiply everything. Each loop iteration is a fresh model call that re-sends the accumulated context. A 10-step agent can cost 10x+ a single call, because the context grows with each step and each step pays for all of it.

The number-one cost lesson: In most production AI apps, the biggest cost driver is how much you put into the context window, multiplied by how many times you call the model. Control those two and you control most of your bill. Almost every optimization below is a way to reduce one or the other.

Providers price models per million tokens (e.g., “$X per million input tokens, $Y per million output tokens”). The crucial structure to understand:

There is a menu of models at very different price points. A flagship model might cost 10–30x more than a small “mini”/“flash”/“haiku”-class model. The expensive one is smarter; the cheap one is faster and far cheaper.
Most requests don’t need the most expensive model. A huge cost win comes from using the right-sized model for each job (see routing, 5.5).
Some token types are discounted. Many providers offer prompt caching (re-using a previously processed prompt prefix at a steep discount) and batch processing (non-urgent requests at a discount). These can cut costs 50–90% for the right workloads.
Self-hosting open-weight models changes the economics entirely: you pay for compute (GPUs) instead of per token. This can be cheaper at very high volume but adds operational burden. It’s a real lever at scale, not a beginner’s first move.

Expert habit: Know the current price of the models you use, and recompute your blended cost per request whenever prices or usage change. Provider prices move frequently — treat them as live numbers, not memorized constants.

5.4 Measuring cost: you can’t manage what you can’t see

Before optimizing, instrument. From Module 02, every request log/trace should carry input tokens, output tokens, and dollar cost. On top of that, build:

Cost per request, tracked over time (a key metric).
Cost attribution: break spend down by feature, customer/tenant, model, and endpoint. This answers “what is expensive?” and “who is driving cost?” — essential for prioritizing and for charging customers fairly.
Cost dashboards and alerts: a daily/weekly spend trend, plus alerts when cost-per-request or total spend jumps. You want to learn about a 5x cost spike from an alert, not from a finance email next month.
Unit economics: cost per user, per conversation, per resolved ticket, per dollar of revenue. This tells you whether the feature is sustainable, not just what it costs.

A useful framing borrowed from cloud FinOps: make cost visible, make someone accountable, and make optimization continuous. The same applies directly to AI.

5.5 The optimization toolkit (the levers)

Here is the full menu of ways to cut AI cost while protecting quality. Apply them in roughly this order — measure with evals (Module 04) after each to confirm quality held.

Lever 1 — Caching

If the same (or similar) request comes in repeatedly, don’t pay to answer it again.

Exact-match caching: identical prompt → return the stored answer. Free and instant. Great for FAQs and repeated queries.
Semantic caching: similar prompts (by meaning, using embeddings) → reuse a prior answer. More coverage, but you must tune the similarity threshold so you don’t return a near-miss answer.
Prompt caching (provider feature): the provider caches the processed prefix of your prompt (e.g., your big system instructions and static context) so you pay a fraction for it on repeat calls. Structure your prompts with the stable parts first to benefit. This is huge for apps with large fixed instructions.

Lever 2 — Model routing / right-sizing

Don’t send every request to the flagship model. Route each request to the cheapest model that can handle it:

Simple/classification tasks → small cheap model.
Hard/nuanced tasks → big expensive model.
A common pattern: a cascade — try the cheap model first, and only escalate to the expensive one if a check says the cheap answer isn’t good enough. You pay the premium only when you need it.

This single lever often cuts costs by more than half, because most traffic is easy. (Use evals to verify the cheap model is actually good enough per task type.)

Lever 3 — Prompt and context trimming

Every token in the context is paid for on every call, so trim ruthlessly:

Tighten system instructions. Long, rambling instructions cost money on every request. Make them concise.
Retrieve fewer, better documents. Returning 20 chunks “to be safe” is expensive and often hurts quality (the model gets distracted). Retrieve the few most relevant; improving retrieval (Module 04) saves cost and improves answers.
Manage conversation history. Don’t re-send the entire history forever. Options: keep the last N turns, or summarize older history into a short synopsis, or drop irrelevant turns.
Cap output length. Set sensible max_tokens; ask for concise answers when long ones aren’t needed (output tokens are the pricey ones).

Lever 4 — Limit agent steps

For agents, cap the number of loop iterations and design tasks so the agent finishes efficiently. A runaway agent (Module 03) is both a reliability bug and a cost bug. Set a hard ceiling and alert when agents routinely hit it.

Lever 5 — Batching and async

For work that isn’t real-time (overnight processing, bulk enrichment), use batch APIs or off-peak processing for steep discounts. Don’t pay real-time prices for non-real-time work.

Lever 6 — Fine-tuning / smaller specialized models (advanced)

At scale, you can sometimes fine-tune a small model to do one job as well as a big general model does it, then run the small model cheaply. This trades upfront effort for ongoing savings — a high-volume optimization, not a first step.

Lever 7 — A gateway / proxy

Route all model calls through a single gateway (Module 01). A gateway is the natural place to enforce caching, routing, rate limits, budgets, and cost logging in one spot instead of scattered through your code. It turns many of the levers above into config rather than custom code.

5.6 Budgets, quotas, and runaway protection

Optimization reduces cost; guardrails prevent catastrophe. You must protect against the nightmare scenarios: a bug that loops forever, a viral spike, or an abuser hammering your endpoint and running up a five-figure bill overnight.

Per-user / per-tenant rate limits and quotas. Cap how much any single user can consume (ties to Module 09’s rate limiting and Module 06’s abuse prevention).
Spending caps and circuit breakers. Set hard budget limits that throttle or halt when exceeded, with alerts. Better to degrade gracefully than to get a surprise bill.
Anomaly alerts. Alert on sudden jumps in cost-per-request, total spend, tokens-per-request, or agent step counts. These usually signal a bug or an attack.
Provider-side budget alerts. Most providers let you set billing alerts — use them as a backstop.

A real and common failure: an agent enters an infinite tool-calling loop and burns thousands of dollars before anyone notices. The fix is layered: step caps (5.5), cost-per-request alerts (5.4), and circuit breakers (here). Defense in depth.

5.7 The cost-quality-latency triangle

You’re always balancing three things, and they pull against each other:

Cost (cheaper is better)
Quality (better answers)
Latency (faster responses — Module 09)

Bigger models and more context/steps improve quality but raise cost and latency. Caching and routing cut cost and latency but require care to protect quality. You cannot maximize all three; you choose the right balance per use case. A throwaway autocomplete suggestion should be cheap and fast even if slightly worse; a legal-document analysis should prioritize quality almost regardless of cost. There is no universal right answer — there’s the right answer for each feature, found by measuring (Module 04) and deciding deliberately.

5.8 Common cost mistakes

Not measuring cost per request at all. You can’t manage the invisible. Start here.
Using the flagship model for everything. Most traffic is easy; route it cheaper.
Stuffing the context “to be safe.” More context costs more and often lowers quality. Less, better-targeted context usually wins twice.
Re-sending the whole conversation forever. Manage/summarize history.
No caps on agents or users. One bug or abuser and the bill explodes.
Optimizing cost without re-running evals. A cheaper setup that quietly tanks quality is a false saving. Always re-measure quality after a cost change.
No cost attribution. If you can’t see which feature/customer drives spend, you can’t prioritize fixes or price your product correctly.

5.9 Try this

Do the token math. A flagship model costs (say) $3 per million input tokens and $15 per million output tokens. Your average request sends 4,000 input tokens and gets 500 output tokens. What does one request cost? What do 1,000,000 requests/month cost? Now redo it assuming a small model at $0.15 input / $0.60 output. How much would routing 80% of traffic to the small model save?
Hunt the waste. For your support assistant, list every place context tokens accumulate (instructions, retrieved docs, history, tool results). For each, name a trimming lever from 5.5.
Design the guardrails. Write the three cost-protection mechanisms you’d put in place before launch to prevent a runaway-bill incident (5.6).

5.10 Self-check

Why is cost an engineering concern rather than only a finance concern in AI?
Explain why output tokens often cost more than input tokens, and why a long conversation gets more expensive per turn.
Name the seven optimization levers and what each reduces (context size vs. number of calls).
What is model routing / cascading and why does it usually save so much?
Describe the cost-quality-latency triangle with an example of choosing differently for two features.
Why must every cost optimization be paired with an eval check?

Next: Module 06 — Security. We’ve made it observable, good, and affordable; now we make it safe.