When to Use RAG (and When Not To)
Module 05 — When to Use RAG (and When Not To)
Knowing the mechanics isn’t expertise. Expertise is knowing when not to reach for RAG. Anyone can bolt on a vector database; the valuable architect is the one who says “you don’t need that here” and saves the team months. This module gives you the judgment.
1. Use RAG when…
- The knowledge is large, changing, or private. It can’t all fit in the prompt, and it wasn’t in the model’s training. (Company wiki, product docs, tickets, contracts.)
- You need citations / auditability. Regulated or high-stakes settings where “where did this come from?” must be answerable.
- Freshness matters. Prices, inventory, news, support tickets, code that changes daily.
- Access control matters. Different users may only be allowed to see different documents.
- Hallucination is costly and answers must be grounded in approved sources.
- The corpus is bigger than the context window — you want to pay for only the relevant slice each time.
If several of these are true, RAG is likely the right backbone.
2. Do NOT use RAG when…
This list is just as important. Reaching for RAG reflexively is a real, common, expensive mistake.
(a) It’s a behavior problem, not a knowledge problem
If the goal is style, tone, format, or a skill — that’s fine-tuning or prompting (Module 04), not RAG. RAG can’t make the model write more like your brand.
(b) The answer is reasoning or computation, not lookup
Math, planning, transforming code, logical deduction — retrieval just adds noise. The model (or a tool/agent it can call) should reason directly. Fetching documents doesn’t help solve an equation.
(c) The question is a structured query over structured data
This is the subtle, frequently-missed one. If the real question is “What was total revenue by region last quarter?”, the answer lives in a database, and the right tool is text-to-SQL (the model writes a database query) or an API/tool call — not semantic search over prose. Forcing tabular/aggregate questions through a vector search is a classic anti-pattern. RAG is for unstructured text (documents, articles, tickets), not for “sum/filter/group this table.”
(d) The whole corpus comfortably fits in the prompt and rarely changes
Modern models have big context windows. If your entire relevant knowledge is small (say it fits in well under the window) and static, just paste it into the prompt (optionally with prompt caching, Module 14). Simpler, no search system to build, no retrieval failures. Building RAG here is over-engineering. (More on this “RAG vs long-context” trade-off in Module 14.)
(e) The latency budget is tiny and the knowledge is small
Every retrieval step adds delay. If milliseconds matter and the knowledge could just be baked in, don’t add a search round-trip.
3. The architect’s three-question litmus test
Run any request through these, in order. It prevents almost all over-engineering:
- Is it a knowledge problem or a behavior problem?
- Behavior → fine-tune / prompt. Stop.
- Knowledge → continue.
- Is the knowledge structured or unstructured?
- Structured (tables, records, aggregations) → SQL / API / tool. Stop.
- Unstructured (documents, prose, semantic meaning) → continue.
- Does it fit in the prompt and stay static?
- Yes → just prompt it (+ caching). Stop.
- No (large / changing / needs citations / access control) → RAG. ✅
Only the path knowledge → unstructured → too big or changing genuinely needs RAG. That filter alone is most of the value you bring as an advisor.
4. Mixed systems are normal
Real products rarely use just one tool. A mature assistant might:
- Use RAG for product knowledge,
- Call a SQL tool for “how many orders today,”
- Be lightly fine-tuned for output format,
- Rely on prompting for guardrails,
- And use an agent (Module 11) to decide which of these to invoke per question.
Being able to decompose a request into “this part is RAG, this part is SQL, this part is fine-tuning” is exactly what solution architects get paid for.
5. Quick scenarios — test your judgment
| Request | Right approach | Why |
|---|---|---|
| ”Answer questions about our 500-page employee handbook.” | RAG | Large, unstructured, updates over time. |
”Always output support replies as {issue, steps, eta} JSON.” | Fine-tune / prompt | Behavior/format, not knowledge. |
| ”What were our top 5 products by sales in May?” | SQL / tool | Structured, aggregate query. |
| ”Summarize this one 4-page memo I’m pasting.” | Just prompt it | Small + already in hand; no retrieval needed. |
| ”Chatbot over our constantly-updating help center with citations.” | RAG | Changing, unstructured, needs sources. |
| ”Solve this logic puzzle / write this algorithm.” | Reasoning / tools | Computation, not lookup. |
Check yourself
- Recite the three-question litmus test from memory.
- Why is “total sales by region last quarter” not a RAG problem?
- When does pasting everything into the prompt beat building RAG?
- Give an example of a single product that needs RAG and SQL and prompting.