Applied AI / Retrieval-Augmented Generation (RAG) / Module 15 · Architect Decision Frameworks
Module 15

Architect Decision Frameworks

Retrieval & Knowledge Retrieval-Augmented Generation (RAG)

Module 15 — Architect Decision Frameworks

This module is your quick-reference toolkit — the cheat sheets you’ll actually use in meetings. Everything here is distilled from the earlier modules into decision flows you can recall under pressure. When a solution architect asks “should we use RAG, and how should we build it?”, these are your answers.

A. The problem-type router (where every request starts)

Run any request through this first. It tells you whether RAG is even the right tool.

What kind of problem is this?

  Behavior / format / tone / skill?       → FINE-TUNE (+ prompt)
  Fact lookup over unstructured text?      → RAG
  Query over structured / tabular data?    → TEXT-TO-SQL / API / tool call
  Pure reasoning / math / planning?        → TOOLS / AGENT (minimal retrieval)
  Small + static knowledge?                → LONG-CONTEXT prompt (+ caching)

The three-question version (Module 05), in order:

  1. Knowledge or behavior? Behavior → fine-tune/prompt.
  2. Structured or unstructured? Structured → SQL/tool.
  3. Fits in context and static? Yes → just prompt it. → Only knowledge + unstructured + large/changing actually needs RAG.

B. The build-up order (don’t start fancy)

Build the simple version first; add complexity only when evaluation demands it.

BASELINE (build this first):
  good parsing → recursive chunking + overlap → solid embedding model →
  vector store → top-K retrieval → grounded prompt → GOLDEN EVAL SET

THEN add, roughly in order of typical payoff, one at a time:
  hybrid search (dense + BM25)
  reranker
  metadata filtering
  query rewriting
  parent-document / small-to-big retrieval
  multi-query / HyDE
  agentic / GraphRAG / RAPTOR   ← only for genuine multi-hop or global needs

Every addition must be justified by a measured gap on your eval set, never by “it sounds impressive.”

C. The “my RAG answers are bad” debugging ladder

The most useful framework you’ll own. (From Module 12’s triage rule.)

1. Measure Recall@K on your golden set.

   ├─ LOW recall  → fix RETRIEVAL, working upstream:
   │     parsing quality
   │     → chunking (size / overlap / structure)
   │     → hybrid search
   │     → embedding model choice
   │     → metadata filters
   │     → increase K

   └─ HIGH recall but bad answers → fix GENERATION:
         add / improve reranker
         → grounding prompt ("use only context; say you don't know")
         → context order (avoid "lost in the middle")
         → reduce distractor chunks
         → require citations
         → add an "I don't know" path

2. Change ONE thing → re-measure → keep only if it helped → repeat.

D. Component-selection drivers

What to weigh when picking each piece:

Embedding model: domain fit, language(s), max input length (must fit your chunks!), vector size, query/document encoding convention. → Validate on your data, not just leaderboards.

Vector store: scale (thousands vs billions), metadata-filtering support, hybrid-search support, operational burden, existing stack. → Default to pgvector if you already run Postgres; adopt a specialized store when scale or features demand it.

Reranker: is the precision gain worth the latency? → Usually yes.

Chunk size & K: these are hyperparameters — tune them on the eval set, never guess a “correct” value.

Advanced pattern (Module 11): match the specific gap (e.g. multi-hop → decomposition or GraphRAG; global summary → RAPTOR), and weigh the added latency/cost/maintenance.

E. RAG vs Long-Context quick call

Small + static + low volume         → long context (+ caching)
Large + changing + high volume      → RAG
Need citations / access control      → RAG
Want simplicity, corpus is tiny      → long context
Best of both                         → RAG to retrieve generously, big window to hold it

F. Production readiness checklist (from Module 14)

  • Golden eval set + current retrieval & generation scores
  • Incremental indexing (add / update / delete)
  • Access control enforced at retrieval
  • Prompt-injection consideration for retrieved content
  • Ability to delete specific data on request (compliance)
  • Caching of stable prompts and/or frequent queries
  • Monitoring of p95 latency and cost per query
  • Logging, user feedback, and failures recycled into evaluation

G. What “expert in the room” sounds like

When you’re advising, these are the moves that signal real expertise:

  • Opens with “Is this a knowledge problem or a behavior problem?” before anyone mentions a vector database.
  • Pushes back on fine-tuning-for-facts and on RAG-for-structured-data.
  • Insists on an eval set before any optimization.
  • Treats parsing and chunking as where quality is won or lost.
  • Defaults to hybrid + reranker, adds complexity only when measured.
  • Treats access control, prompt injection, freshness, and cost as first-class.
  • Has a clear, current RAG-vs-long-context stance.
  • Is willing to say “you don’t need RAG here” — and explain why.

That last one — knowing when not to use RAG — is the single clearest marker of expertise.

Check yourself

  • From memory, recite the problem-type router.
  • A client’s answers are bad and Recall@K is low — what do you investigate, in what order?
  • What’s your one-line answer to “why not just use a huge context window instead of RAG?”
  • Which two things are “hyperparameters you tune on the eval set, never guess”?

Next: Module 16 — Learning Roadmap & Capstone →

← Module 14: Production Concerns