Module 8 — Building RAG End to End

Time to build the thing everyone actually wants: a system that answers questions from your documents, using a vector database for retrieval and an LLM for the answer. This is RAG — Retrieval-Augmented Generation — the #1 reason vector databases exploded.

Why RAG exists (the problem it solves)

LLMs are smart but have three weaknesses:

1. They don’t know your private data (your wiki, your contracts, your tickets).
2. They go stale — frozen at their training cutoff.
3. They hallucinate — confidently invent facts.

You could retrain or fine-tune the model on your data, but that’s expensive, slow, and has to be redone whenever data changes. RAG is the cheaper, faster fix: keep the model as-is, and at question time, retrieve the relevant facts from your documents and paste them into the prompt. The model then answers using facts in front of it, with citations, far less likely to make things up.

The analogy: A fine-tuned model is a student who memorized a textbook (expensive, fixed). RAG is an open-book exam — the student looks up the right page right now (cheap, always current). Vector search is how it finds the right page.

The architecture in one picture

INDEXING (offline, done once / on update):
  Documents → Chunk → Embed each chunk → Store vectors + text + metadata in vector DB

QUERYING (online, per question):
  User question → Embed question → Vector DB finds top-k similar chunks
       → (optional) re-rank chunks → Build prompt (question + chunks)
       → LLM generates answer grounded in those chunks → return answer + citations

Two pipelines: indexing (slow, offline, builds the knowledge base) and querying (fast, online, answers questions). Keep them mentally separate.

Step 1 — Load and chunk your documents

Chunking quality (Module 2) drives retrieval quality more than almost anything else.

def chunk_text(text, chunk_size=500, overlap=80):
    """Split into ~chunk_size-char pieces with overlap, preferring paragraph breaks."""
    paras = [p.strip() for p in text.split("\n\n") if p.strip()]
    chunks, current = [], ""
    for p in paras:
        if len(current) + len(p) <= chunk_size:
            current += ("\n\n" + p) if current else p
        else:
            if current:
                chunks.append(current)
            # carry overlap from the tail of the previous chunk for continuity
            current = (current[-overlap:] + "\n\n" + p) if current else p
    if current:
        chunks.append(current)
    return chunks

Best practices: split on natural boundaries (paragraphs, headings, sentences, code blocks), keep ~10–20% overlap, and prepend context — e.g., put the document title/section at the top of each chunk so “It grew 20%” still knows what “it” is. Store useful metadata with every chunk: source URL, title, section, date, author, access permissions.

Step 2 — Embed the chunks

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("BAAI/bge-small-en-v1.5")   # 384-dim, fast, good

def embed(texts, is_query=False):
    # BGE wants a query instruction prefix; passages get none
    if is_query:
        texts = ["Represent this sentence for searching relevant passages: " + t for t in texts]
    return model.encode(texts, normalize_embeddings=True)

Note the asymmetric prefix for queries — skipping it quietly hurts quality (Module 2). Batch your encode calls when embedding many chunks; it’s far faster and cheaper.

Step 3 — Store in a vector database

Using Chroma for a minimal, runnable example (swap for Qdrant/pgvector in production):

import chromadb
client = chromadb.Client()
col = client.get_or_create_collection("kb", metadata={"hnsw:space": "cosine"})

def index_document(doc_id, title, url, text):
    chunks = chunk_text(text)
    contextualized = [f"{title}\n\n{c}" for c in chunks]   # prepend title for context
    vectors = embed(contextualized)
    col.add(
        ids=[f"{doc_id}:{i}" for i in range(len(chunks))],
        documents=chunks,
        embeddings=[v.tolist() for v in vectors],
        metadatas=[{"doc_id": doc_id, "title": title, "url": url, "chunk": i}
                   for i in range(len(chunks))],
    )

Step 4 — Retrieve

def retrieve(question, k=5, where=None):
    qv = embed([question], is_query=True)[0].tolist()
    res = col.query(query_embeddings=[qv], n_results=k, where=where)  # where = metadata filter
    return [
        {"text": d, "meta": m, "score": 1 - dist}     # convert cosine distance to similarity
        for d, m, dist in zip(res["documents"][0], res["metadatas"][0], res["distances"][0])
    ]

The optional where is a metadata filter — “only this user’s docs,” “only 2025.” Module 9 goes deep on filtering and hybrid search.

Step 5 — Build the prompt and generate

The retrieved chunks become context. Two rules matter enormously: (a) instruct the model to answer only from the context, and (b) ask it to cite sources and say “I don’t know” when the context lacks the answer. These two instructions are most of what keeps RAG honest.

def build_prompt(question, chunks):
    context = "\n\n".join(
        f"[{i+1}] (source: {c['meta']['title']})\n{c['text']}"
        for i, c in enumerate(chunks)
    )
    return f"""Answer the question using ONLY the context below.
If the context does not contain the answer, say "I don't have enough information."
Cite sources with their bracket numbers like [1], [2].

Context:
{context}

Question: {question}

Answer:"""

# Then send build_prompt(...) to any LLM (Claude, GPT, a local model) and return its reply.

That’s a complete RAG system: chunk → embed → store → retrieve → prompt → generate, with citations.

The retrieval-quality ladder (how beginners become experts)

A naive RAG works; a great RAG adds these, roughly in order of payoff:

1. Better chunking — biggest lever. Semantic/structure-aware splitting, right size, overlap, context prefixes.
2. Better embedding model — try a few; measure on your data (Module 11). Domain-matched models win.
3. Hybrid search — combine vector + keyword (BM25) so exact terms, names, and codes aren’t missed (Module 9). Huge, reliable gains.
4. Metadata filtering — scope to the right docs (tenant, recency, permissions) before/while searching (Module 9).
5. Re-ranking — retrieve top ~50 cheaply, then a cross-encoder reranker re-scores them for precision; feed the top 5–8 to the LLM (Module 12). Often the single biggest quality jump after hybrid.
6. Query transformation — rewrite vague questions, expand with synonyms, or generate multiple sub-queries (multi-query, HyDE) before retrieving.
7. Right k and context budget — too few chunks miss facts; too many add noise and cost and can bury the answer (“lost in the middle”). Tune it.
8. Citations + guardrails — force grounding, detect when retrieval found nothing relevant, refuse gracefully.

Common RAG failure modes (and the fix)

Symptom	Likely cause	Fix
Answers miss obvious facts	Bad chunking; missing exact-term match	Improve chunking; add hybrid/keyword search
Right doc retrieved, wrong answer	Too many/noisy chunks; weak prompt	Re-rank; fewer, better chunks; stricter prompt
Hallucinations persist	Model not constrained to context	”Answer only from context; cite; say IDK”
Good in tests, bad in prod	Eval not on real queries	Build a real eval set (Module 11)
Misses recent info	Stale index	Re-index on document changes
Leaks other users’ data	No tenant filter	Enforce metadata/permission filters (Module 9)
Vague queries fail	Under-specified question	Query rewriting/expansion

Where the vector database fits in the bigger system

RAG is one application on top of a vector database. The database’s job is steps 3–4 (store + retrieve) reliably, fast, filtered, and at scale. Everything else — chunking, embedding choice, prompting, reranking, evaluation — lives in your pipeline. Mastering both halves is what makes you an expert, not just a user.

Hands-on challenge

Build a mini “chat with your docs”: index 10–20 markdown files from a folder, then loop on input() questions through retrieve() + build_prompt() + your LLM of choice. Then measure it (Module 11) and improve one rung of the quality ladder at a time. You’ll learn more from improving a real RAG than from any reading.

Gotchas

• Treating RAG as “just embeddings.” Retrieval quality is a system property: chunking + model + hybrid + reranking + prompt all compound.
• No evaluation. Without a test set you’re tuning blind; every “improvement” is a guess (Module 11).
• Stuffing the prompt with 50 chunks. More context isn’t better — it’s noisier, slower, costlier, and important facts get “lost in the middle.”
• Forgetting permissions. In multi-user systems, a missing filter is a data breach, not just a bad result.
• Never re-indexing. Documents change; a stale index gives confidently outdated answers.
• Ignoring “I don’t know.” A RAG that always answers will confidently answer wrongly when retrieval fails. Let it abstain.

Check yourself

1. Why is RAG usually preferable to fine-tuning for “answer from my documents”?
2. What are the two separate pipelines in a RAG system?
3. Name the top three levers for improving RAG retrieval quality.
4. Two prompt instructions do most of the work in reducing hallucination — what are they?

Answers: (1) It’s cheaper, faster, and always current — you retrieve facts at query time instead of retraining whenever data changes, and it enables citations. (2) Indexing (offline: chunk → embed → store) and querying (online: embed question → retrieve → prompt → generate). (3) Better chunking, hybrid (vector + keyword) search, and re-ranking (plus a domain-matched embedding model). (4) “Answer using only the provided context (say ‘I don’t know’ if it’s not there)” and “cite your sources.”

➡️ Next: 09-hybrid-search-and-filtering.md — combining keywords, vectors, and metadata.