RAG vs Training vs Fine-Tuning vs Prompting
Module 04 — RAG vs Training vs Fine-Tuning vs Prompting
This is the module you’ll quote out loud to colleagues and executives more than any other. People constantly mix these four things up, choose the wrong one, and waste months. By the end you’ll be able to pick correctly in seconds.
First, let’s define the four tools — using only ideas from earlier modules (weights, training, tokens, prompt).
1. The four ways to influence what an AI does
(a) Pre-training
Pre-training is the original, massive process from Module 01: setting all the billions of weights from scratch by showing the model trillions of tokens. It creates the base model.
- Costs: millions of dollars, weeks/months, huge compute.
- Who does it: a handful of big labs. You almost certainly never do this.
- Changes: everything — this is the model’s general intelligence and broad knowledge.
(b) Fine-tuning
Fine-tuning takes an already-trained model and nudges some of its weights a bit further by training it on a smaller, focused set of examples. Think “specialization course” after the model’s “general education.”
- A common efficient version is LoRA (you don’t need the math — just know it’s a cheap way to fine-tune by adjusting a small add-on set of weights instead of all of them).
- Costs: modest — hundreds to thousands of dollars, hours to days.
- Changes: the model’s behavior, style, and format — how it acts.
(c) RAG (Retrieval-Augmented Generation)
RAG changes nothing in the model. It changes the prompt — by fetching relevant documents and inserting them, as we saw in Module 03.
- Costs: building a search system; very cheap to update (edit a document).
- Changes: what the model knows for this question — the facts in front of it.
(d) Prompt engineering
Prompt engineering is simply writing the instructions/question well — giving examples, telling it the format you want, asking it to think step by step. No training, no documents fetched; just careful wording.
- Costs: free.
- Changes: how the model responds within a single request, using what it already has.
2. The single most important distinction
Burn this into memory:
Fine-tuning changes how the model BEHAVES. RAG changes what the model KNOWS. Prompting nudges both, within one request.
- Want the model to know a fact (a policy, a spec, a customer record, yesterday’s data)? → RAG.
- Want the model to adopt a behavior (always reply in this JSON shape, speak in our brand voice, follow our classification taxonomy, use a niche reasoning style)? → Fine-tuning.
- Want a quick adjustment for one task? → Prompting.
- Want both knowledge and behavior? → Fine-tune for behavior + RAG for knowledge. They work together; they are not rivals.
3. Side-by-side comparison
| Pre-training | Fine-tuning | RAG | Prompting | |
|---|---|---|---|---|
| What it changes | All weights | Some weights | The prompt (adds documents) | The prompt (wording only) |
| Good for | Building a base model | Behavior, style, format | Knowledge, facts | Quick steering |
| Update a fact | Retrain (∞ effort) | Retrain the tune | Edit a document (seconds) | N/A |
| Add today’s data | No | Painful, unreliable | Yes, instantly | No |
| Can cite sources | No | No | Yes | No |
| Handles private docs | No | Sort of, badly | Yes | Only if pasted |
| Typical cost | $Millions | $$–$$$ | $ (build once) | Free |
| Who does it | Big labs | You, sometimes | You, often | Everyone |
4. Why fine-tuning is the WRONG tool for facts (say this a lot)
The #1 RAG misconception: “Let’s fine-tune the model on our documents so it learns them.” It sounds reasonable and it mostly fails. Here’s the honest why, in beginner terms:
- Facts get diluted. A fact appearing a few times in a small fine-tuning set barely moves billions of pre-trained weights. The model doesn’t reliably “memorize” it.
- No easy updates. Policy changed? You must fine-tune again. With RAG you just edit the file.
- No citations. A fine-tuned model still can’t tell you where a fact came from, or admit it doesn’t know. RAG points at the exact paragraph.
- Hallucination can get worse. You may teach the model to sound more confident about things it only half-absorbed.
- It can forget other things. Pushing weights hard toward your data can degrade the model’s general ability (called catastrophic forgetting).
So: fine-tune for form, retrieve for facts. Fine-tuning is great for structured output, classification, tone/voice, following a specific procedure, or making a small cheap model imitate a big one. It is bad at being your fact database.
5. A worked example to lock it in
Brief: “Build a customer-support assistant for our software.”
- It must know our product, pricing, and troubleshooting steps → RAG over your docs and past tickets. (Knowledge → changes often → use RAG.)
- It must always reply in our calm, 3-step troubleshooting format and escalate politely → prompting, and if that’s not consistent enough, fine-tuning. (Behavior/format.)
- It must know today’s outage status → RAG pointed at a live status source. (Fresh facts.)
- It must not answer questions about competitors → prompting (an instruction/guardrail).
One assistant, four different tools, each chosen by asking: is this about knowledge or behavior, and how often does it change?
Check yourself
- Complete the sentence: “Fine-tuning changes ___; RAG changes ___.”
- Give two reasons fine-tuning is a poor way to teach a model your private facts.
- A client wants the bot to always answer in a fixed JSON format and know their live inventory. Which tool for which part?
- True or false: RAG and fine-tuning are competitors you must choose between. (Explain.)