Applied AI / Retrieval-Augmented Generation (RAG) / Module 01 · How AI Language Models Work (in plain English)
Module 01

How AI Language Models Work (in plain English)

Retrieval & Knowledge Retrieval-Augmented Generation (RAG)

Module 01 — How AI Language Models Work (in plain English)

You can’t understand RAG until you understand what a language model is and where its “knowledge” lives. We’ll build this from the ground up. No math required.

1. What is a language model, really?

A language model (the thing inside ChatGPT, Claude, etc. — also called an LLM, for Large Language Model) does one deceptively simple thing:

Given some text, it predicts what word (or piece of a word) is most likely to come next.

That’s it. You give it “The capital of France is” and it predicts “Paris.” You give it the start of an essay and it predicts the next word, then the next, then the next — one at a time — until a whole answer comes out.

Everything impressive an LLM does — answering questions, writing code, summarizing — is built on this single “predict the next piece of text” ability, repeated over and over.

2. Tokens — the pieces of text

Models don’t actually work with words. They work with tokens: chunks of text, often a word or part of a word. For example, “unbelievable” might be split into un, believ, able. The word “cat” is one token; “Paris” is one token.

Why care? Two reasons that matter later:

  • Cost and limits are measured in tokens. When people say “this model has a 128,000-token context window,” they mean it can read about that many chunks of text at once.
  • The model predicts one token at a time. “Generating text” = predicting the next token, adding it, and repeating.

A rough rule of thumb: 1 token ≈ ¾ of a word, so 1,000 tokens ≈ 750 words.

3. The neural network — a giant adjustable machine

Inside the model is a neural network. Think of it as an enormous mathematical machine with millions of little dials. Text (as tokens, turned into numbers) goes in one end; a prediction comes out the other. As the numbers flow through, they get multiplied, added, and combined in layers.

Here’s the only mental image you need:

A neural network is a machine with billions of tiny dials. Setting all those dials to just the right positions is what makes the machine good at predicting text.

Those dials have a name. They’re called weights.

4. Weights — the heart of it (this is the key concept)

Weights (also called parameters) are simply the numbers inside the neural network — the positions of all those dials. A modern LLM has billions of them. When you hear “a 70-billion-parameter model,” that means the machine has 70 billion of these numbers.

What do the weights do? Collectively, they encode everything the model has learned about language and the world. The fact that “Paris” follows “the capital of France is,” the grammar of English, the style of a sonnet, basic facts about biology — all of it is stored as patterns across those billions of numbers.

Crucial point: the model’s knowledge is not stored as readable sentences in a database somewhere. It’s smeared across billions of weights as statistical patterns. You can’t open a file and read “fact #4,201.” The knowledge is baked into the dial settings.

Analogy: Weights are like the precise tuning of every string and component in a huge piano that plays itself. The “music” (the model’s knowledge) doesn’t live in any one string — it emerges from how all of them are tuned together.

5. Training — how the dials get set

The weights don’t start out useful. Training is the process of setting them.

It works roughly like this:

  1. Start with all the dials at random positions. The model’s predictions are gibberish.
  2. Show it an enormous amount of text — a large chunk of the internet, books, code: trillions of tokens.
  3. For each bit of text, ask it to predict the next token, then check the right answer.
  4. When it’s wrong, nudge the dials slightly so it would have been less wrong. Repeat billions of times.

Slowly, the dials settle into positions that make the model good at predicting text — and, as a side effect, good at “knowing” facts and language. This first, massive stage is called pre-training. It costs millions of dollars and takes weeks or months on thousands of specialized computers.

The most important consequence for us:

Once training is done, the weights are frozen. The model’s knowledge is fixed at that moment — its knowledge cutoff. It knows nothing that happened afterward, and nothing it never saw (like your private documents).

6. Why this matters for RAG (the punchline)

Three facts about weights set up the entire need for RAG:

  1. Weights are frozen. A model trained with data up to, say, last year knows nothing newer. It can’t learn today’s news by itself.
  2. Weights are blurry. Knowledge is compressed and averaged across billions of numbers — like a heavily compressed photo. The model remembers the gist of common things but garbles rare details, exact numbers, names, and quotes.
  3. Weights are private to the model. Your company’s internal wiki, your customer database, last week’s support tickets — none of that was in the training data, so none of it is in the weights.

So if you ask a raw model about your refund policy or yesterday’s sales, it has three options: (a) say it doesn’t know, (b) refuse, or (c) — most dangerously — make up a confident, plausible-sounding answer. That last behavior is called hallucination, and it’s the subject of the next module.

RAG is the technique that hands the model the real documents at question time, so it doesn’t have to rely on frozen, blurry, private weights at all.

Check yourself

  • What are weights, in one sentence?
  • What’s the difference between training and just running (using) a model?
  • Why can’t a finished model know about events after its knowledge cutoff?
  • Why is the model’s knowledge described as “blurry”?

Next: Module 02 — The Knowledge Problem →

← Tutorial overview