Module 04 — Data Foundations: The Fuel of Fine-Tuning

Goal: Master the part of fine-tuning that actually determines success — the data. You’ll learn the formats, the all-important chat templates, how tokenization and label masking work, and how to structure a dataset correctly. Pros say “fine-tuning is 90% data.” This module is why.

1. The uncomfortable truth: your data is the model

You can copy an expert’s training script exactly and still get a bad model — because the script was never the hard part. The quality, format, and relevance of your data decide almost everything. A mediocre method on excellent data beats an excellent method on mediocre data, every time.

So before any code, internalize the goal of a fine-tuning dataset:

A fine-tuning dataset is a collection of examples that demonstrate the exact behavior you want. The model learns by imitation. Whatever your data shows — including its mistakes, biases, and quirks — is what the model will become.

If your examples are inconsistent, the model learns to be inconsistent. If they’re all one style, it learns that style. The data is the specification.

2. What an example looks like: the structure of training data

For the most common kind of fine-tuning (SFT, the next module), each training example is fundamentally an input → desired output pair. But there are a few standard shapes you must recognize because libraries expect them.

2a. Instruction format (single-turn)

The classic shape, popularized by the Alpaca dataset. Each example has an instruction, optional input, and the desired output:

{
  "instruction": "Translate the following sentence to French.",
  "input": "The weather is nice today.",
  "output": "Le temps est agréable aujourd'hui."
}

Good for single-shot tasks: classification, extraction, translation, summarization.

2b. Chat / conversational format (the modern default)

Most modern instruct models expect data as a list of messages with roles — system, user, assistant — exactly like a chat:

{
  "messages": [
    {"role": "system", "content": "You are a concise customer-support agent for Acme Corp."},
    {"role": "user", "content": "How do I reset my password?"},
    {"role": "assistant", "content": "Go to Settings → Security → Reset Password, then check your email for the link."}
  ]
}

This format naturally supports multi-turn conversations (more user/assistant pairs) and is what you’ll use most often. The assistant messages are the parts you want the model to learn to produce; the system and user messages are the context it should respond to. Remember that distinction — it’s the key to “masking” in §5.

2c. Preference format (for Module 12)

For preference tuning (DPO/RLHF), each example shows a prompt with a better and a worse response:

{
  "prompt": "Explain gravity to a five-year-old.",
  "chosen": "Gravity is the invisible pull that brings things down to the ground...",
  "rejected": "Gravity is a fundamental interaction described by general relativity..."
}

You’ll use this later; just recognize the shape now.

File formats on disk

These records are typically stored as JSONL (JSON Lines — one JSON object per line), or as CSV/Parquet, or as a Hugging Face dataset. JSONL is the lingua franca:

{"messages": [...]}
{"messages": [...]}
{"messages": [...]}

One example per line, easy to stream and inspect. Get comfortable reading and writing it.

3. Chat templates: the detail that breaks beginners

Here’s something that quietly ruins a huge fraction of first fine-tunes, so we’ll be very explicit.

A chat model wasn’t trained on raw {"role": "user", "content": "..."} JSON. It was trained on text with special control tokens that mark where each role’s turn begins and ends. Every model family has its own format, called its chat template. For example, one family might format a turn as:

<|im_start|>user
How do I reset my password?<|im_end|>
<|im_start|>assistant
Go to Settings → Security...<|im_end|>

The rule that saves you days of pain: When you fine-tune a chat model, you must format your data with the exact chat template that model expects. Use a different template (or none) and the model gets confused, never learns to stop talking, or behaves erratically — because the structural signals it relies on are missing or wrong.

The good news: you don’t hand-write these. The tokenizer knows its own template, and one function applies it:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("the-base-model")

messages = [
    {"role": "user", "content": "How do I reset my password?"},
    {"role": "assistant", "content": "Go to Settings → Security..."},
]

text = tok.apply_chat_template(messages, tokenize=False)
print(text)   # ← shows the exact special-token formatting this model wants

apply_chat_template is one of the most important functions in this course. It guarantees your data matches the model. Modern trainers (like trl’s SFTTrainer) often call it for you if you pass data in the messages format — but you should always know it’s happening and inspect its output at least once. Always look at one fully-formatted, tokenized example before training. It catches more bugs than any other habit.

4. Tokenization in practice: turning examples into model input

Recall from Module 01 that models consume token IDs, not text. The tokenizer does this conversion. Two practical concerns for training:

Maximum sequence length

You must pick a max length (in tokens) for training examples — e.g., 1,024, 2,048, 4,096. Examples longer than this get truncated (cut off) or split. Why it matters:

Memory scales with sequence length (often steeply). Doubling max length can more than double memory use. Long-context fine-tuning is a real cost.
Truncation can silently destroy examples — if your important assistant answer gets cut off the end, the model trains on a broken example. Choose a max length that comfortably fits your real examples, and check how many get truncated.

Padding

Within a batch, examples must be the same length to be processed together, so shorter ones get padded with a filler token up to the batch’s longest example. Padding tokens are ignored in the loss (via an attention mask) so they don’t teach the model anything. Modern setups also use packing — concatenating several short examples into one sequence to avoid wasting space on padding, which speeds up training. Recognize these terms; trainers handle the mechanics.

5. Label masking: teaching the model to respond, not to parrot

This is a subtle but crucial idea that, once understood, makes you noticeably more competent than the average beginner.

When we train on a chat example, the full sequence contains the system prompt, the user’s question, and the assistant’s answer. But we don’t want the model to learn to generate the user’s question — that’s not its job. We only want it to learn to produce the assistant’s part, given everything before it.

The mechanism is label masking (a.k.a. completion-only or assistant-only loss). We mark the non-assistant tokens (system + user) so they don’t count toward the loss:

TOKENS:   [system prompt] [user question] [assistant answer]
LABELS:   [   masked    ] [   masked    ] [   LEARN THESE   ]
                ↑                ↑                  ↑
         ignored in loss   ignored in loss   contributes to loss

So the model still reads the system and user text (it needs that context to know what to respond to), but it’s only graded on predicting the assistant’s words. This focuses all the learning where you want it.

Why you must know this: If masking is misconfigured, the model wastes capacity learning to imitate user phrasing, or — a common silent failure — learns from the whole sequence and behaves oddly. Libraries like trl handle masking when you use the right data format and settings (e.g., a “completion-only” collator or train_on_responses_only). Knowing what it’s for lets you verify it’s working and recognize the symptoms when it isn’t.

There’s a nuance: for some tasks (like pure style transfer or “continue this text”) you do train on the whole sequence. The point isn’t “always mask” — it’s “know what you’re computing loss on, and make sure it matches what you want the model to learn.”

6. How much data do you need?

A question everyone asks. The honest answer is “it depends on the task,” but here are working guidelines that will serve you well:

Style / format / tone tuning: Often surprisingly little — a few hundred to ~1,000 high-quality, consistent examples can strongly shift style and format.
A narrow skill or task (classification, extraction, a specific transformation): 1,000–10,000 examples is a common sweet spot for solid results.
Broad capability or robust behavior across many situations: tens of thousands+.
Teaching genuinely new, complex behavior: more still, and you may hit the limits of what fine-tuning a small model can do.

Two principles that matter more than the raw numbers:

Quality and consistency beat quantity. 500 clean, consistent, on-target examples beat 5,000 noisy contradictory ones — often dramatically. (Module 10 is devoted to this.)
Diversity matters within the target behavior. You want varied examples that all demonstrate the same desired behavior across different inputs, so the model learns the general pattern rather than memorizing specifics.

If you don’t have enough real data, that’s not a dead end — it’s the reason synthetic data generation (Module 09) exists, and it’s now a core professional skill.

7. Splitting your data (do this from day one)

From Module 03 you know why; here’s the how. Before training, split your dataset into:

Training set (~80–90%) — what the model learns from.
Validation set (~5–10%) — watched during training to detect overfitting and tune hyperparameters. The model never trains on it.
Test set (~5–10%, optional but ideal) — touched only once, at the very end, to get an honest final score. Keeping it untouched prevents you from fooling yourself by tuning until you accidentally overfit the validation set too.

Critical detail: make the splits representative (each split should contain the same variety of cases) and non-overlapping (no example, or near-duplicate, in two splits — a leak that makes your model look better than it is). For conversational or user-grouped data, split by user/document/conversation, not by individual rows, so pieces of the same conversation don’t end up on both sides.

8. A minimal end-to-end data example

Here’s the whole pipeline in miniature, to make it concrete:

import json
from datasets import load_dataset
from transformers import AutoTokenizer

# 1. Load your JSONL of chat examples
#    each line: {"messages": [ {role, content}, ... ]}
ds = load_dataset("json", data_files="my_data.jsonl", split="train")

# 2. Split into train / validation
ds = ds.train_test_split(test_size=0.1, seed=42)
train_ds, val_ds = ds["train"], ds["test"]

# 3. Load the tokenizer that matches your base model
tok = AutoTokenizer.from_pretrained("the-base-model")

# 4. SANITY CHECK — always look at one fully-formatted example!
example = train_ds[0]["messages"]
print(tok.apply_chat_template(example, tokenize=False))
# Read this output carefully: are the role markers right?
# Is the assistant answer complete (not truncated)? Does it look like
# what the model was pretrained to expect? This 30 seconds saves hours.

Notice step 4 isn’t optional ceremony — inspecting a real formatted example is the highest-leverage habit in the whole course. Most catastrophic fine-tunes were visibly broken at this step and nobody looked.

Module 04 checklist

I can describe the instruction, chat, and preference data formats and read JSONL.
I understand chat templates and why using the model’s exact one is mandatory.
I know what apply_chat_template does and that I should inspect its output.
I can explain max length, truncation, padding, and packing.
I can explain label masking and why we usually train only on assistant tokens.
I have realistic expectations for how much data different goals need.
I can split data into train/validation/test without leakage.

➡️ Next: Module 05 — Supervised Fine-Tuning (SFT)