Module 10 — Data Quality & Curation

Goal: Master the discipline that quietly decides whether your fine-tune succeeds or fails. You already know “the data is the model” (Module 04). This module is the craft of making data good: cleaning, deduplicating, balancing, filtering, and formatting. Expert practitioners spend more time here than anywhere else — and it’s where you’ll get your biggest quality wins.

1. Why this module might be the most important one

If you skipped every advanced technique and only got really good at data curation, you’d still produce better models than someone who mastered every algorithm but fed them mediocre data. That’s not an exaggeration — it’s the consensus of practitioners:

“Garbage in, garbage out” is not a cliché in fine-tuning; it’s the central law. The model imitates exactly what you show it, including the contradictions, errors, and sloppiness you didn’t notice.

There’s a famous, repeatable finding: a small, carefully curated dataset often beats a large, noisy one. A thousand clean, consistent, on-target examples can outperform tens of thousands of scraped, inconsistent ones. Curation is the highest-leverage work in fine-tuning. Treat this module as a craft to practice, not facts to memorize.

2. The dimensions of data quality

“Good data” isn’t one thing. It’s several, and you should audit each:

Correctness. Are the outputs actually right? Wrong labels teach wrong behavior, confidently.
Consistency. Do similar inputs get similar-style outputs? Inconsistent formatting/tone/length in your data → an inconsistent model. This is the most common silent killer, especially with multiple human authors or mixed sources.
Relevance. Does every example reflect the behavior you actually want? Off-task examples dilute learning and can teach things you don’t want.
Diversity / coverage. Does the data span the real range of inputs (topics, phrasings, difficulty, edge cases)? Narrow data → a model that fails outside its narrow comfort zone.
Format correctness. Does every example match the exact structure (chat template, JSON schema, fields) you’ll use? (Module 04 §3.)
Difficulty balance. A healthy mix of easy and hard cases. All-easy → a model that can’t handle hard ones. All-hard → unstable training.
Cleanliness. No broken encodings, stray HTML, truncated text, leftover metadata, or junk tokens.

A practical audit habit: take 50 random examples and score each on these dimensions. The problems you find in 50 are almost always present in all of them.

3. The cleaning pipeline

A typical curation pipeline, in order. You’ll adapt it per project, but the stages are general.

Stage 1 — Inspect (always start here)

Before automating anything, read a few dozen examples yourself. You cannot clean what you haven’t seen. You’ll spot patterns (a recurring formatting glitch, a category of bad answers) that guide every later step. Skipping this is the #1 mistake.

Stage 2 — Basic cleaning

Strip junk: stray HTML tags, markdown artifacts, “As an AI language model…” boilerplate, broken Unicode, excessive whitespace.
Fix encoding issues (mojibake, smart-quote messes).
Remove or fix truncated examples (an answer cut off mid-sentence teaches the model to stop abruptly).
Normalize obvious inconsistencies (e.g., consistent quote style) — carefully, without destroying meaningful variation.

Stage 3 — Filter for quality

Drop examples with wrong/empty/placeholder outputs.
Drop off-task examples (irrelevant to your goal).
Remove examples that are too short to be useful or too long to fit your max length (Module 04 §4).
For verifiable tasks, execute/validate outputs and drop the ones that fail (Module 09 §5).
Optionally use LLM-as-judge scoring to rank and keep the best (Module 09 §5, Module 11 §6).

Stage 4 — Deduplicate

A whole topic — see §4. Remove exact and near-duplicates.

Stage 5 — Balance

Adjust the mix so no category, style, or difficulty dominates unintentionally — see §5.

Stage 6 — Format & split

Convert to the exact training format (Module 04 §2), apply/verify the chat template, and split into train/validation/test without leakage (Module 04 §7). Inspect one fully-formatted example (Module 04 §8). Then you’re ready to train.

4. Deduplication (more important than it sounds)

Duplicates and near-duplicates are everywhere — in scraped data, in merged sources, and especially in synthetic data (Module 09). They cause real harm:

They make the model overfit to the repeated examples (Module 03 §7) — effectively training extra epochs on those.
They waste compute and skew your data distribution.
Worst of all, a duplicate spanning your train and test sets is data leakage — the model “passes the test” because it memorized the answer, and your evaluation lies to you (Module 04 §7).

Levels of deduplication:

Exact dedup: remove identical examples (hash each example, drop repeats). Trivial and mandatory.
Near-dedup: remove examples that are almost identical (e.g., differ by a word or two). Done with techniques like MinHash/LSH (fast approximate matching on text) or embedding similarity (embed each example, drop pairs above a similarity threshold). This catches the subtle repetition synthetic data loves to produce.
Cross-split dedup: specifically check that no train example is a near-duplicate of any validation/test example. Non-negotiable for trustworthy evaluation.

Even a simple exact-dedup pass plus a similarity check will put you ahead of most beginners.

5. Balancing and coverage

Two related goals: don’t let any group dominate, and make sure you cover the real space.

Class/category balance. If you’re teaching classification and 90% of examples are one class, the model may just learn to guess that class. Balance the categories (downsample the majority, generate/collect more of the minority — Module 09).
Style/length balance. If most answers are long, the model learns to always be long. Ensure variety matching real needs.
Difficulty coverage. Include edge cases, tricky inputs, and “hard negatives” (plausible-but-wrong situations the model must handle). Models get good at what you show them; show them the hard stuff deliberately.
Real-distribution matching. The proportions in your training data teach the model what’s “normal.” Try to match the real-world frequency of cases — unless you’re deliberately oversampling rare-but-important cases (a legitimate, common choice; just do it knowingly).

A useful technique: embed all examples and cluster them, then look at the cluster sizes. Giant clusters = over-represented; tiny/empty regions = gaps to fill. This turns “is my data diverse?” from a guess into a measurement.

6. Handling tricky data issues

Real datasets throw curveballs. Expert moves for common ones:

Noisy labels you can’t fully fix: train more gently (fewer epochs, more regularization — Module 03), or use LLM-judge filtering to keep the cleanest subset. Sometimes less data, higher quality is the answer.
Sensitive content / PII: detect and remove or anonymize personal data before it enters training (and before sending to any third-party API — Module 09 §7). A fine-tuned model can memorize and leak training data, so PII in equals PII potentially out.
Contradictions in the data: if two examples teach opposite behavior for similar inputs, the model learns to be inconsistent or just averages them into mush. Hunt for and resolve contradictions — this is a top cause of “the model is unreliable.”
Format drift across sources: when merging datasets, normalize them to one schema/template first. Mixed formats confuse the model and the masking logic (Module 04 §5).
Length outliers: a few enormous examples can dominate memory and skew the model. Cap or split them.

7. Documenting your data (the professional habit)

Experts treat datasets like code: versioned and documented. For each dataset keep a short record (a “data card”):

Source(s) and how it was collected/generated (real, synthetic, which generator — Module 09).
Size, splits, and how the splits were made (and that they’re leak-free).
Cleaning/filtering steps applied (so results are reproducible).
Known limitations and biases.
Version (so you can tie a model back to the exact data that made it).

Why bother? Because when a model misbehaves in production, the answer is almost always in the data — and you can only investigate data you can reconstruct. This habit also makes you trustworthy to teammates and auditors. It’s a small effort that signals real professionalism.

8. A curation mindset to carry forever

Three principles that summarize the whole module:

Look at your data. Repeatedly. Automated metrics never replace reading real examples. Most breakthroughs in a fine-tuning project come from someone finally noticing what the data actually contains.
Quality and consistency over raw quantity. When in doubt, cut the noisy half rather than add a noisier double.
Curation is iterative. You’ll clean, train, evaluate (Module 11), discover a data problem the evaluation revealed, and curate again. The loop is the work. Embrace it instead of expecting one clean pass.

Expert truth: Most of the time, “make the model better” means “make the data better.” Internalize this and you’ll outperform people chasing fancier algorithms.

Module 10 checklist

I believe (and can defend) that data quality dominates algorithm choice.
I can list the dimensions of data quality and audit a sample against them.
I can run the cleaning pipeline stages in order, starting with inspect.
I can deduplicate (exact, near, cross-split) and explain why leakage is dangerous.
I can balance categories, styles, difficulty, and measure coverage via clustering.
I can handle noisy labels, PII, contradictions, and format drift.
I keep a data card documenting source, splits, cleaning, and version.

➡️ Next: Module 11 — Evaluation