Module 09 — Synthetic Data Generation

Goal: Learn to create high-quality training data using AI models when you don’t have enough real data — now a core professional skill. We’ll cover why and when it works, the main generation patterns, how to avoid the traps (and the legal/ethical pitfalls), and a concrete pipeline you can run. Remember Module 04’s law: the data is the model. So generating good data is generating a good model.

1. Why synthetic data exists

The single biggest blocker to fine-tuning, in the real world, is not having a dataset. You’ve decided fine-tuning is right (Module 02) and you know the format you need (Module 04) — but you have 30 examples, not 3,000. Collecting and hand-labeling thousands of high-quality examples is slow, expensive, and sometimes impossible (the data is private, rare, or doesn’t exist yet).

Synthetic data generation solves this by using a capable model (often a larger, stronger one) to generate training examples for you. Instead of paying humans to write 5,000 examples, you prompt a strong model to produce them — in minutes, at a fraction of the cost.

This isn’t a fringe hack. Many of the best open instruct models were trained substantially on synthetic data, and “generate-then-filter” pipelines are standard industry practice. Knowing how to do this well is one of the highest-leverage skills you can have.

2. The core patterns (your generation toolkit)

There are a handful of repeatable patterns. Learn the names and shapes; you’ll mix and match them.

Pattern A — Distillation (teacher → student)

Use a strong “teacher” model to generate the desired outputs, then fine-tune a smaller “student” model to imitate them. You’re “distilling” the teacher’s capability into a cheaper model.

  real/seed inputs  ──►  STRONG teacher model  ──►  high-quality outputs
                                                          │
                          (input, teacher-output) pairs ──┘
                                                          │
                                                          ▼
                                    fine-tune small STUDENT model

This is the workhorse: it’s how you get a cheap 8B model to do one task nearly as well as a frontier model (Module 02 §3, reason #6). Important legal caveat in §7 — some providers’ terms restrict using their outputs to train competing models. Check before you build a product on this.

Pattern B — Self-Instruct / instruction generation

You have a model generate the inputs too, not just the outputs. Start from a small set of seed examples and ask a strong model to produce many new, varied instructions in the same spirit, then generate answers for them. This bootstraps a large, diverse instruction dataset from a tiny seed. (The “Self-Instruct” and “Alpaca” datasets were built this way.)

Pattern C — Evol-Instruct (complexity evolution)

Take existing examples and prompt a model to make them harder/deeper/more varied — add constraints, increase complexity, rephrase, branch into edge cases. This deliberately grows the difficulty and diversity of your data, which improves robustness. Great for pushing a model beyond easy cases.

Pattern D — Templated / programmatic generation

For structured tasks, generate data with code + templates, optionally with a model filling slots. Example: to teach English→SQL, programmatically combine table schemas, question templates, and known-correct SQL. This gives you guaranteed-correct labels (the SQL is generated by rules, not guessed), which is gold — no hallucinated answers.

Pattern E — Real data augmentation

Start from your real (but scarce) examples and use a model to paraphrase, vary, translate, or perturb them into many more. Multiplies a small real dataset while keeping it grounded in reality.

Pattern F — Persona / scenario seeding

To get diversity, give the generator varied personas, contexts, or scenarios (“Generate a support question from a frustrated first-time user,” “…from a technical admin”). This combats the biggest synthetic-data weakness: everything sounding the same (§4).

3. A concrete synthetic-data pipeline

Here’s an end-to-end pattern you can actually run for, say, building a customer-support assistant when you only have a handful of real tickets.

# Conceptual pipeline — generate, then FILTER (filtering is half the job, §5)
import json
from openai import OpenAI   # or any strong model's API / a local strong model
client = OpenAI()

# 1) SEED with diversity: vary the scenario so outputs aren't clones (Pattern F)
scenarios = [
    "a first-time user who can't log in",
    "an enterprise admin asking about SSO",
    "a frustrated customer wanting a refund",
    "a developer asking about API rate limits",
    # ... dozens of varied scenarios ...
]

generation_prompt = """You are generating training data for a support assistant for Acme Corp.
Scenario: {scenario}

Produce ONE realistic example as JSON:
{{"messages": [
  {{"role": "user", "content": "<a realistic question for this scenario>"}},
  {{"role": "assistant", "content": "<an ideal, concise, on-brand Acme support reply>"}}
]}}
Make the question specific and natural. Make the reply accurate, friendly, and under 4 sentences."""

raw = []
for scenario in scenarios:
    for _ in range(20):                      # many samples per scenario for variety
        resp = client.chat.completions.create(
            model="a-strong-teacher-model",
            messages=[{"role": "user", "content": generation_prompt.format(scenario=scenario)}],
            temperature=0.9,                 # higher temp = more diverse (but watch quality)
        )
        try:
            raw.append(json.loads(resp.choices[0].message.content))
        except json.JSONDecodeError:
            pass                             # discard malformed generations

# 2) FILTER & DEDUP (Module 10) — never skip this
clean = filter_and_dedup(raw)               # see §5 and Module 10

# 3) Save as JSONL for fine-tuning (Module 04 §2)
with open("synthetic_support.jsonl", "w") as f:
    for ex in clean:
        f.write(json.dumps(ex) + "\n")

The structure to internalize: generate broadly and diversely → filter ruthlessly → format. The generation is the easy, fun part. The filtering is where quality is won or lost.

4. The dangers of synthetic data (read carefully)

Synthetic data is powerful and dangerous. Used carelessly it produces a confident, fluent, subtly-wrong model. The specific risks:

Inherited errors and hallucinations. The teacher model’s mistakes become your training labels. If it confidently states a wrong fact, your student learns that wrong fact as truth. Synthetic data is only as correct as its generator — and worse, it launders errors into authoritative-looking examples.
Lack of diversity (mode collapse). Models love to produce similar outputs. Without deliberate diversity seeding (Pattern F, varied prompts, temperature), you get 3,000 examples that are really 5 examples in a trench coat. The model then overfits to that narrow distribution and fails on real variety.
Distribution mismatch. Synthetic inputs may not look like real user inputs (too clean, too well-formed, too polite). A model trained on tidy synthetic questions can flounder on messy real ones. Ground generation in real examples where you can (Patterns D, E).
Style/format artifacts. Generators leave fingerprints — overusing certain phrases, always formatting a certain way, being over-verbose. Your model inherits these tics.
“Model collapse” over generations. Training models on the outputs of models, repeatedly, can degrade quality over generations as diversity and rare cases get smoothed away. Keep real data in the mix as an anchor.
Amplified bias. Any bias in the generator gets concentrated and amplified in a dataset built entirely from it.

The professional’s stance: treat synthetic data as a draft written by a fast but unreliable intern. Useful, fast, scalable — but it must be checked. The value is in generation plus rigorous filtering, never generation alone.

5. Quality control: turning raw generations into a real dataset

This is where amateurs and experts diverge. After generating, you filter aggressively. Techniques (which flow directly into Module 10):

Validity checks. Does it parse as the required JSON/format? Does the output actually satisfy the task (valid SQL runs? answer addresses the question?). Drop or repair invalid ones. For verifiable tasks (code, math, SQL), execute the output to check correctness — the single most powerful filter you have.
Deduplication. Remove exact and near-duplicate examples (similar wording). Synthetic generation produces lots of near-dupes; they waste training and worsen overfitting (Module 10 §4).
Quality scoring (LLM-as-judge). Use a strong model to rate each generated example (1–5) for correctness, helpfulness, and adherence to your spec; keep only high scorers. You’re using a model to grade a model — cheap and surprisingly effective, with caveats (Module 11 §6).
Diversity auditing. Cluster or embed your examples and check you’re covering the space, not piling up in one corner. Add targeted generation for thin areas.
Human spot-checks. Read a random sample yourself — say 50–100 examples. Always. Automated filters miss things your eyes catch in two minutes. This habit is non-negotiable and constantly underrated.
Difficulty/length balance. Ensure a healthy mix; generators skew easy and verbose. Trim and balance.

A good rule of thumb: expect to discard 20–60% of raw generations. If you’re keeping 100%, your filter isn’t doing its job.

6. Combining synthetic with real data (best practice)

The strongest datasets are usually hybrids:

Use your real data as the anchor and quality bar — it defines what real inputs look like and grounds the distribution.
Use synthetic data to scale, fill gaps, and cover edge cases your real data is missing.
Seed synthetic generation from real examples (Patterns D/E) so it stays realistic.
Consider keeping a validation/test set made of real data only (Module 04 §7, Module 11), so you’re always measuring against reality, not against your generator’s quirks. This last point is crucial: never let synthetic data sneak into your test set, or you’ll measure how well you imitate the generator instead of how well you solve the real problem.

7. Legal, ethical, and licensing considerations (don’t skip)

This bites real companies; an expert thinks about it up front:

Provider terms of service. Some commercial model providers prohibit using their outputs to train competing models. If you distill from a proprietary API into a model you’ll ship as a product, you may be violating terms. Read them; when in doubt, use models with permissive licenses for output, or open models you can legally distill from.
Open dataset/model licenses. Synthetic datasets generated by a given model sometimes inherit usage restrictions. Track provenance.
Privacy. If you seed generation with real user data, you must handle PII appropriately (anonymize, get consent, comply with GDPR/CCPA, etc.). Don’t paste customer data into third-party APIs without authorization.
Bias and harm. You’re shaping a model’s behavior; generated data can encode and amplify bias (§4.6). Audit for it, especially for anything user-facing or high-stakes.
Disclosure/integrity. In research and many products, be transparent that data is synthetic. Don’t pass synthetic benchmarks off as real-world performance.

Building this awareness in early is part of being a professional, not just a tinkerer.

8. A worked mindset: “I have no data, what do I do?”

Put it together as a playbook for the most common real situation:

Find or write 10–50 real seed examples (or representative inputs). Even a handful anchors everything.
Pick patterns: Self-Instruct to expand inputs, distillation to generate ideal outputs, persona seeding for diversity, programmatic generation if the labels are verifiable.
Generate broadly with varied prompts and reasonable temperature; aim for more than you need (you’ll discard some).
Filter ruthlessly (§5): validity, dedup, LLM-judge scoring, diversity audit, human spot-check.
Hybridize: mix in your real data; reserve a real-only test set.
Fine-tune (Modules 05/07/08), evaluate on the real test set (Module 11), find the gaps.
Targeted generation to fill the specific gaps evaluation revealed, and loop.

That loop — generate, filter, train, evaluate, target the gaps — is exactly how modern datasets are actually built. You now know it.

Module 09 checklist

I can explain why synthetic data exists and that “data is the model” makes its quality critical.
I can describe the main patterns (distillation, self-instruct, evol-instruct, templated, augmentation, persona seeding).
I can sketch a generate→filter→format pipeline.
I can list the dangers (inherited errors, low diversity, distribution mismatch, collapse, bias).
I can describe the filtering techniques and that I should discard a meaningful fraction.
I know to hybridize with real data and keep a real-only test set.
I’m aware of the legal/ethical/licensing pitfalls.

➡️ Next: Module 10 — Data Quality & Curation