Module 07 — Loading & Chunking Documents

This is the most underrated module in the whole tutorial. Boring as it sounds, parsing and chunking decide most of your final quality. Experts know it; beginners skip it. Don’t skip it.

1. Parsing — getting clean text out of messy files

Parsing (or extraction) means pulling the actual text out of a file. Sounds trivial — until you meet a real PDF.

Your documents arrive as PDFs, Word files, web pages, slide decks, scanned images, spreadsheets. Each hides text in awkward ways:

PDFs often store text in a layout-driven way. A two-column page can come out with the columns interleaved into nonsense. Tables flatten into jumbled numbers. Headers and footers repeat on every page.
Scanned documents are images — there’s no text to extract until you run OCR (Optical Character Recognition: software that “reads” text out of an image).
Slides and web pages are full of navigation, captions, and boilerplate mixed with the real content.

Why this matters so much: every later step builds on this text. If your table came out scrambled, every answer about that table will be wrong — no clever prompt can fix corrupted input. Garbage in, garbage out, literally.

Best practices for parsing:

Use layout-aware parsers for PDFs (ones that understand columns, tables, and reading order) — not a crude “dump all text.”
Handle tables specially. Consider extracting each table on its own and storing both a clean text version and the structured rows.
OCR scanned pages, and check the OCR quality before trusting it.
Strip boilerplate — repeated headers/footers, nav menus, cookie banners — so it doesn’t pollute your library.
Keep the structure you find (headings, sections, page numbers) — you’ll reuse it for chunking and citations.

2. Why we chunk at all

A chunk is a small piece of a document — usually a paragraph or a few. Instead of storing whole documents, we split them into chunks and store those. Two reasons:

The model can only read so much at once (the context window). We want to fetch just the relevant passages, not entire documents.
Search works better on focused pieces. (In Module 08 you’ll see each chunk gets summarized into a single “meaning” — and a focused paragraph has a clear meaning, while a whole 50-page document has a muddy, averaged one.)

So chunking = cutting documents into retrievable, meaningful pieces.

3. The chunking trade-off (the core tension)

There’s no free lunch in choosing chunk size:

Chunks too small → each piece is missing context. The fact you need is split across several chunks, or a chunk is too thin to answer anything on its own. (“What does ‘it’ refer to? The chunk got cut before the subject.”)
Chunks too big → the piece covers many topics at once, so its “meaning summary” becomes a blurry average and search gets imprecise. You also waste context-window space and money fetching irrelevant material along with the relevant bit.

Goldilocks rule: a chunk should be big enough to stand on its own as a coherent thought, small enough to be about one thing. A paragraph or a short section is often the sweet spot.

There is no universal best size. It depends on your documents and is something you tune and measure (Module 12). Treat chunk size as a setting to experiment with, never a fixed truth.

4. Chunking strategies, from simplest to smartest

(a) Fixed-size with overlap (the baseline)

Cut every ~256–512 tokens. Add overlap — repeat the last ~10–20% of one chunk at the start of the next.

Why overlap? So a fact sitting right on a boundary isn’t sliced in half and lost from both chunks. Overlap is cheap insurance.

(b) Recursive / structural (a good default)

Split on natural boundaries — paragraphs first, then sentences if a paragraph is too long — so you rarely cut mid-thought. This respects how the text is actually written.

(c) Document-aware

Split on the document’s own structure: Markdown headings, sections, slide boundaries, or — for code — whole functions. Keeps naturally-related text together.

(d) Semantic chunking

Use meaning (Module 08’s embeddings) to group sentences that are about the same thing, starting a new chunk when the topic shifts. More accurate, more compute.

(e) Contextual / “small-to-big” chunking (a pro favorite)

Two powerful ideas:

Add context to each chunk: prepend the document title and section heading to every chunk, so an isolated paragraph still “knows” where it came from. (A chunk that just says “The limit is 30 days” is useless alone; “[Returns Policy → Laptops] The limit is 30 days” is searchable and self-explanatory.)
Small-to-big (parent-document) retrieval: store small chunks for precise matching, but when one matches, hand the model the larger surrounding passage for answering. Best of both worlds — precise search, rich context.

5. Metadata — the cheap superpower

Metadata is extra information you attach to each chunk: source filename, title, section heading, page number, date, author, document type, and — critically — access tags (who’s allowed to see it).

Why it’s a superpower:

Filtering: “only search HR documents from this year” narrows the haystack before searching — often the cheapest accuracy boost you can get.
Citations: to say “from the Returns Policy, page 3,” you need that stored on the chunk.
Security/access control: to stop user A retrieving user B’s documents, each chunk needs permission tags (Module 14).

Rule: every chunk should carry metadata. It’s a few extra fields now that unlock filtering, citations, and security later.

6. Common chunking mistakes (preview of Module 13)

Fixed-size cutting with no overlap → facts sliced in half.
Chunks longer than your embedding model can handle → silently truncated (Module 08).
One chunking strategy for everything → code, tables, prose, and slides need different handling.
No metadata → can’t filter, cite, or secure.
Ignoring parsing quality → feeding scrambled text into the whole pipeline.

Check yourself

Why can a real PDF produce scrambled text, and why is that catastrophic downstream?
Explain the too-small vs too-big chunking trade-off.
What problem does chunk overlap solve?
What is “small-to-big” retrieval, and why is it clever?
Give three things you’d store as metadata and what each enables.

Next: Module 08 — Embeddings & Vector Databases →