Document Chunking in RAG

After loading, documents are usually too large to embed as a single vector — so you split them into smaller pieces called chunks. Chunking is one of the highest-leverage steps in RAG: retrieval returns chunks, not whole documents, so how you split directly controls what the LLM sees. Good chunking is often the difference between a RAG system that works and one that doesn't.

💡 In one line: Chunking splits documents into smaller passages so retrieval can return precise, relevant pieces — balancing precision against context.

What is Chunking?

Chunking breaks a document into smaller passages that each get embedded and stored separately. At query time, retrieval finds the most relevant chunks and feeds those to the LLM. The chunk is the unit of retrieval.

Why Chunk?

• Size limits — embedding models and LLM context windows can't take an entire document at once.
• Precision — smaller chunks give targeted retrieval (less irrelevant text).
• Noise — a whole document embedded as one vector is too vague to match well.

Chunk Size & Overlap

Two knobs define most chunking:

• Chunk size — how big each piece is (often 200–500 tokens). Smaller = more precise; larger = more context.
• Overlap — repeat some text between adjacent chunks (often 10–20%) so an idea isn't cut in half at a boundary.

Remove

The Core Trade-off

Chunking is a balance:

• Too small → precise matches, but missing context (the answer's surrounding detail is elsewhere).
• Too large → rich context, but noisy retrieval and wasted tokens.

Finding the sweet spot for your content is the whole game.

Chunking Strategies

Different content calls for different splits. 

Whiteboard

• Fixed-size — split by characters/tokens. Simple, but cuts mid-sentence.
• Recursive — split on natural boundaries (paragraphs → sentences → words). The common default.
• Semantic — split where the meaning shifts, using embeddings.
• Structure-aware — split by headings, sections, tables, or code blocks.

Keep Metadata on Every Chunk

Each chunk should carry its source, page, and section metadata. That's what enables citations ("this came from page 12") and metadata filtering at retrieval time. Don't drop it during splitting.

Advanced Patterns

• Small-to-big (parent-child) — retrieve small chunks for precision, then return the larger parent for context.
• Contextual headers — prepend the section/title to each chunk so it's self-describing.

Best Practices

• Match the strategy to the content (prose vs. structured vs. code).
• Tune size + overlap, then measure retrieval quality.
• Respect structure — don't split tables or code blocks mid-way.
• Preserve metadata on every chunk.

Summary

• Chunking splits documents into the passages that retrieval actually returns.
• Tune chunk size and overlap to balance precision vs. context.
• Pick a strategy to fit the content: recursive, semantic, or structure-aware.
• Keep metadata on every chunk for citations and filtering.
• Consider small-to-big retrieval to get precision and context. EOF echo created