RAG Chunking Strategies Explained

Chunking is the step in a retrieval-augmented generation pipeline where source documents are split into smaller pieces before being embedded and stored in a vector database. It sounds like a preprocessing detail, but chunking strategy is often the single biggest factor in whether a RAG system retrieves the right passage — get it wrong and no amount of tuning the retrieval or generation side will fix it.

The core tension is size. Chunks that are too large dilute a search query’s relevance — a 5,000-word document embedded as one vector produces a blurry average that doesn’t match any specific question well. Chunks that are too small lose context — a two-sentence fragment about “the deductible” is useless if it doesn’t say which insurance plan it’s describing.

Fixed-size chunking

The simplest approach: split text every N tokens or characters, often with some overlap between consecutive chunks so a sentence spanning a boundary isn’t orphaned entirely in either piece.

chunk_size = 500 tokens
chunk_overlap = 50 tokens

This is fast, predictable, and requires no understanding of the document’s structure. The downside is that it splits blindly — a chunk boundary can land in the middle of a sentence, a table row, or a code block, breaking the semantic unit that made the passage meaningful in the first place.

Recursive character/token splitting

A refinement of fixed-size chunking: instead of cutting at a raw character count, try a hierarchy of separators — paragraph breaks first, then sentences, then words — recursively splitting only when a piece still exceeds the target size. This keeps natural boundaries (paragraphs, sentences) intact wherever possible, falling back to a harder cut only when a single paragraph is too long to fit in one chunk.

This is the default strategy in most RAG frameworks and libraries, because it’s a good balance of simplicity and respecting document structure without needing any model inference to decide where to split.

Structure-aware chunking

Rather than splitting by size at all, split along the document’s own structure: Markdown headings, HTML sections, code function boundaries, or table rows. A technical doc chunked by ## heading keeps each section’s content together as one retrievable unit, and lets you attach the heading as metadata so the chunk carries its own context even in isolation.

This works well for documents with strong, consistent structure — API references, structured knowledge bases, legal contracts with numbered clauses — but needs custom logic per document type and falls apart on unstructured prose.

Semantic chunking

A newer approach uses embeddings to find natural topic boundaries: embed consecutive sentences, measure the similarity between neighbors, and cut where similarity drops sharply — the sign that the topic just shifted. Chunks end up variable-length but semantically coherent, each one covering a single idea rather than an arbitrary token count.

The cost is computational: semantic chunking requires embedding every sentence (or a sliding window of sentences) just to decide where to split, which is meaningfully more expensive than counting tokens. It tends to pay off most on long-form, topic-shifting content like articles or transcripts, where fixed-size splitting would otherwise cut across topic changes constantly.

Comparing the strategies

Strategy	Respects structure	Compute cost	Best for
Fixed-size	No	Very low	Quick prototypes, uniform data
Recursive splitting	Partially (paragraphs/sentences)	Low	General-purpose default
Structure-aware	Yes (explicit markup)	Low	Docs with consistent headings/sections
Semantic	Yes (topic-based)	High	Long-form prose, mixed topics

Chunk size and overlap in practice

There’s no universally correct chunk size — it depends on what questions the system needs to answer. A few practical guidelines:

• Match chunk size to query granularity. If users ask narrow factual questions, smaller chunks (100–300 tokens) retrieve precisely. If they ask questions needing broader context, larger chunks (500–1,000 tokens) avoid losing the surrounding explanation.
• Use overlap to avoid boundary loss, typically 10–20% of the chunk size, so content near a cut point appears in two chunks instead of being split with neither half making sense alone.
• Attach metadata to every chunk — source document, section heading, page number — so the retrieved passage can be traced back and so the generation step can cite it accurately.
• Test retrieval, not just generation. A common failure mode is tuning the prompt and model while an upstream chunking bug is quietly returning the wrong passages; an LLM eval that scores retrieval hit rate against a labeled question set catches this before it shows up as a wrong answer downstream.

Chunking is a retrieval decision, not a storage decision

It’s easy to treat chunking as a mechanical preprocessing step and spend most of the engineering effort tuning the embedding model or the vector search index instead. In practice, a mismatched chunking strategy caps retrieval quality regardless of how good the embeddings are — you can’t retrieve a passage that was never stored as a coherent unit. Treating chunk boundaries as a design decision tied to how the content will actually be queried, rather than an implementation detail, is usually the highest-leverage fix available in an underperforming RAG system.

The takeaway

Chunking strategy shapes what a RAG system can retrieve more than most of the tuning that happens downstream of it. Fixed-size splitting is fast but structure-blind; recursive splitting is a solid general default; structure-aware chunking suits documents with strong headings; semantic chunking handles long, topic-shifting prose at a higher compute cost. Pick based on your documents’ structure and your queries’ granularity, attach metadata to every chunk, and evaluate retrieval quality directly rather than assuming it’s fine because the generated answers read well.