Why How You Split Documents Decides Your RAG Quality

You've built a RAG chatbot on top of your product docs. The embeddings are solid, you picked a well-regarded model, retrieval "works" in the sense that it returns something for every query. And yet the answers are weirdly off — half-right, missing the second half of a procedure, or confidently stitching together two unrelated sections into one answer that sounds right and isn't. You start suspecting the LLM. You swap models. Same problem. You suspect the embedding model. You swap that too. Still the same problem.

The actual culprit is usually upstream of all of it: how you split the document into pieces before you ever embedded anything. Chunking sounds like the most boring step in the RAG pipeline — just cut the text into pieces, right? — but it quietly decides how good every answer downstream can possibly be. Get it wrong and no amount of model swapping fixes it, because the model never even saw the right piece of text.

What Chunking Actually Is

Imagine explaining a long manual to someone, but you're not allowed to hand them the whole thing — you can only pass them index cards, one at a time, and they have to answer your question using only the cards you chose to hand over. Chunking is the process of deciding what goes on each card.

Cut the manual at completely arbitrary points — mid-sentence, mid-step — and a card might contain half an instruction with no context for what it's finishing. Now your reader (the LLM) is guessing. Cut it into cards so small that each one holds a single isolated fact, and the reader gets a pile of correct-but-disconnected facts with no sense of how they relate. Cut it into one giant card containing the entire manual, and you're back to handing over the whole book — which defeats the purpose of retrieval and burns a lot of context window on mostly irrelevant text.

Good chunking finds the size and the split points where each card is a self-contained, meaningful unit: big enough to carry context, small enough to stay focused and specific. That's the whole game. Everything else in this post is really just different strategies for finding that sweet spot.

The core technical pieces

Fixed-size and recursive splitting (the unglamorous default)

The most common approach just slices text into chunks of a target token count, usually with a secondary rule that tries to break on sentence or paragraph boundaries rather than mid-word — this is what LangChain's RecursiveCharacterTextSplitter does, and it's the splitter behind most RAG tutorials you've seen.

It's unglamorous, but it's also hard to beat. In a February 2026 benchmark from Vecta/FloTorch that tested seven chunking strategies across 50 academic papers (roughly 905,000 tokens), recursive splitting at 512 tokens scored 69% end-to-end accuracy — the best of everything tested. Plain fixed-size chunking at 512 tokens came in close behind at 67%. Several other 2026 guides independently converge on the same recommendation: start with recursive splitting around 512 tokens, and only reach for something fancier once you've confirmed the document type actually needs it.

Semantic chunking (where retrieval and accuracy disagree)

Semantic chunking tries to be smarter: instead of cutting at a fixed token count, it looks for places where the meaning shifts — topic boundaries — and splits there. It sounds like it should strictly beat fixed-size splitting, and on one metric it does. Chroma's research measured semantic chunking at 91.9% retrieval recall, the best of any method it tested.

But in the FloTorch benchmark, semantic chunking's end-to-end accuracy was the worst of the seven strategies tested, at 54%. The reason is a bit counterintuitive: semantic chunking in that test produced fragments averaging only 43 tokens. Each fragment matched a query well in isolation — hence the great recall — but was too thin to actually support generating a full answer. This is worth sitting with: retrieval recall and the accuracy of the final answer are not the same number, and optimizing for one can quietly hurt the other.

Overlap (the "best practice" that may not be one)

Conventional wisdom says to overlap consecutive chunks by 10–20% of their length, so a sentence that gets cut at a chunk boundary still shows up intact in the neighboring chunk. It's repeated so often it reads like settled science.

A January 2026 systematic analysis on arXiv, summarized by Digital Applied, found overlap gave no measurable benefit in its tested setup — it only increased indexing cost (more chunks to embed and store, for the same source text). That doesn't mean overlap is useless everywhere, but it's a reminder to treat "best practices" in this space as hypotheses to test against your own documents, not settled facts to copy.

Hierarchical chunking (small to find, big to answer)

Hierarchical chunking tries to get the best of both worlds by keeping two layers: small "child" chunks that are precise enough to retrieve well, and larger "parent" chunks that contain them along with surrounding context. Amazon Bedrock Knowledge Bases documents exactly this pattern — retrieval searches over the small child chunks, then swaps in the broader parent chunk before handing anything to the model. You get the precision of small chunks for finding the right spot, and the context of large chunks for actually answering.

Contextual retrieval and late chunking (chunk-level context injection)

Two more advanced techniques attack the same problem — a chunk that's missing context — from different ends. Anthropic's Contextual Retrieval, introduced in September 2024, uses an LLM to prepend a short, chunk-specific summary (roughly 50–100 tokens) to each chunk before it's embedded and indexed for keyword search. In Anthropic's own testing, this cut retrieval failure rates by about 35% on its own, about 49% when combined with contextual keyword search, and about 67% once a reranking step was added on top. In the accompanying cookbook, contextual embeddings alone moved Pass@10 from 87% to 92%, and adding reranking pushed it past 95%.

Late chunking takes the opposite approach: instead of adding context to each chunk, it embeds the entire document first, so every token's embedding already reflects the full surrounding text, and only splits into chunk-level embeddings afterward. Per a Firecrawl writeup, this improved nDCG@10 by 6.5 points on NFCorpus, a dataset of longer documents — but the benefit shrinks sharply on very short documents, where there isn't much surrounding context to lose in the first place.

A real-world example

Picture a SaaS company building an internal docs assistant on Amazon Bedrock Knowledge Bases, which ships four built-in chunking strategies — FIXED_SIZE, HIERARCHICAL, SEMANTIC, and NONE (no chunking, feed whole documents) — plus the option to write custom chunking logic via a Lambda function.

In one practitioner's writeup of testing all of these against a real support-docs corpus (a community blog post, not an official AWS benchmark, so treat the exact numbers as illustrative), a custom chunking function scored highest at 0.94 correctness, hierarchical chunking followed at 0.92, and fixed-size came in at 0.88. The NONE and SEMANTIC configurations didn't even make it to evaluation — they failed during ingestion on that corpus. Worth noting separately: choosing NONE also means you lose the ability to cite specific page numbers in answers, since there's no chunk boundary to anchor a citation to. The lesson isn't "custom chunking always wins" — it's that the "best" strategy is corpus-dependent, and some strategies can fail outright on documents they weren't built for, long before accuracy even enters the picture.

Why not just skip chunking, or go the opposite way?

Why not dump the whole document into context instead of chunking it at all? Mostly cost and noise — you pay to process every token of every candidate document on every query, most of which is irrelevant to the specific question, and you crowd out space a large context window needs for everything else happening in the conversation.

Why not chunk as small as possible, to make retrieval maximally precise? Because, as semantic chunking's 54% end-to-end score shows, a chunk that's precise to retrieve can still be too thin to answer from. Precision in finding the right spot and sufficiency in answering the question are different properties, and over-optimizing for one tends to cost you the other.

Why not just pick whatever strategy a blog post recommends and move on? Because, as the Bedrock example shows, the right strategy depends on your documents — their structure, length, and how self-contained a typical paragraph already is — not on a universal ranking.

A pitfall worth knowing

Be careful which numbers you trust, including the ones in this post. A 2025 clinical decision-support study is widely cited as showing adaptive chunking hitting 87% accuracy against a fixed-size baseline — but the baseline figure isn't consistent across the sources citing it. Some report the baseline at 50%, others at 13%, for what's presented as the same study. That's a large enough gap to change the entire conclusion, and it's the kind of discrepancy that spreads invisibly once one blog post copies another's number without checking the original paper.

The practical habit this should build: when a chunking benchmark number matters to a real decision you're making, trace it back to the primary source before you rely on it. Secondary summaries blur tiers and round numbers more than you'd expect, and chunking strategy is exactly the kind of decision where a wrong number can send you optimizing in the wrong direction for weeks.

The takeaway

Chunking isn't a preprocessing detail you set once and forget — it's a design decision with real, measurable consequences for whether your RAG system actually works, and it deserves to be tuned to your specific documents rather than copied from whichever strategy a tutorial happened to use. Start with recursive splitting around 512 tokens — it's the benchmark-validated default for good reason — and only move to semantic, hierarchical, or contextual techniques once you've confirmed your document type actually needs them, measuring end-to-end answer accuracy the whole time, not just retrieval recall. Those two numbers can disagree, and when they do, accuracy is the one that matters to the person asking the question.

Once chunking gets your retrieval landing on the right pieces of text, the next question is what you do with the pile of candidates you retrieve — because vector search alone often isn't the whole story. That's next: hybrid search and reranking, and where vector search quietly falls short on its own.