Content Chunking Strategies for Retrieval-Augmented Generation

Retrieval-augmented generation lives or dies on one decision most teams make in the first week and revisit only after everything else has failed to fix the real problem: how do you cut a document into pieces? The pipeline itself is simple enough to fit on a napkin: retrieve, embed, generate. But chunking is the step that happens before any of that, and it sets a hard ceiling on what the rest of the system can ever do.

Here's the mechanical reality nobody argues with. Chunking takes continuous discourse, prose that flows from one sentence to the next, and cuts it into discrete spans that get embedded and indexed separately. Once that cut is made, it's made. No reranker recovers a pronoun whose antecedent got sliced off into the previous chunk. No clever prompt reconstructs an acronym whose definition landed forty tokens on the wrong side of a split. No retrieval algorithm, however well-tuned, retrieves a proposition that was itself fragmented across an arbitrary token boundary. These are the three failure modes that show up over and over in chunking research, and they all trace back to the same root cause: the split happened without regard for where meaning actually lived in the text.

This is where the granularity tension enters, and it's worth sitting with because it never fully goes away no matter which strategy gets chosen. Large chunks drag in irrelevant content alongside the relevant part, diluting the embedding and confusing the retriever about what the chunk is even about. Small chunks lose the surrounding context that would let a language model interpret them correctly, and that's precisely the condition under which hallucination creeps in. A systematic evaluation out of Wrocław University (Śmigielski et al., arxiv 2606.00881) frames this tension directly, and it's the throughline for nearly everything that follows in this piece. Teams at BBVA have reported spending three to four months per use case just iterating on chunking, before touching embeddings, rerankers, or prompts. That's not a rounding error in a project timeline. That's most of a fiscal quarter spent deciding how to cut up a PDF.

What follows walks through the major chunking strategies in roughly the order teams tend to discover them: what each one optimizes for, where it quietly falls apart, and which conditions actually favor it. None of them wins universally. That's sort of the whole point.

Fixed-size chunking: what it gets right and where it runs out

Fixed-size chunking does exactly what it sounds like: pick a token or character count, usually somewhere between a few hundred and around a thousand tokens, and slice the document at that interval, sometimes with a bit of overlap so context doesn't vanish entirely at the seams. It's the chunking equivalent of cutting a baguette into even slices without checking where the crust is thickest.

Teams reach for it first for reasons that have nothing to do with retrieval quality and everything to do with convenience. It's cheap to compute, trivial to implement, deterministic, and fast to index. If a document is homogeneous, short sentences, consistent topic density throughout, fixed-size chunking performs adequately enough that nobody notices the seams.

The trouble starts with anything long-form, technical, or semantically layered, where sentences depend on each other across long stretches. A cross-domain evaluation by Shaukat et al. (arxiv 2603.06976) put simple fixed-size character chunking through its paces and found it scored below an nDCG@5 of 0.244 and a Precision@1 of roughly 2 to 3 percent, the weakest results across the entire study. That's not a marginal underperformance. That's a strategy failing on its core job across nearly every domain tested.

None of which means fixed-size chunking deserves to be thrown out entirely. It's a prototyping baseline, a way to get a RAG pipeline running on day one so there's something to measure against. Treating it as a production strategy, though, is a bit like treating a first draft as a final manuscript because the word count looked right. The natural next step, and the one most engineering teams take almost immediately, is recursive character splitting.

Recursive character splitting as the practical default for most teams

Recursive splitting adds one idea to fixed-size chunking: try to respect the document's own structure before falling back to a brute-force cut. It attempts paragraph breaks first, then sentence boundaries, then word boundaries, only resorting to finer splits when a chunk still runs over the target size. It's fixed-size chunking with a bit of manners.

The sweet spot reported across engineering guides sits around 400 to 512 tokens with 10 to 20 percent overlap, numbers that show up often enough in production configs to be treated as a reasonable starting default rather than a magic formula. And here's the part that tends to surprise people who assumed more sophistication always wins: a NAACL 2025 Findings paper (Qu, Tu, and Bao) found that fixed 200-word chunks matched or beat semantic chunking across both retrieval and answer generation tasks. The cheap, boring approach held its own against a method built specifically to be smarter.

Separate research backs this up from another angle entirely. A large-scale systematic evaluation found that recursive token-based chunking consistently outperformed fancier alternatives while asking for a fraction of the compute. So the pattern isn't confined to one paper or one domain.

Recursive splitting still has a ceiling, though, and it's worth naming plainly: it respects structure, not meaning. It knows where a paragraph ends. It has no idea whether that paragraph switches topics halfway through. Topic-straddling splits still happen, they just happen less often than with naive fixed-size cuts. For teams working with reasonably structured documents where latency and index cost actually matter, this is the sane starting point. Move past it only once retrieval metrics reveal a gap worth the extra engineering.

Semantic chunking: the case for topic-coherent splits and when the cost isn't justified

Semantic chunking tracks embedding similarity between consecutive sentences and splits wherever that similarity drops sharply, on the theory that a topic shift shows up as a dip in how similar two neighboring sentences look to an embedding model. Each resulting chunk should, in theory, cover one coherent idea rather than a paragraph that wanders from topic to topic. LlamaIndex's SemanticSplitterNodeParser is a widely used implementation for testing this out.

The appeal is obvious on paper. Some evaluations show semantic chunking improving recall by up to 9 percent over simpler methods. But that gain doesn't come free: it requires embedding every single sentence at indexing time, which multiplies the compute cost of building the index in the first place.

A Vecta benchmark from February 2026, testing seven chunking strategies across 50 academic papers, complicates the picture considerably. Recursive 512-token splitting came out on top at 69 percent accuracy. Semantic chunking landed at 54 percent, and its chunks averaged just 43 tokens, small enough that they'd lost the contextual continuity that made them worth reading in the first place. Splitting too aggressively at every topic boundary, it turns out, can produce fragments so small they no longer carry enough context to be useful, which is a strange kind of failure for a method whose entire premise is preserving topic coherence.

So the real question isn't whether semantic chunking helps. It's whether the gain clears the cost. If recursive splitting delivers 88 percent recall and semantic chunking delivers 91 percent, is that 3-point bump worth the substantially higher processing and embedding overhead? For most corpora, probably not. Semantic chunking earns its keep specifically when topic coherence genuinely varies across a document and when retrieval on topic-boundary queries is measurably weak under simpler methods, not as a default upgrade applied everywhere out of habit.

The tension semantic chunking only half-solves, precision versus preserved context, is exactly what hierarchical chunking was built to address, not by choosing one over the other, but by refusing to choose at all.

Hierarchical (parent-child) chunking and why it became the dominant production pattern

Here's the insight that hierarchical chunking is built on: precision and context aren't actually in conflict if retrieval and generation are allowed to operate at different granularities. Index small chunks for matching. Return large chunks to the model for generation. Problem, in theory, solved.

The structure works like this: small child chunks get embedded and searched against, but when a child chunk matches a query, its parent, a larger document section that contains the child plus its surrounding context, gets handed to the language model instead of the tiny fragment. A 2026 implementation called H-RAG (SemEval 2026, arxiv 2605.00631) shows the pattern in concrete detail: documents get segmented into overlapping three-sentence child chunks with a two-sentence stride, indexed alongside their parent documents in a Weaviate hybrid vector store weighted 70 percent dense and 30 percent sparse (α=0.7). Child embeddings run through BAAI/bge-large-en-v1.5, with reranking handled by BAAI/bge-reranker-v2-m3. LlamaIndex's HierarchicalNodeParser implements a version of this same pattern with comparatively little setup work, which is part of why it's become close to a default choice in production RAG systems.

But hierarchical chunking has a specific failure mode worth naming, and it's a little unsettling once noticed: child chunks can turn into what researchers call "semantic islands." If a child chunk doesn't carry enough intrinsic context on its own, even successfully retrieving its parent doesn't necessarily restore the referential links that got severed. The parent is there. The meaning that used to connect the fragment to it isn't automatically resurrected just because the container got bigger.

Merola and Singh quantify what's lost when parent context gets stripped away entirely. Their benchmark found that replacing document-level hybrid retrieval with chunk-level search alone produced a measurable drop in both Precision@1 and MRR. The numbers aren't dramatic in isolation, but the direction is consistent and the mechanism is exactly what the semantic-islands problem predicts. Hierarchical chunking is a sound default for structured, multi-section documents in production. The semantic-island issue is the signal that it's time to layer contextual enrichment on top, which is precisely what the next two techniques attempt.

Late chunking and contextual retrieval: two ways to inject global context after the split

Both of these methods start from the same complaint about everything discussed so far: splitting first and embedding second throws away global context before the embedding step ever gets to see it. They just solve it in opposite directions.

Late chunking (Günther et al., 2024) flips the order entirely. Embed the whole document first, at the token level, using a long-context model, and only apply chunk boundaries afterward through mean pooling. Context from the entire document flows into every chunk's embedding before the chunk technically exists as a separate unit. It's a clever trick, and it shows measurable gains, measurable gains of up to 10 to 12 percent in retrieval accuracy on documents with anaphoric references, cases where a pronoun points back to something mentioned paragraphs earlier.

Late chunking comes with hard constraints, though, not soft suggestions. It needs a long-context embedding model, at least many thousands of tokens of capacity, because the whole point is encoding the full document before splitting. Standard models like E5-v2 or BGE-v1.5, capped around 512 tokens, simply can't do the job, their sequence-length limit prevents full-document encoding outright. And it only works with mean-pooling architectures; CLS-pooling models perform poorly under this approach. So the entry ticket is a specific kind of embedding model already sitting in the stack, not something bolted on casually.

Contextual retrieval, published by Anthropic in September 2024, solves the same underlying problem from the opposite direction. Instead of embedding the whole document up front, it has an LLM generate a short context summary, typically 50 to 100 tokens, and prepends that summary to each chunk before embedding, explaining in plain terms what role that chunk plays within the larger document. Reported gains run 5 to 15 percent in retrieval precision across varied datasets, and when paired with BM25 and reranking, the reduction in top-20 retrieval failures reaches as high as 67 percent. The cost is real, though: approximately $1.02 per million tokens of source document in preprocessing, since every chunk needs its own LLM-generated summary before indexing even starts.

Merola and Singh, writing in Springer Nature's Knowledge-Enhanced Information Retrieval (2026), put the trade-off plainly: contextual retrieval preserves semantic coherence more effectively but costs more compute, while late chunking is more efficient but tends to sacrifice relevance and completeness. So which one fits? Late chunking makes sense when a long-context embedding model is already part of the stack and anaphora is a known, specific failure mode in the retrieval logs. Contextual retrieval makes more sense when coherence across a wide range of query types matters more than preprocessing cost.

Proposition and agentic chunking: maximum granularity and its limits

What happens if chunking goes as fine-grained as it possibly can? Proposition chunking answers that question by breaking content down into self-contained atomic facts rather than spans of text at all, aiming for maximum precision in fact-level retrieval. It's chunking taken to its logical extreme, one idea, one unit, no exceptions.

It doesn't work as well as the theory suggests. A 2025 multi-hazard study (arxiv 2511.14010) found proposition-based chunking averaging 72 percent accuracy, well below the 94.5 percent achieved by agentic chunking in the same evaluation. Excessive granularity, it turns out, weakens contextual continuity and makes the whole system oddly sensitive to how a query happens to be phrased. Break something into small enough pieces and it stops carrying the connective tissue that made it findable in the first place.

Agentic chunking takes a different route to fine granularity: instead of a fixed rule slicing at atomic facts, an LLM reads the content, combines related propositions, summarizes where useful, and decides the chunk boundaries itself. The model, not a formula, decides where one idea ends and the next begins. In that same multi-hazard study, agentic chunking topped out at 94.53 percent accuracy, the highest result in the evaluation and roughly 4 percentage points above fixed-token and paragraph-based methods.

The catch is exactly what anyone would guess from "an LLM reads every document and makes chunking decisions one at a time." It's expensive at scale, and it remains largely experimental rather than a standard production pattern. The same per-document LLM call that makes agentic chunking accurate is the same call that makes it slow and costly to run across a large corpus.

Put proposition and agentic chunking side by side and a pattern emerges that applies well beyond just these two methods: granularity by itself doesn't determine quality. What separates a genuinely good chunking strategy from a merely fine-grained one is whether it preserves local context while still achieving precise matching. Cutting finer isn't automatically cutting smarter.

Query-adaptive and mixture-of-granularity approaches: routing the decision dynamically

At this point a fair question surfaces: if every strategy above has a domain where it wins and a domain where it quietly falls apart, why commit to just one? The Shaukat et al. cross-domain evaluation (arxiv 2603.06976) found exactly that kind of domain dependence baked into the data: dynamic token sizing performed best in biology, physics, and health, while paragraph grouping won out in legal and math documents. No single strategy dominated across the board. That finding is the entire justification for query-adaptive and mixture approaches, which treat the chunking decision as something to be made at retrieval time rather than locked in at indexing time.

Query-Adaptive Semantic Chunking, or QASC (Rastogi, arxiv 2605.22834), folds the user's actual query into the segmentation process itself, rather than pre-deciding chunk boundaries before anyone asks anything. It works through three mechanisms: cosine similarity scoring between sentence and query embeddings to find seed sentences relevant to the question, contextual window expansion around those seeds to keep local coherence intact, and chunk-level score aggregation to judge overall relevance. Tested across 100 technical documents and 40 queries spanning four query types, QASC hit an F1-score of 0.85, an 18 to 27 percent relative improvement over fixed chunking baselines and 8 to 12 percent over semantic and agentic alternatives. Three independent annotators validated the results by hand, landing at a Cohen's κ of 0.82, a solid level of agreement for this kind of manual check.

Mix-of-Granularity (MoG) takes a related but distinct approach: segmenting documents at multiple granularity levels to pick which level actually gets used at retrieval time. Mixtures of Chunking Learners, or MoC, goes further still, combining multiple chunking approaches to generate candidate segmentations, then routing to a meta-chunker chosen based on chunk-quality metrics. The chosen small-language-model-based meta-chunker is trained specifically to predict structured chunk boundaries so it learns boundaries rather than content, which cuts overhead while improving semantic alignment. Related work under the Meta-Chunking framework introduces dynamic approaches to boundary detection and chunk construction designed to repair semantic discontinuities after the initial split has already happened.

What all of these share is a shift in how chunking gets treated conceptually: not as a fixed architectural parameter decided once during setup, but as a learned or query-conditioned function that adapts per request. That's clearly the direction the research is heading. It's also, at the moment, mostly research. Routing and mixture approaches add real infrastructure complexity, and most aren't available as off-the-shelf components in the major RAG frameworks teams already use. Worth watching, not yet worth betting a production system on for most teams.

How to read benchmark results when selecting a strategy

Every benchmark cited throughout this piece points at the same uncomfortable conclusion: results don't transfer across domains, and treating any single number as universal is how teams end up rebuilding their chunking pipeline twice.

The Shaukat et al. cross-domain evaluation (arxiv 2603.06976) found dynamic token sizing winning in biology, physics, and health, while paragraph grouping won in legal and math. The top performer overall, Paragraph Group Chunking, reached a mean nDCG@5 of about 0.459 and Precision@1 of roughly 24 percent, numbers that sound solid until remembering they came from a specific mix of domains that may have nothing in common with whatever corpus a given team is actually indexing.

Chemistry data makes the same point from a different angle. A large-scale systematic evaluation of chunking for RAG, covering numerous configurations across multiple embedding models, found recursive token-based chunking consistently winning, a result that looks almost nothing like the general-domain leaderboard above it. And in clinical decision support, Gomez-Cabello et al. (2025) found fixed-token chunking falling meaningfully short of the accuracy that adaptive chunking achieved (50 percent versus 87 percent), a gap the researchers confirmed statistically. That's not a rounding difference. That's the difference between a clinical tool that's useful and one that isn't, decided almost entirely by a chunking parameter set months before anyone touched the model.

So before adopting any benchmark's conclusion wholesale, a couple of questions are worth asking honestly. Does the test corpus actually resemble the document type and length distribution in question, or is it a convenient stand-in that happens to be publicly available? Does the query set reflect real usage patterns, factual lookups, comparative questions, multi-hop reasoning, or does it test something narrower than what users will actually ask? Chunking benchmarks are useful precisely because they show what's possible in a given setting, not because any one of them hands over a universal answer. The strategy that wins depends on the document, the query, and the domain, which is really just the granularity tension from the opening section, still unresolved, still the thing worth testing before committing to anything in production.