Why Finer Chunking Takes the Answer Further Away

Returns only the body text.

The expectation that finer chunking makes search more accurate tends to slide in the opposite direction in a real pipeline. If you shrink chunk size to 128 tokens, the embedding vector carries only the local meaning of a sentence or two, and when you run similarity search on that vector, the query misses the paragraph that actually holds the answer entirely. What gets returned is a half-finished fragment unrelated to the correct answer.

The cause lies in the nature of embeddings. A vector compresses all the tokens in a chunk into one. The shorter the chunk, the more easily that compressed result gets swayed by a single word, and there's no room for information worth calling context. A chunk that overlaps with the query only in vocabulary scores high on similarity, while the paragraph with the actual answer sits split into a different chunk and earns no score. This failure occurs deterministically the moment a split boundary falls out of alignment with a semantic unit.

A concrete example: suppose the source text is a single sentence: "On violation, the full deposit must be paid as a penalty." If a fixed-token splitter cuts this sentence into "On violation, the full deposit" and "must be paid as a penalty," a query for "the amount charged on contract violation" retrieves the second chunk. But the core of the sentence, "full deposit," stays in the first chunk and never makes it into the context. The search succeeded, and the answer was wrong.

What matters here isn't size itself but the splitting criterion. A splitter that cuts by a fixed token count breaks in the middle of a paragraph, sometimes in the middle of a sentence. When the two halves of a split sentence are each embedded, the meaning the original single sentence carried survives intact in no vector. Token-based splitters are also sensitive to language-specific characteristics. In languages like Korean, where particles and endings attach to words, cuts tend to land off morpheme boundaries, and where a Korean document gets split varies greatly depending on how the tokenizer handles newlines and whitespace.

When Context Makes It Into the Vector

Content-based splitters beat fixed-size ones because natural boundaries like paragraphs, subheadings, and list items usually align with a complete semantic unit. They exploit the rule of thumb that a paragraph generally carries one claim. But that heuristic only works for prose documents. In documents whose body is tables, code blocks, or procedural lists, paragraph-level boundaries can actually misalign queries with semantic units. If a table cell and its explanatory sentence are split into different chunks, a chunk that retrieves only the table has no material to reconstruct context during context retrieval.

The problem of subheadings not traveling with body chunks is the same family. Splitting at a subheading leaves the subheading behind while the body moves into the next chunk. A query searching by the subheading "Termination procedure" finds the subheading chunk, but the chunk holding the actual procedure body earns no score. The alternative is to move the split boundary to just after the subheading and attach the subheading path to the body chunk as metadata. A path like "3. Contract / 3.2 Termination procedure" anchors the location of a retrieved chunk during context retrieval.

The habit of adding overlap is worth reviewing too. Using windows with 10–20% overlap to avoid losing context at neighboring chunk boundaries duplicates the same sentences across multiple chunks. The consequences show up in two ways. One is self-duplication in search results. With top-k set to 5, the same paragraph appears twice and you're left with only four real candidates. The other is dilution of the embedding. When a sentence whose context is half contained in the previous chunk gets mixed back into the new vector's average, the chunk's own meaning gets buried.

This isn't to say you should drop overlap altogether. For documents with clear section-level boundaries, context on both sides of a boundary survives without overlap. In wiki-style documents of endlessly flowing short paragraphs, overlap can be the only mechanism that preserves boundary context. It depends on the situation. The deciding factor is whether missing boundary context is actually observed in search results.

When the Search Is Right but the Answer Is Wrong

Even with perfect splitting, the correct answer can be lost between the search stage and the response stage. A pipeline that doesn't include the neighboring chunks before and after the top-k results in the context can never retrieve an answer that spans chunk boundaries. That's the case when a sentence like "The contract termination conditions are as shown in the table below" and the actual table sit in different chunks.

Candidate chunk validation becomes necessary at this point. It checks whether retrieved chunks overlap and whether a neighboring chunk from the same paragraph was dropped, and if so, retrieves it together. The simplest implementation is window expansion, appending the adjacent chunks of a retrieved chunk. The cost is clear too. Blindly appending neighbors inflates the context and can blow past the token limit, cutting off the very answer. It's better to set the token budget for the response first, then work backwards to decide how many adjacent chunks to append.

Top-k itself deserves a second look. The default of 3–5 is convenient, but with too few candidates the correct chunk never enters the pool to begin with, and with too many the token budget is drained by search. Reranking — pulling an ample set of candidates and re-ranking them — serves as a compromise between the two. You pull a few dozen initial candidates, re-rank them, and feed only the top few into the context. There's no guarantee that similarity order at the search stage is the order of answer accuracy.

Recall and Completeness, Look at Them Separately

Looking only at search recall points improvements in the wrong direction. Recall only checks whether the correct chunk made it into the top-k. Even with high recall, if the chunk's content is half-finished the response won't be complete. Conversely, looking only at response quality mixes search failures with response failures, making it impossible to tell which side to fix.

It's more practical to look at recall@k and response completeness separately. Recall is measured by whether the correct chunk is included per query, while response completeness is measured by how many of the required facts the answer contains when given the correct chunk as context. Putting both metrics in one table distinguishes whether search, context retrieval, or response generation broke down. If recall is 0.9 but completeness is 0.4, the cause is context retrieval or chunk validation, not splitting. If recall itself is 0.5, you should go back to the splitting criterion.

The evaluation query set is drawn from production data. Pick ten chunks per document, tag each with its correct chunk, and mix in queries that showed low recall over a month of traffic. Around a hundred queries is enough to compare splitting-criterion changes. Every time you change the splitting criterion, re-measure recall on the same set and open the original chunks of the dropped queries directly. Patterns show up there. If recall repeatedly drops on "graph title queries," respond by adding a rule that includes figure captions in the body chunk.

The right chunk size ultimately comes from your own data. Around 256 tokens is often cited as the standard, but it depends on the situation. For documents full of tables, it's better to set boundaries at table level even if a chunk exceeds a few hundred tokens, while documents of short consecutive paragraphs do well below 200 tokens. There's one criterion for judgment: what percentage of chunks have boundaries misaligned with semantic units? Move size and criterion together in the direction that lowers that ratio. Adjusting size alone won't eliminate misalignment with semantic units, and splitting finer won't bring the answer any closer.

Comments

Loading comments.