The finer you chop, the more context you lose; the bigger you cut, the more noise you catch

When you first build a RAG pipeline, most people spend time picking an embedding model. But the variable that actually separates good from bad retrieval quality is far more often chunking. If switching models doesn't improve recall, you should suspect the splitting strategy.

Chunking mistakes largely fall along two axes: context loss and noise injection. The two break retrieval from opposite directions.

Splitting a single sentence in half evaporates the paragraph that would be the answer

The classic case of context loss is when a chunk boundary lands in the middle of a sentence. For example, "The default is 30 seconds, and it can extend to 60 seconds depending on network conditions" is a sentence that belongs in one chunk, but if the split rule works at the sentence level, the first part and the second part end up in different chunks. For a question like "What's the maximum timeout?", the embedding only sees one chunk at a time, so 30 seconds and 60 seconds each become separate vectors. Neither fully answers the question.

Chunking by fixed token count makes this worse. When the token boundary cuts through the middle of a sentence, the first half is left with just a number and the second half with a number with no unit. Vector search doesn't find such fragments with regular expressions. Measured by similarity, a half sentence often ends up far from a question about "timeout."

To avoid this, split by sentence or paragraph, but don't cut a sentence that crosses the boundary—hand it over whole to the following chunk. For Korean documents, splitting sentences by punctuation and adding overlap so adjacent sentences overlap works well. An overlap of about 10–15% of the total chunk size is enough in most cases. Adding more barely changes the search score and only increases storage.

The bigger you cut, the answer still comes back—but buried in noise

The mistake on the opposite axis is making chunks too large. If you put an entire page into one chunk, retrieval always returns the "right" chunk. The problem is that the answer is diluted inside it. Even if you pull the top-5 and feed them to the LLM, 3,000 tokens unrelated to the answer sit in front of the question. During the generation step, if the model gets swayed by that noise, it stops citing the correct answer and settles for a vague summary.

Context loss is a recall problem; noise injection is a precision problem. Both can occur in the same pipeline. The longer a document section is, the more these two create a trade-off. Cutting by section keeps context intact but drags in unrelated parts. So when a section is long, it's better to split the section internally into paragraphs and attach the section title to each paragraph as metadata.

Size settings that ignore the embedding model's context window

One more thing here. When deciding chunk size, you should first check the embedding model's maximum input length. Context windows differ from model to model, and some models truncate the tail or average tokens when the input sequence exceeds the limit. If you feed such a model a 1,000-token chunk, only the leading context survives and half the point of splitting collapses.

Staying under the limit isn't necessarily safe either. Even with an 800-token chunk into a 1,000-token model, if the model was trained to weight the front part more, the information in the tail barely makes it into the vector. In that case, swapping to a long-context embedding model may beat shrinking the chunk size. Which one is right depends on the situation. Comparing the two configurations on a fixed set of test queries settles the question.

When evaluating splitting, look at hits, not scores

A common mistake in judging whether splitting is good is looking only at the similarity score. There's no guarantee a 0.85 similarity is better than a 0.80. Score distributions differ by model, and even with the same model they differ by document type. Instead, it's better to build a query set pairing real incoming queries with the chunks containing their answers, and judge by whether vector search brings that chunk into the top-k. Building it from real queries in the search logs is even more realistic.

Can't you just judge by generation results? That reflects whole-pipeline performance, so it's hard to isolate chunking issues alone. To evaluate only the retrieval stage, attach a source id to each chunk and compute recall@k by checking whether the source of the search results matches the ground truth. Keep k fixed at the top-k value used in production. A setup you like after cranking k up to 10 for testing is not the same as the count that actually goes into the prompt.

Good chunk design doesn't come together in one shot. Splitting rule, size, overlap, evaluation query set. Change these four variables one at a time and record recall and precision in a table. The moment you see queries with buried answers and queries with vanished answers cross over in the same table is when it's time to fix the design.

Comments

Loading comments.