When Search Keeps Pulling the Wrong Documents, What to Suspect Before Embeddings

The signal that something is wrong with search first shows up when the wrong document climbs to the top. You ask about "refund policy" and a shipping policy document comes back. At this point most people swap out the embedding or fiddle with the vector DB's index settings. Dig into the logs and the cause is much further upstream. The sentence containing the answer is already cut into two pieces, or its preconditions are scattered across other pieces.

Chunk splitting is the quietest step in the pipeline. Embeddings and retrieval are evaluated by scores, but splitting is treated as preprocessing. When something goes wrong, no error fires. Retrieval returns results well enough, and quality just erodes little by little. The problem is that this degradation accumulates. As the top documents drift slightly out of alignment, the entire answer drifts with them.

Fixed-length cuts kill sentences first

The most common mistake is mechanically cutting at a fixed length like 500 characters or 1,000 tokens. When a cut lands mid-sentence, one piece loses its subject and the other loses its conclusion. The embedding treats the severed fragment as a complete unit of meaning. When "Returns are accepted within 7 days of receipt" splits into "Returns are accepted within" and "7 days of receipt," neither piece matches the question precisely. The vectors for the two pieces land in different places than the original sentence.

Overlap patches some of the cutting damage. In exchange, storage cost and retrieval duplication both rise. Set overlap to half the chunk and the same sentence gets retrieved twice, filling the top ranks with duplicate results. As a rule of thumb, prioritize paragraph or sentence boundaries, and only apply overlap of around 10–20% when boundaries are ambiguous. In table terms, overlap is closer to a bandage. It covers the wound but doesn't set the bone.

When the embedding quietly throws away the tail

Fixed-length splitting has a more insidious failure mode. A chunk that exceeds the embedding input's max tokens has only its front portion turned into a vector, while the rest is silently truncated. Put a 1,000-token chunk into a 512-token limit and the back half disappears from retrieval entirely. No error, no warning. To check your documents, plot the token-count distribution as a histogram and count the share of chunks that exceed the limit. If it's over 5%, that much context is already being thrown away.

On top of this, language-specific pitfalls compound. Korean produces more tokens than English for the same character count, depending on the tokenizer. A "split at 1,000 characters" rule can stretch to 1,200–1,500 by token count. To respect a token limit, you have to count with the actual tokenizer, not character count. Character-based rules keep passing chunks through without ever detecting that they exceed the limit.

The moment you pull tables and headings apart

Ignoring document structure is nastier still. When the heading "3.2 Return Fees" lands in a different chunk from the body, the body fragment loses what it is about. Same with tables. If the header row is separated from the body, only a cell like "Round-trip shipping fee: 3,000 KRW" remains, and what that 3,000 means vanishes. Retrieval can find the number but not the context.

Structure-aware splitting isn't hard. For Markdown or HTML, split along heading levels, and prefix each chunk with its parent heading path.

# When building a chunk, prepend parent headings as a breadcrumb
chunk_text = f"{section_path}\n\n{body}"  # e.g. "Chapter 3 Returns > 3.2 Fees"

That one line often changes recall noticeably. It's because the question and the chunk share the same vocabulary. For tables, duplicating the header as the first row and attaching it to every row group reduces the disconnect. To an embedding, a headerless blob of numbers is basically noise.

The document decides the size

The moment you unify chunk size, you start fighting the document structure. In a contract, where clause units are clear, one clause is the natural chunk. In step-by-step instructions, a single step spans multiple paragraphs, so you need to capture the whole step. In something like an API reference with deep hierarchy, it is better to bundle one function together with its description.

More important than size itself is "can the question be answered by reading this chunk alone?" If the conditions needed for the answer live in a neighboring chunk, that split has failed. Whether it's 200 tokens or 800, the criterion is whether a single chunk forms a self-contained meaning.

There is also an approach where you retrieve with small chunks and pass the parent paragraph of the retrieved fragment as context. This is the parent-child structure. Retrieval precision goes up, but you have to manage two indexes and latency increases. If documents are short and structure is simple, it's an overkill choice.

Semantic splitting isn't a silver bullet either

Semantic chunking, which computes sentence embeddings and cuts where similarity drops sharply, is said to beat fixed-length splitting. But its behavior is determined by a single threshold. If the threshold is low, the document scatters into sentence units; if high, sections of different character clump into one chunk. On top of that, every document ingestion triggers extra embedding calls, raising indexing cost.

The decision criterion is simple. If you have a lot of documents and paragraph lengths vary widely, the semantic approach has the advantage. If most documents are structured and have clear boundaries like clauses or steps, structure-based splitting is cheaper and more predictable. When mixing the two, you should first check that the boundary rules don't conflict.

Recall isn't visible to the eye

When discussing retrieval quality, an "it feels better" judgment is risky. You need at least a minimal measurement setup. Label the correct chunks for 50–100 questions and compute recall@k and MRR. If recall@5 is 0.6, that means only 60% of the top 5 contain the correct chunk. Looking at the rank distribution of retrieved correct chunks alongside it gives you a clue to distinguish a splitting problem from an embedding problem.

With small samples, the numbers wobble. A 5%p recall difference measured with 20 questions gets buried in noise. At least 50, and preferably 100, is recommended. If the correct chunk is never retrieved at all, it's a splitting or embedding problem. If it's retrieved but ranks low, it's a ranking or reranking problem. Without this distinction, swapping the embedding first just burns cost while the cause remains.

If you don't have correct-chunk labels, marking questions and correct documents at the document level still diagnoses half the issue. If document-level recall is low, it's very likely a fragment problem.

What to check before changing anything

Before touching the splitting method, first build a set of 50 questions and correct-chunk labels. Then see whether recall@5 moves by 10%p or more. If there's no change that big, hold off on swapping the embedding, and apply low-cost corrections first, like attaching heading paths and duplicating table headers. Adjusting chunk size comes last. Most retrieval failures start at the boundary, not the size.

Comments

Loading comments.