Common Mistakes in RAG Pipelines and How to Fix Them

Every retrieval pipeline looks fine in a demo. You point it at a dozen clean documents, ask the obvious question, and the answer comes back sharp. Then you load the real corpus — scanned contracts, two-column PDFs, a support wiki nobody has pruned since 2019 — and the same pipeline starts answering confidently with the wrong facts. Nothing throws an error. Retrieval hands over a set of plausible, adjacent, mostly irrelevant passages, and the model writes something fluent around them.

Most of these failures trace back to a handful of decisions made early and never revisited. Here are the ones that show up again and again, and what to do about each.

Chunking is the decision you will pay for later

Chunking gets treated as a setting. It is really the shape of everything downstream. A fixed-size splitter — say 1,000 characters with 100 characters of overlap — is a reasonable default and a poor final answer, because character counts know nothing about meaning. It will cut a table away from its caption, separate a heading from the list beneath it, and drop the one sentence that says which region a policy applies to.

Symptoms worth watching for:

• Retrieved chunks that begin mid-sentence, leaving the model to guess the subject.
• Answers that are directionally right but attach a rule to the wrong team, product, or year.
• Citations pointing to a passage that does not contain the figure quoted in the answer.

The fixes are mostly structural:

• Split on document structure first — headings, list boundaries, table rows, code blocks — and fall back to size-based splitting only inside a section.
• Prepend context to each chunk before embedding: document title, section heading, and a date if one exists. This single line often helps retrieval more than a fancier embedding model.
• Try small-to-big retrieval: match on a precise 200–400 token chunk, then hand the model the surrounding section.
• Sample 20 chunks by hand after every change. Ten minutes of reading catches problems no metric will flag.

Embedding mismatches

Two failure modes dominate. The first is mixing models — documents embedded with one model, queries with another, or a half-finished re-index after an upgrade. Nothing errors; results just get subtly wrong, and unevenly so, with older documents hit hardest. Treat the embedding model version as part of the index schema and re-embed everything when it changes.

The second is vocabulary. If your corpus is full of internal product names, ticket prefixes, and abbreviations, a general-purpose embedding model may place them all in roughly the same neighborhood. Fine-tuning is one option, but cheaper fixes often work: expand query terms with a short glossary, or store keywords as metadata and let hybrid search handle exact strings.

Keep the similarity metric consistent with how the vectors were produced, too. Mixing normalized embeddings with a distance function that assumes unnormalized ones is a quiet way to make scores meaningless.

Retrieval is for recall; reranking is for precision

Vector search returns nearest neighbors, and nearest is a rough proxy for relevant. It is weakest exactly where it matters: dense terminology, short queries, and corpora full of near-duplicate passages. Sending all 20 results to the model spends context on noise and raises the odds it latches onto the wrong passage.

The standard fix is retrieve wide, rerank narrow. Pull 30–50 candidates, score them with a cross-encoder or reranking model that reads query and passage together, then keep the top handful. Add a score threshold, because a reranker that always returns its full candidate list is just a slower version of what you had. And measure the change — reranking costs latency, so it needs to earn its place.

Filters, metadata, and exact-match queries

Pure vector search is bad at literal strings: error codes, SKUs, version numbers, names, dates. Keyword search is good at them. Run both and combine the results, whether through rank fusion or a weighted score.

Filters deserve the same attention. Applying permission checks after retrieval is a correctness problem, not just a security one — content can land in the context window that the person asking should never see, and you burn result slots on documents they cannot open. Push filters into the query when the store supports it, and keep a last_updated field so time-sensitive questions can favor recent material. If your corpus carries regulated or personal data, involve whoever owns compliance before you widen what retrieval can reach.

Prompt assembly is where hallucination starts

If the instruction is "answer the question using the context" and nothing else, the model has no defined behavior for missing information, so it fills the gap. Give it an explicit exit: if the context does not contain the answer, say so and name what is missing.

A few habits reduce invented detail:

• Number the passages and ask for citations by number, such as [3].
• Put the question after the context, or repeat it at the end when the context is long.
• Deduplicate near-identical chunks. Three copies of one paragraph crowd out the passage that would have answered the question.
• Check the output cheaply: do the cited passage numbers exist, and does the figure in the answer appear in the passage it cites? Simple string checks catch a surprising amount.

Fix the cheap things first

Build a small evaluation set before optimizing anything — 25 to 50 real questions, each paired with the passage that should answer it. That turns every change below into a measurement instead of an opinion.

1. Look at your chunks. If the boundaries are bad, nothing downstream will save you.
2. Normalize retrieval. One embedding model, one version, one similarity metric, fully re-indexed.
3. Add hybrid search for literal strings and identifiers.
4. Add a reranker and keep it only if the evaluation improves.
5. Tighten the prompt with an explicit "not in the context" path and numbered citations.
6. Log retrieved chunk IDs and scores for every query, so a weak answer can be traced to retrieval or to generation within a minute.

Most pipelines gain more from steps one and two than from anything you can buy. Reading your own chunks, keeping the index consistent, and measuring before and after is unglamorous work, and it is where the quality actually comes from.

Photo: Sunriseforever / Pixabay