> ## Content Index
> Fetch the complete content index at: https://blog.netpy.in/llms.txt
> Use this file to discover other available public pages before exploring further.

# Why RAG systems fail in production
- URL: https://blog.netpy.in/why-rag-systems-fail-in-production/
- Published: 2026-11-18T15:52:43.000Z
- Updated: 2026-11-18T16:09:28.000Z
- Description: Most RAG failures are retrieval failures. Where chunking, recall and reranking break in production, and the three fixes that actually move the numbers.
- Author: Netpy Technologies
- Tags: AI, LLM, RAG, Engineering

Every RAG demo works. You paste a PDF, ask a question, get a clean answer with a citation. Then real users arrive — with typos, with questions that span three documents, with questions whose answer is a table on page 47 — and the system starts making things up with total confidence.

The uncomfortable finding from most of the RAG systems we review at NetPy is the same: the model was rarely the problem. Retrieval was. If the right passage never reaches the context window, no amount of prompt engineering rescues the answer.

Here is where RAG systems actually break in production, and the three fixes that move the numbers.

## The shape of a RAG request

![Diagram of a four-stage RAG pipeline with the failure mode under each stage](https://blog-minio-api.netpy.in/blog/content/images/2026/11/rag-failure-map.jpg)

Four stages, four different failure modes, and teams usually instrument only the last one.

## Failure 1: chunking by character count

The default chunker splits text every 500 tokens with 50 tokens of overlap. It is cheap, deterministic, and quietly destructive: it cuts tables in half, separates a heading from the paragraph it governs, and splits a numbered procedure between steps 3 and 4.

The result is a retrieval unit that *looks* relevant by embedding similarity but is semantically incomplete. The model then answers from a fragment.

Structure-aware chunking costs one afternoon and pays for itself immediately:

```python
from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter

header_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[("#", "h1"), ("##", "h2"), ("###", "h3")],
    strip_headers=False,
)
tail_splitter = RecursiveCharacterTextSplitter(
    chunk_size=900, chunk_overlap=120,
    separators=["\n\n", "\n", ". ", " "],   # never split mid-sentence first
)

def chunk(doc_markdown: str):
    for section in header_splitter.split_text(doc_markdown):
        # keep the heading path with every chunk — it is retrieval signal
        breadcrumb = " > ".join(section.metadata.values())
        for piece in tail_splitter.split_text(section.page_content):
            yield {"text": f"{breadcrumb}\n\n{piece}", "meta": section.metadata}

```

Two details matter more than chunk size: **carry the heading path into the chunk text**, and **never split on a separator before trying paragraph breaks**. The heading breadcrumb is often what makes the chunk retrievable at all.

## Failure 2: top-k without recall measurement

Most teams set `k=5`, eyeball a few answers, and move on. Nobody measures recall@k — the share of questions whose supporting passage is even present in the retrieved set. If recall@5 is 62%, the ceiling on answer quality is 62%, no matter what model you plug in.

You cannot fix what you do not measure, and you cannot measure without a small labelled set. Fifty real questions with their source passages is usually enough to expose the problem:

```python
def recall_at_k(cases, retriever, k=5):
    hits = 0
    for case in cases:                     # case = {"question", "gold_chunk_id"}
        ids = [c.id for c in retriever.search(case["question"], k=k)]
        hits += case["gold_chunk_id"] in ids
    return hits / len(cases)

```

Run that before and after every retrieval change. A reranker that lifts recall@5 from 0.62 to 0.81 is worth more than any prompt rewrite you will do this quarter.

## Failure 3: retrieval tuned on the wrong distribution

Demo queries are written by the people who built the index. Production queries come from users who do not know the vocabulary of the corpus. They say "the refund thing" when the document says "credit note reversal".

Two cheap corrections:

- **Hybrid retrieval.** Vector search handles paraphrase; BM25 handles exact identifiers, product codes and acronyms. Run both, fuse with reciprocal rank fusion, rerank the union. This is the single highest-leverage change in most stacks.
- **Query rewriting.** One small model call to expand the user's phrasing into the corpus's vocabulary before retrieval. Cost: a few hundred tokens. Effect: often double-digit recall gains on support-style corpora.

```sql
-- pgvector + tsvector fusion in one pass
WITH vec AS (
  SELECT id, row_number() OVER (ORDER BY embedding <=> :q_emb) AS r
  FROM chunks ORDER BY embedding <=> :q_emb LIMIT 50
), lex AS (
  SELECT id, row_number() OVER (ORDER BY ts_rank_cd(tsv, plainto_tsquery(:q)) DESC) AS r
  FROM chunks WHERE tsv @@ plainto_tsquery(:q) LIMIT 50
)
SELECT id, SUM(1.0 / (60 + r)) AS score      -- RRF, k=60
FROM (SELECT * FROM vec UNION ALL SELECT * FROM lex) t
GROUP BY id ORDER BY score DESC LIMIT 20;

```

## Failure 4: retrieval ignores document permissions

The one that turns a quality problem into a legal one. A vector index is usually a flat copy of every chunk, and the retriever ranks by similarity alone. If a junior employee asks a question that resembles a paragraph in an HR document, similarity search will happily hand it over.

Bolting an authorisation check on *after* retrieval is not a fix: by then the content has already been scored, logged, and possibly summarised into an answer. Filter **inside** the query, at the same layer as the similarity search:

```sql
-- row-level security that the retriever cannot bypass
CREATE POLICY chunk_read ON chunks
  USING (tenant_id = current_setting('app.tenant')::uuid
         AND acl_group = ANY (current_setting('app.groups')::text[]));

ALTER TABLE chunks ENABLE ROW LEVEL SECURITY;

```

Two properties matter. Permissions must be enforced by the database, not by the application code that the retriever happens to call. And they must be re-evaluated per request — caching the filtered set for an hour is exactly how stale access leaks.

Under India's DPDP Act this stops being a nice-to-have: an unauthorised disclosure through a chat interface is still a disclosure. If you cannot prove which chunks were returned to which user, you cannot answer a data-principal request.

## The three fixes, ranked by effort-to-impact

1. **Hybrid retrieval + reranking** — highest impact, roughly a day of work, no re-indexing needed.
2. **Structure-aware chunking with heading breadcrumbs** — requires a re-index, changes recall fundamentally.
3. **An eval set of 50 real questions** — the cheapest of the three and the only one that tells you whether 1 and 2 worked.

Everything else — bigger context windows, cleverer prompts, a newer model — is tuning at the margins. Feeding the right passage to an average model beats feeding noise to the best one.

## Production checklist

- Recall@k measured on a labelled set, tracked per release.
- Every chunk carries source, section path, and version.
- Retrieval logs the query, the rewritten query, and the returned chunk ids.
- A reranker in front of generation, with a latency budget you have actually measured.
- Answer-level citations that resolve to a real chunk, not a document title.
- A fallback that says "not found in the corpus" instead of guessing.

That last one is not a limitation. It is the feature that makes the other five trustworthy.

## What this looks like in practice

When we rebuild a failing RAG system, the sequence is boring and repeatable: instrument retrieval, build the eval set, fix chunking, add hybrid retrieval, rerank, then — only then — touch the prompt. Teams that start at the prompt spend months tuning a system whose ceiling is set somewhere else entirely.

If you are planning a RAG feature this quarter, start by measuring recall@5\. It is one script and it will tell you more than any vendor demo.