Introduction
Building a basic Retrieval-Augmented Generation (RAG) prototype is now relatively straightforward. But designing a reliable RAG architecture for production is a different engineering challenge.
In early testing, a basic RAG pipeline is simple: you split a few documents into text chunks, generate embeddings, run a similarity search against a user question, and send the top matches to the LLM. On a small, clean test set, this retrieve-and-generate flow creates a false sense of readiness.
Once the same setup is applied to larger and more varied knowledge bases, however, its limitations become much clearer.
What is a RAG pipeline?
A RAG pipeline is a process that helps a large language model answer questions using information from external sources rather than relying only on what the model already knows.
It typically includes three main stages:
- Preparing the data - documents are collected, processed, divided into smaller sections, and stored in a searchable index.
- Finding the right information - the user’s question is analyzed, and the most relevant passages are retrieved and ranked.
- Generating a grounded answer - the selected information is passed to the LLM, which uses it to produce an answer linked to its sources.
Documents → processing and indexing → retrieval and reranking → LLM answer with sources
Why naive RAG pipeline demos fail at scale
Naive RAG pipelines often perform well in controlled demos but become less reliable when exposed to larger, more varied enterprise knowledge bases.
Pure vector search works well for broad semantic matching, but it struggles as query types diversify. It often brings back passages that match the general topic while skipping over the exact error code, policy limit, or clause needed to answer the question. It also has no native way to distinguish between current and superseded versions of the same file.
When basic RAG setups fail in production, it usually traces back to four main issues:
Irrelevant retrieval: The system finds passages that are related to the query but don’t include the information needed to answer it.
Context dilution: Redundant, irrelevant, or conflicting passages crowd the prompt and make it harder for the model to identify the evidence that matters. Long prompts can also create a “lost in the middle” effect, in which information placed in the middle receives less attention.
Missed exact matches: Embeddings handle semantic similarity well, but they frequently gloss over exact identifiers - think part numbers, error codes, dates, or legal clauses. Lexical methods such as BM25 retain strong signals from exact terms, including part numbers, error codes, dates, and legal wording.
Ungrounded generation: If retrieval returns incomplete, conflicting, or irrelevant information, the model is more likely to produce a convincing answer that the source documents don’t actually support.
Do newer AI approaches make retrieval engineering unnecessary?
When a basic setup fails, I see teams often reach for quick fixes: a larger LLM, a 1-million-token context window, GraphRAG, or an agentic framework. None of these replace sound retrieval engineering - they simply solve different problems in the stack.
Very large context windows allow a model to process more material in a single request. However, sending more text increases cost and response time, and it doesn’t guarantee that the model will use every part of the context equally well. The “lost in the middle” effect shows that important information is sometimes overlooked when it appears in the middle of a long prompt.
GraphRAG maps entities and their relationships into a graph, helping the system answer complex questions that require information from multiple documents or connected parts of the knowledge base. However, it introduces additional indexing and maintenance costs and usually complements other retrieval methods rather than replacing them.
Agentic RAG extends the retrieval workflow by using agents to decompose questions, plan multi-step tasks, and decide which tools or sources to use. These workflows still depend on search systems, APIs, databases, or knowledge graphs for reliable source information.
Retrieval is not disappearing - the industry is simply realizing that a single vector search and a basic prompt can't handle production traffic.
Naive RAG vs. advanced RAG: What changes in production?
Bringing a RAG system from prototype to production requires broader AI development consulting expertise. It means replacing the assumptions of naive RAG with a modular architecture in which document preparation, retrieval strategy, query processing, context validation, source attribution, and evaluation are designed around the organization’s data, users, and risk requirements.
Production-ready RAG implementation solutions bring these components together in a system designed around the organization’s data, users, and risk requirements.
RAG pipeline ingestion: Reducing hallucinations at the source
When engineering teams ask why a RAG system is hallucinating in production, the ingestion layer is one of the first places I investigate.
Naive setups often perform well in early testing because the documents are small, clean, and relatively uniform. Once the pipeline is connected to more varied and interconnected document repositories, arbitrary chunk boundaries can separate a heading from its explanation, a table value from its label, or a policy from its effective date.
Document processing must reflect both the structure of the source material and the questions users are likely to ask. Small chunks suit precise lookups, such as finding an error code or checking a policy’s effective date.
Broader questions, such as summarizing a long report or comparing themes across several documents, require the system to retrieve information that preserves both detailed passages and broader context.
A larger context window allows the model to process more material at once, but it doesn’t identify which passages are relevant or how they should be organized.
Preserving meaning across different content types
From what I observe, applying the same fixed-size chunking method across every content type is a common source of retrieval problems because it can break important structural relationships within the data.
Structured values: Part numbers, dates, codes, and numerical limits need to retain their precise wording and format.
Narrative documents: Structure-aware chunking preserves headings and sections, while semantic chunking keeps related ideas together.
Tables and semi-structured content: Table-aware processing preserves headers, rows, columns, units, and the relationships between values.
For example, a technical manual lists an equipment limit in one sentence and explains its unit or related safety requirement in the next. If the text is split between them, the retrieved passage no longer makes sense.
Similarly, converting a financial table into plain text leaves raw numbers intact while stripping the row and column headers that give them meaning.
Keeping retrieved passages in context
A passage that appears clear inside its original document often becomes ambiguous when retrieved in isolation. Consider this - two policy excerpts use similar wording but come from different versions, departments, or dates. Without that context, the system retrieves the wrong one or treats both as equally authoritative.
That’s why metadata such as the source, section, date, and version helps the system preserve the context correctly and trace the answer back to its origin. Control rules must also be applied during retrieval so users only receive information they’re authorized to access.
Treating retrieved content as untrusted input
Access controls enforce permissions, but they don't guarantee content safety. A retrieved document might contain malicious instructions designed to influence the model through indirect prompt injection.
The pipeline must treat all retrieved content strictly as untrusted data to analyze, not as trusted instructions the model should follow. Safeguards include source validation, content filtering, clear separation between retrieved context and system instructions, and strict limits on actions triggered from retrieved material.
Managing the index lifecycle
A retrieval system becomes unreliable when its index no longer matches the underlying source data. Updated documents need to be processed again, while deleted documents must also be removed from the retrieval index.
The required update frequency depends on how quickly the knowledge base changes and how current answers need to be.
Changing the embedding model typically requires generating new embeddings, updating or rebuilding the index, and validating retrieval quality before introducing the new version. For large knowledge bases, this creates recurring costs for computing, storage, testing, and maintenance.
For a broader introduction to how RAG uses external data to provide relevant context, see our guide to context-driven AI.
Limitations of vector-only RAG pipelines
A naive setup sends every prompt to the same vector index. That works fine on predictable test sets, but enterprise traffic is rarely uniform - different categories of questions require completely different retrieval pathways:
- Broad conceptual queries (like policy guidelines) rely heavily on semantic similarity and thrive on standard vector search.
- Exact-identifier lookups (for part numbers, error codes, or contract clauses) need strong lexical matching. Pure vector search routinely ranks these terms too low when surrounding passages use similar language.
- Relational questions, such as tracing supply-chain dependencies across multiple entities, benefit from graph-based retrieval such as GraphRAG when the answer depends on multi-hop connections.
- Live operational checks such as current inventory or account balances, should retrieve the relevant values from an authoritative database or real-time API rather than relying on a static document index.
To improve retrieval for questions that depend on both meaning and precise terms, production RAG pipelines often combine dense vector retrieval with lexical methods such as BM25, which preserve exact words, identifiers, and specialist terminology.
Even an appropriate search method is insufficient when the request itself is vague, overloaded, or expressed in language that’s different from the source material. Query rewriting clarifies vague wording or internal shorthand, while query decomposition breaks a complex question into smaller searches so the relevant information is retrieved separately and then combined.
Unchecked retrieval results and hallucinations
The highest-ranking search results aren’t always ready to be passed directly to the language model. Search systems rank information by estimated relevance, but a high-ranking result may still be incomplete, outdated, repetitive, or insufficient to answer the question.
Production systems must check not only whether a passage is related to the topic, but if it contains the information required, reflects the correct source version, and supports a traceable answer.
Reranking passages to filter misleading context
Initial retrieval often returns several passages that use similar terminology but differ in how well they answer the specific question. Reranking compares those passages more closely with the query and prioritizes the ones that contain the information needed for the answer, rather than those that merely mention the same topic.
This lets us strip out weak, repetitive, or loosely relevant results before they ever hit the model’s context window.
For example - a search for an engineering tolerance may retrieve an installation guide, a maintenance checklist, and the technical specification containing the actual limit. All three relate to the equipment, but only the specification answers the question. Reranking moves that passage above the broader supporting material, while filtering removes results that add no useful evidence.
Resolving outdated or conflicting sources
Enterprise knowledge bases often contain outdated policies, incomplete records, or conflicting document versions.
Take a classic enterprise edge case: a search query brings back a 2022 travel policy (PLN 200/day allowance) alongside the revised 2026 policy (PLN 300/day). Vector search sees both as semantically identical hits. Without strict versioning and date metadata built into the retrieval layer, the model has no way of knowing which rule governs today's expense reports.
This is why I insist on making effective dates, document versions, and source authority first-class metadata attributes.
When retrieval returns weak or conflicting information, the system should assess the retrieved evidence before generating an answer. Corrective RAG does this by assessing the quality of the retrieved documents. If the evidence is insufficient, it triggers another search,refines or replaces the retrieved context.
Source attribution and traceability
Even correctly selected information isn’t enough if users don’t see which sources supported the final answer. Important claims should remain connected to the documents, sections, pages, or records used to generate them.
This allows them to verify the response and makes later review easier. Detailed attribution is especially important in regulated, legal, financial, security, and technical environments. For lower-risk internal assistants, document-level attribution is usually sufficient.
Retrieved information shouldn’t be accepted automatically. A production pipeline must check whether the evidence is relevant and complete, identify missing or conflicting information, and preserve a clear path from the final answer back to its sources.
System latency and cost bottlenecks
Passing every incoming prompt through the same heavy pipeline is the fastest way I know to destroy response times and blow up token budgets. Multi-step reasoning and unconstrained agent loops add failure points that make debugging nearly impossible.
Semantic caching for latency reduction
Semantic caching reduces latency and cost by storing previous answers and reusing them when a new question has the same or very similar meaning. Unlike exact-match caching, it recognizes when differently worded requests, such as “How many remote-work days are allowed?” and “How often can employees work from home?”, require the same underlying information.
Cached responses must still account for source updates, access permissions, model and prompt changes, and freshness requirements. Without these checks, there’s a risk the cache will return outdated, unauthorized, or contextually incorrect information.
Adaptive processing paths
A production system should apply only the processing needed to answer each request reliably:
- Narrow lookups typically require one targeted retrieval step.
- Broader comparisons often need several searches, evidence synthesis, and additional validation.
- Ambiguous requests need query clarification or decomposition before retrieval.
- Higher-risk requests require stricter source checks, confidence thresholds, or human review.
Adaptive RAG is one way to route each request through the appropriate level of retrieval and validation, keeping simple questions fast while assigning more complex ones additional processing.
For tasks that genuinely require multi-step planning, agentic RAG coordinates several searches, data sources, or tools. That flexibility increases latency, cost, and the number of possible failure points, so it shouldn’t be the default for every request.
Evaluate production performance over time
The quickest way to let a RAG system degrade is to treat evaluation as a one-time launch task. Changes to source files, chunking, routing rules, or prompts reduce answer quality or increase latency and cost. Evaluation should separate retrieval from generation so teams are able to identify where a failure happened:
- Retrieval quality: Did the system find the evidence needed to answer the question? For example, did it retrieve the current pricing table or return an outdated version?
- Generation quality: Did the model use the retrieved evidence accurately? For example, given the correct table, did it report the rates without adding unsupported fees?
Production monitoring covers citation quality, response latency, token usage, and user feedback.
These signals reveal whether answers remain traceable, if more complex retrieval paths are becoming too slow or expensive, and where users repeatedly fail to find useful information.
When the available information is missing, weak, or contradictory, the system searches again, asks for clarification, or declines to answer rather than produce an unsupported response.
The production lesson I learned is that RAG quality isn’t established through a one-time demo. It must be measured as the data, architecture, and usage patterns change.
When can you trust RAG in production?
Bringing a RAG application from proof of concept into an enterprise environment takes more than a larger language model or a single new feature. Retrieval must be engineered as a modular, multi-stage process rather than treated as one script.
Small, clean test sets sometimes create a false sense of readiness because they hide the complexity of enterprise data. Production readiness depends on whether the system is able find the right information, handle different types of requests, show where its answers came from, and respond safely when the available evidence is insufficient.
A trustworthy RAG system connects important claims to authoritative sources, makes those sources visible, and declines to answer when the evidence does not support a reliable response.
