From naive retrieval to agentic pipelines: a practitioner’s map of every RAG variant, what each one actually solves and where the next generation is heading.
Most developers who have shipped a RAG system remember the moment their confidence cracked.
The demo worked beautifully. The production system returned irrelevant chunks, ignored the right documents and confidently answered a question with information that was not in the corpus at all. The retrieval pipeline looked fine. The LLM looked fine. The combination of both, under real query load, was another thing entirely.
That experience is so common it has become a genre. And it happens because “RAG” has come to mean at least six different architectures, each with different failure modes, each requiring different engineering decisions. Building a naive RAG system and calling it production-ready is like buying a car tyre and calling it a vehicle.
This article maps the full RAG family. What each type solves, what it costs and where a practitioner should reach for each one. Then, at the end, some ideas that do not exist yet but probably should.
A Brief History of How We Got Here
RAG as a term comes from a 2020 paper from Meta AI, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” which combined a pre-trained retriever with a generator to ground language model outputs in external documents. The core insight was simple: a model that can look things up makes fewer things up.
That idea was correct. The implementation has been getting more sophisticated ever since.
The RAG Family: Six Core Types
1. Naive RAG
The original form. You chunk your documents, embed them, store the vectors, receive a query, embed the query, retrieve the top-k closest chunks by cosine similarity and stuff them into the context window.
# Naive RAG: the basic pattern
def naive_rag(query: str, vectorstore, llm, k: int = 5) -> str:
# Embed the query and retrieve top-k chunks
docs = vectorstore.similarity_search(query, k=k)
context = "\n\n".join([doc.page_content for doc in docs])
prompt = f"""Use the following context to answer the question.
Context:
{context}
Question: {query}
Answer:"""
return llm.invoke(prompt)What it solves: Getting a language model to answer questions about documents it was not trained on.
Where it breaks: Fixed chunk sizes lose semantic coherence. Cosine similarity is a poor proxy for relevance on abstract queries. No mechanism exists to catch when retrieved documents are wrong or irrelevant. Late in 2024, the “lost in the middle” phenomenon (where models struggle to use information in the middle of long context windows) added another documented failure mode.
When to use it: Prototypes, internal tools with a clean well-structured corpus and questions that map well to exact keyword or semantic matches.
2. Advanced RAG
Advanced RAG wraps the same core pipeline with optimisation at three stages: before retrieval, during retrieval and after retrieval.
Pre-retrieval improvements focus on data quality and indexing. Metadata enrichment, sliding window chunking, parent-child chunk relationships and hierarchical indexing all address the same problem: naive chunking destroys context.
Retrieval improvements include hybrid search (combining dense vector similarity with sparse BM25 keyword matching), query rewriting (using an LLM to rephrase the query before retrieval) and multi-query retrieval (generating several query variants and aggregating results).
Post-retrieval improvements include reranking (using a cross-encoder to re-score retrieved documents against the query), contextual compression (stripping irrelevant sentences from retrieved chunks) and Maximum Marginal Relevance (MMR) selection to reduce redundancy across retrieved chunks.
# Advanced RAG: hybrid search + reranking example
from langchain.retrievers import EnsembleRetriever, BM25Retriever
from langchain.retrievers.document_compressors import CrossEncoderReranker
def advanced_rag(query: str, docs, vectorstore, llm) -> str:
# Hybrid: dense + sparse retrieval
bm25 = BM25Retriever.from_documents(docs, k=10)
dense = vectorstore.as_retriever(search_kwargs={"k": 10})
ensemble = EnsembleRetriever(
retrievers=[bm25, dense], weights=[0.4, 0.6]
)
# Rerank retrieved documents
reranker = CrossEncoderReranker(model_name="cross-encoder/ms-marco-MiniLM-L-6-v2", top_n=4)
retrieved = ensemble.invoke(query)
reranked = reranker.compress_documents(retrieved, query)
context = "\n\n".join([doc.page_content for doc in reranked])
return llm.invoke(f"Context:\n{context}\n\nQuestion: {query}\nAnswer:")When to use it: Most production systems should start here rather than at naive RAG. The marginal cost over naive RAG is low and the reliability improvement on messy real-world corpora is significant.
3. Modular RAG
Modular RAG treats the pipeline as a set of composable modules: retriever, reranker, filter, memory, router and fusion. You wire them together based on the task rather than fixing a single pipeline for all query types.
A routing layer decides which module combination to use. A financial question might trigger a different retrieval path than a general knowledge question. A question requiring multi-document synthesis triggers a different post-processing chain than a factual lookup.
The cost is orchestration complexity. Modular RAG is harder to debug because the execution path varies. But for systems that serve heterogeneous query types, it is the only approach that avoids cramming everything through one pipeline.
When to use it: Enterprise knowledge bases with multiple distinct document types, query intents or user contexts.
4. Graph RAG
In April 2024, Microsoft Research published “From Local to Global: A Graph RAG Approach to Query-Focused Summarization” (arXiv:2404.16130). The paper introduced a fundamentally different retrieval strategy: instead of searching over document chunks, search over a knowledge graph built from the corpus.
The pipeline extracts entities and relationships from source documents using an LLM, then runs the Leiden community detection algorithm on the resulting graph to identify clusters of related concepts. Each cluster gets a pre-generated summary. At query time, the system retrieves community summaries rather than raw document chunks.
# Graph RAG: conceptual structure (using Microsoft's graphrag library)
# pip install graphrag
# 1. Index phase: build the knowledge graph from your corpus
# graphrag index --root ./my-project
# 2. Query phase: global search uses community summaries
# graphrag query --root ./my-project --method global --query "your question"
# The library handles entity extraction, graph construction,
# Leiden clustering and summary generation automatically.What it adds over standard RAG: Graph RAG handles “global” questions: the kind that require synthesising patterns across an entire corpus rather than locating specific facts. “What are the recurring themes in these 500 engineering post-mortems?” is a question naive RAG cannot answer. Graph RAG can.
What it costs: The indexing phase is expensive. Every document passes through an LLM for entity extraction. The graph construction and community summarisation add further cost. It is not a real-time solution. It is a pre-built index strategy.
When to use it: Long-form analytical corpora (research archives, legal document sets, large internal knowledge bases) where users ask synthesis and pattern questions, not just lookup questions.
5. Self-RAG and Corrective RAG (CRAG)
These two are best understood as quality control layers on top of any retrieval strategy.
Self-RAG (2023) introduces a critic model that generates reflection tokens at inference time: should I retrieve? Is this retrieved passage relevant? Is my answer supported by the retrieved content? The model decides whether to retrieve at all, evaluates what it got and flags whether its output is grounded. Rather than always retrieving, it retrieves only when it judges retrieval will help.
Corrective RAG (CRAG) (arXiv:2401.15884) runs a lightweight evaluator on each retrieved document. If the evaluator scores documents as ambiguous or irrelevant, CRAG triggers a web search fallback before generation. It also applies a decompose-then-recompose operation to strip irrelevant sentences from partially useful documents.
Both address the same failure mode: retrieved garbage in, confident garbage out. They are the RAG equivalent of not shipping code without tests.
6. Agentic RAG
Agentic RAG replaces the fixed retrieval pipeline with an LLM-based agent that decides the retrieval strategy at runtime. A January 2025 survey (arXiv:2501.09136) on Agentic RAG documents three architectural patterns:
A single-agent architecture uses one agent to plan, retrieve, synthesise and iterate. If the first retrieval does not yield a sufficient answer, the agent reformulates the query and retrieves again.
A multi-agent architecture assigns specialist agents to different retrieval sources or subtasks. One agent queries the vector store, another queries a SQL database, a third handles web search. A coordinator agent merges results.
A hierarchical architecture uses a planner agent to decompose complex queries into subquestions, dispatches each to a retrieval sub-agent and assembles the final answer from partial responses.
# Agentic RAG: tool-using agent pattern (LangGraph sketch)
from langgraph.prebuilt import create_react_agent
from langchain_core.tools import tool
@tool
def search_documents(query: str) -> str:
"""Search the internal document store for relevant context."""
return vectorstore.similarity_search(query, k=4)
@tool
def search_web(query: str) -> str:
"""Search the web for recent information not in the document store."""
return web_search_tool.run(query)
@tool
def query_database(sql: str) -> str:
"""Run a SQL query against the structured data store."""
return db.run(sql)
# The agent decides which tool to call, in what order,
# and whether to iterate based on intermediate results.
agent = create_react_agent(llm, tools=[search_documents, search_web, query_database])When to use it: Complex, multi-step questions that require combining information from heterogeneous sources. Support systems, research assistants and developer tools that need to traverse multiple knowledge bases to assemble an answer.
What it costs: Latency and unpredictability. An agent that decides its own retrieval path can take many more round-trips than a fixed pipeline. Production agentic RAG requires careful timeout handling, fallback paths and cost guardrails.
Honourable Mentions
RAPTOR (arXiv:2401.18059, ICLR 2024, Stanford): Recursively clusters chunks, summarises each cluster and builds a tree of abstractions. Retrieval searches across tree levels, pulling in both specific detail and high-level summaries as needed. Demonstrated a 20% absolute accuracy improvement on the QuALITY benchmark when paired with GPT-4.
HyDE (Hypothetical Document Embeddings): The retriever generates a hypothetical ideal answer to the query first, then uses that hypothetical text as the retrieval query rather than the original question. Significantly improves retrieval for abstract questions where the question embedding differs from the answer embedding.
Multimodal RAG: Extends retrieval to non-text content. Two main paths: convert images, tables and charts to text via OCR or vision models, then retrieve normally; or maintain a multimodal index where images are embedded directly and retrieved alongside text.
Retrieval Strategies Inside Any RAG Type
The architecture you choose determines the structure. These strategies determine the quality of retrieval within that structure.
Chunking strategy: Fixed-size chunks are cheap and predictable but break sentences and paragraphs mid-thought. Semantic chunking (splitting at natural boundaries using embedding similarity) preserves coherence. Hierarchical chunking maintains parent-child relationships so small chunks can be retrieved for precision but expanded to parent chunks for context.
Embedding model choice: General-purpose embeddings (such as OpenAI’s text-embedding-3-small or the open-source BGE family) work well for general corpora. Domain-specific corpora covering legal, medical or code content benefit from fine-tuned or domain-adapted embedding models.
Hybrid search: Combining dense (vector) and sparse (BM25 keyword) retrieval consistently outperforms either alone on heterogeneous real-world corpora. The ratio matters: typical production systems weight dense retrieval between 0.5 and 0.7 and sparse between 0.3 and 0.5.
Reranking: A cross-encoder reranker (running both the query and each retrieved document through a single model) is significantly more accurate than cosine similarity for relevance scoring. The trade-off is latency: re-scoring 20 candidates with a cross-encoder adds 50–200ms depending on model size and hardware.
Query transformation: Rewriting the query before retrieval (using an LLM to expand abbreviations, infer intent or decompose compound questions) gives you the best return on engineering effort of any single improvement in a RAG pipeline.
Where RAG Actually Fails in Production
The documented failure modes of RAG in production are worth mapping explicitly, because most tutorials do not show them.
Retrieval misses: The right document exists but the retrieval does not surface it. Common causes: poor chunking that splits relevant information across boundaries, a semantic gap between how the query is phrased and how the document is written, or an embedding model not suited to the domain.
Retrieval irrelevance: The retrieval surface documents but they answer the wrong question. The LLM then generates a confident answer from those documents, anchoring on irrelevant content. No mechanism in a naive pipeline catches this.
Context window misuse: Retrieved chunks get stuffed into a context window in order of similarity score. Research consistently shows that LLMs give less weight to information in the middle of long contexts (the “lost in the middle” effect). Important context at position 5 of 10 retrieved chunks may be effectively ignored.
Knowledge staleness: The vector index reflects the corpus at indexing time. For fast-moving domains, retrieved documents can be accurate at index time and outdated at query time. Most RAG systems have no mechanism to flag document age.
Confident wrong answers: The most dangerous failure. The LLM receives retrieved context, generates an answer that is inconsistent with or extends beyond the retrieved content, and returns it with no uncertainty signal. Without CRAG-style evaluation or Self-RAG reflection tokens, this failure is invisible.
RAG Paradigms That Do Not Exist Yet But Should
This section is conjecture and proposal. Nothing here has a paper behind it. Treat it as a design space worth exploring.
Temporal-Decay RAG (TD-RAG)
Current RAG systems treat all documents in the index as equally current. For knowledge bases that span years, this is wrong. A document about Kubernetes networking written in 2021 is a different kind of source than one written in 2024.
TD-RAG proposes scoring documents at retrieval time with a temporal weight:
import math
from datetime import datetime
def temporal_score(doc_timestamp: datetime, query_time: datetime,
decay_rate: float = 0.1) -> float:
"""
Returns a recency multiplier between 0 and 1.
decay_rate controls how quickly older documents lose weight.
A decay_rate of 0.1 means a document 10 years old
scores roughly 0.37 of a brand-new document.
"""
age_in_years = (query_time - doc_timestamp).days / 365.0
return math.exp(-decay_rate * age_in_years)
def temporal_rag_score(semantic_score: float, doc_timestamp: datetime,
query_time: datetime, alpha: float = 0.3) -> float:
"""
Blends semantic relevance with temporal recency.
alpha controls the weight given to recency vs. semantic similarity.
"""
recency = temporal_score(doc_timestamp, query_time)
return (1 - alpha) * semantic_score + alpha * recencyThe decay rate should be configurable per domain. For legal precedent, you may want the opposite: older established doctrine should rank higher. For cloud infrastructure, you want aggressive decay. The same indexing infrastructure, different scoring policy.
Adversarial RAG (A-RAG)
Standard RAG retrieves evidence that supports an answer. For domains where the correct answer is genuinely contested, such as legal analysis, medical treatment decisions or policy questions, one-sided evidence retrieval produces one-sided answers.
Adversarial RAG runs two retrieval passes: one looking for evidence that supports the most likely answer, one looking for evidence that contradicts or qualifies it. The generator receives both sets and produces an answer that explicitly acknowledges the tension.
def adversarial_rag(query: str, vectorstore, llm) -> dict:
# Pass 1: retrieve supporting evidence
supporting_docs = vectorstore.similarity_search(
f"evidence supporting: {query}", k=4
)
# Pass 2: retrieve contradicting evidence
# A richer implementation would use a trained contrastive retriever.
contradicting_docs = vectorstore.similarity_search(
f"evidence against or qualifying: {query}", k=4
)
supporting_context = "\n".join([d.page_content for d in supporting_docs])
contradicting_context = "\n".join([d.page_content for d in contradicting_docs])
prompt = f"""You have been given two sets of evidence on the following question.
Supporting evidence:
{supporting_context}
Contradicting or qualifying evidence:
{contradicting_context}
Question: {query}
Provide a balanced answer that acknowledges the tension between these sources.
Where the evidence conflicts, say so explicitly rather than picking a side."""
return {
"answer": llm.invoke(prompt),
"supporting_sources": [d.metadata for d in supporting_docs],
"contradicting_sources": [d.metadata for d in contradicting_docs],
}This is most useful in legal tech, medical decision support and any domain where the correct answer to “should I do X?” is “it depends and here is why.”
Memory-Tiered RAG (MT-RAG)
Inspired by CPU cache hierarchy. The idea: not every query needs to hit the full knowledge base. Most queries repeat patterns. A tiered retrieval system could resolve the majority of queries from a fast in-memory cache while reserving expensive full-corpus search for genuinely novel queries.
Three tiers:
- L1 cache (hot): Recent conversation history and the last N resolved queries with their retrieved context. Checked first. Near-zero latency.
- L2 cache (warm): A smaller, frequently-accessed subset of the knowledge base maintained in a fast vector store. Covers the 80% of queries that hit common topics.
- L3 store (cold): The full knowledge base. Hit only when L1 and L2 yield insufficient results, based on a confidence threshold.
class TieredRAG:
def __init__(self, l1_cache, l2_store, l3_store, llm,
l1_threshold=0.9, l2_threshold=0.75):
self.l1 = l1_cache # In-memory recent context
self.l2 = l2_store # Fast reduced-corpus vector store
self.l3 = l3_store # Full knowledge base
self.llm = llm
self.l1_threshold = l1_threshold
self.l2_threshold = l2_threshold
def retrieve(self, query: str) -> list:
# Check L1 first
l1_results = self.l1.search(query, k=3)
if l1_results and l1_results[0].score >= self.l1_threshold:
return l1_results, "L1"
# Fall through to L2
l2_results = self.l2.search(query, k=5)
if l2_results and l2_results[0].score >= self.l2_threshold:
return l2_results, "L2"
# Full corpus search
return self.l3.search(query, k=8), "L3"The practical benefit: p95 latency drops significantly because the majority of production traffic hits L1 or L2. Full corpus search is reserved for the long tail.
Confidence-Gated RAG (CG-RAG)
This one challenges a core assumption: that all queries need retrieval. A language model already has extensive parametric knowledge. For many straightforward questions, the model would answer correctly without retrieval, and the retrieval round-trip just adds latency and cost.
Confidence-Gated RAG asks the model to answer with a self-reported confidence score first. Retrieval triggers only if that score falls below a configured threshold.
def confidence_gated_rag(query: str, vectorstore, llm,
confidence_threshold: float = 0.8) -> dict:
# Step 1: ask the model to answer and rate its own confidence
confidence_prompt = f"""Answer the following question. After your answer,
on a new line write CONFIDENCE: followed by a number from 0.0 to 1.0
representing how confident you are that your answer is accurate and complete.
Question: {query}
"""
initial_response = llm.invoke(confidence_prompt)
# Parse confidence from response
lines = initial_response.strip().split("\n")
confidence_line = [l for l in lines if l.startswith("CONFIDENCE:")]
confidence = float(confidence_line[-1].replace("CONFIDENCE:", "").strip()) \
if confidence_line else 0.5
if confidence >= confidence_threshold:
# Trust parametric knowledge, skip retrieval
answer = "\n".join([l for l in lines if not l.startswith("CONFIDENCE:")])
return {"answer": answer, "retrieved": False, "confidence": confidence}
# Confidence too low: retrieve and re-answer with context
docs = vectorstore.similarity_search(query, k=5)
context = "\n\n".join([d.page_content for d in docs])
grounded_answer = llm.invoke(
f"Context:\n{context}\n\nQuestion: {query}\nAnswer:"
)
return {"answer": grounded_answer, "retrieved": True, "confidence": confidence}
The limitation is obvious: LLM self-reported confidence is not reliably calibrated. A model can express high confidence and be wrong. This approach works best as a cost optimisation for clearly separable query types (general knowledge vs. corpus-specific) rather than as a correctness guarantee.
Where RAG Is Going (2026–2028): A Projection
A few directions are already in motion and will reach production maturity within the next two years.
Long context vs. RAG convergence. As context windows grow (Gemini 1.5 Pro already supports 1 million tokens, and this number will not stop growing), the question “do I even need RAG?” will arise more often. The answer for most production use cases will still be yes, because stuffing millions of tokens into every request is expensive and attention degrades over long contexts. But hybrid strategies that combine long context for recent or critical documents with RAG for the broader corpus will become standard.
End-to-end differentiable RAG. Current systems treat the retriever and generator as separate components. Research is moving toward jointly training both, so the retriever learns from generation quality signals rather than just retrieval metrics. This is expensive to train but produces systems where retrieval and generation are actually optimised for each other.
Evaluation tooling maturation. The biggest practical gap in production RAG today is not architecture. It is measurement. Ragas, TruLens and similar frameworks are addressing this, but the field has not yet reached the maturity that unit testing brought to application code. Expect this to change significantly by 2027.
Federated RAG for privacy-sensitive domains. Healthcare, legal and financial organisations need RAG over data they cannot centralise. Federated RAG, where each data silo runs its own retriever and a central layer merges results without seeing raw data, will become a compliance requirement in regulated industries.
The Practitioner’s Decision Map
If you are building a RAG system today, here is the shortest path to the right architecture:
Start with Advanced RAG, not naive RAG. The incremental engineering cost is low and the reliability improvement on real-world data is not.
Add CRAG-style evaluation before you add agentic complexity. Catching retrieval failures is higher ROI than adding retrieval autonomy.
Reach for Graph RAG when your users ask synthesis questions rather than lookup questions. If your most common queries are “find me the document about X,” Graph RAG is overkill. If they are “what patterns appear across all our incident reports,” it earns its indexing cost.
Use Agentic RAG when the query requires traversing genuinely heterogeneous sources and no fixed pipeline covers all the cases. Do not reach for it because it sounds impressive. The debugging cost is real.
Treat chunking, embedding model choice and reranking as first-class engineering decisions, not configuration details. They determine the ceiling of what any RAG architecture can achieve.
And measure. Not just end-to-end answer quality. Measure retrieval precision, retrieval recall and the rate at which generation diverges from retrieved context. You cannot improve what you cannot see.
