Retrieval failures are architecture failures, not model failures, and switching models won’t save you.
Everyone builds the same RAG pipeline. You pick an embedding model, dump your documents into a vector database, wire up LangChain and ship. The demo looks great. The chatbot answers questions with what sounds like authority. Your stakeholders are impressed.
Then your first real user asks something slightly off-script, and the bot confidently cites a policy from two years ago as if it’s current. Or it blends information from two different products and presents the mashup as a coherent answer. Or it straight-up ignores the most relevant paragraph in your entire knowledge base because that paragraph got split in half during ingestion.
The instinct is to blame the model. Upgrade to the latest GPT or Claude release? Swap out your embedding model? The real problem is that your retrieval pipeline has architectural debt, and no model upgrade will paper over it.
The Chunking Problem Nobody Takes Seriously
Chunking is the first place RAG systems go wrong, and it gets treated like an afterthought. Most teams pick a chunk size, set an overlap value and move on. The default is usually fixed-size splitting at 512 tokens with around 50 tokens of overlap, because that’s what the tutorial used.
Fixed-size chunking is the worst strategy for almost every document type except the narrow set where it accidentally performs okay. A paragraph boundary means nothing to a fixed-size splitter. Neither does a table, a code block or a numbered list that spans four paragraphs.
A 2026 benchmark testing seven chunking strategies across 50 academic papers found that recursive character-based splitting at 512 tokens reached 69% retrieval accuracy. Semantic chunking, which splits on embedding similarity boundaries and sounds much smarter, landed at 54%. The reason: semantic chunking produces fragments that average around 43 tokens. Far too short to carry enough context for meaningful retrieval, it slices paragraphs into so many pieces that each piece becomes meaningless in isolation.
Adaptive chunking aligned to logical topic boundaries outperformed both, hitting 87% accuracy in a peer-reviewed clinical decision support study. The principle is simple: your chunks should map to the logical units of your source documents. For a legal contract, that’s clauses. For a technical manual, that’s procedures. For internal Slack exports, it might be conversation threads.
The code fix isn’t complicated:
from langchain.text_splitter import RecursiveCharacterTextSplitter
# Don't blindly accept tutorial defaults
splitter = RecursiveCharacterTextSplitter(
chunk_size=512,
chunk_overlap=50,
separators=["\n\n", "\n", ".", " "]
)
# Match your separators to your document structure
splitter = RecursiveCharacterTextSplitter(
chunk_size=800,
chunk_overlap=120,
separators=["\n## ", "\n### ", "\n\n", "\n", ". ", " "]
)That separator list matters more than the chunk size. You’re telling the splitter: prefer breaking at a Markdown heading first, then a double newline, then a sentence boundary. The chunk size is a fallback for when no natural boundary exists within range. Tune the separators for your actual content, not for the generic tutorial document somebody wrote to sell a course.
Vector Similarity Is Not Relevance
The second architectural failure is treating cosine similarity as a proxy for relevance. Your retrieval step returns the top-k chunks where k is usually 3, 5 or 10. The chunks with the highest embedding similarity to the query win. This sounds correct and mostly works, until it spectacularly doesn’t.
The failure mode is called retrieval noise: chunks that are topically related to the query but don’t actually answer it. A query about “how to cancel a subscription” might retrieve chunks about “subscription pricing” and “subscription benefits” because those documents share vocabulary with the query. Your retriever sees similarity. Your user sees a confusing non-answer.
The fix is a two-stage pipeline. Retrieve more than you need in stage one, then rerank aggressively in stage two.
import cohere
from langchain.vectorstores import Qdrant
cohere_client = cohere.Client(api_key="...")
def retrieve_and_rerank(query: str, vectorstore: Qdrant, k_initial: int = 30, k_final: int = 5):
# Stage 1: broad retrieval
results = vectorstore.similarity_search(query, k=k_initial)
# Stage 2: rerank with cross-encoder
docs_text = [doc.page_content for doc in results]
reranked = cohere_client.rerank(
model="rerank-english-v3.0",
query=query,
documents=docs_text,
top_n=k_final
)
return [results[hit.index] for hit in reranked.results]Cross-encoder reranking typically improves RAG accuracy by 20 to 35 percent. The cost is latency: expect 200 to 500 milliseconds per query depending on your volume of candidates. For most internal tools and chatbots, that trade-off is worth it. For real-time autocomplete at scale, it is not.
Cohere released Rerank 4 in December 2025, adding self-learning capability that adapts the model to your domain without requiring labeled training data. It is worth evaluating if you are already on their stack. For self-hosted setups, the BGE cross-encoder family from BAAI remains a strong open-source alternative.
The Context Fragmentation Trap
Related to chunking is a subtler failure: critical information that exists in your knowledge base but never reaches the LLM because it got split across two chunks at indexing time.
Imagine a policy document that reads:
“Refund requests must be submitted within 30 days of purchase. Requests submitted after this window will not be processed.”
If that sentence sits right at a chunk boundary, your retriever might return the chunk ending with “must be submitted within 30 days” without connecting it to what comes next. The LLM sees an incomplete fact. The user gets a partial answer, delivered with full confidence.
The overlap parameter exists to address this, but 50 tokens of overlap on a 512-token chunk is barely two sentences. Increasing overlap to 15 to 20 percent of your chunk size is a reasonable baseline. For 800-token chunks, that’s 120 to 160 tokens of overlap.
A more reliable approach for structured documents is parent-child indexing. You embed small child chunks for retrieval precision, but return the full parent section to the LLM for complete context.
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.retrievers import ParentDocumentRetriever
from langchain.storage import InMemoryStore
# Small chunks for precise retrieval
child_splitter = RecursiveCharacterTextSplitter(chunk_size=300, chunk_overlap=30)
# Larger parent sections returned to the LLM
parent_splitter = RecursiveCharacterTextSplitter(chunk_size=1200, chunk_overlap=100)
store = InMemoryStore()
retriever = ParentDocumentRetriever(
vectorstore=vectorstore,
docstore=store,
child_splitter=child_splitter,
parent_splitter=parent_splitter,
)
retriever.add_documents(docs)LangChain’s ParentDocumentRetriever handles this pattern directly. The retrieval step finds via child chunks; what reaches the LLM is the full parent. This eliminates context fragmentation for documents with clear hierarchical structure without much extra engineering effort.
You Can’t Fix What You Don’t Measure
Here is the failure that senior engineers overlook most often: most RAG systems ship with zero evaluation infrastructure.
You built the pipeline, tested it with a dozen queries you already knew the answers to and declared it working. That is not evaluation. That is confirmation bias wearing a demo hat.
A minimal evaluation setup requires three things: a test set of real user queries, ground-truth answers for those queries and a scoring mechanism that doesn’t just eyeball outputs. RAGAS is the current standard for this. It scores across retrieval relevance, faithfulness to retrieved context and answer completeness without requiring you to label every single output manually.
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision
from datasets import Dataset
test_data = {
"question": ["How do I cancel my subscription?", ...],
"answer": [generated_answers],
"contexts": [retrieved_contexts],
"ground_truth": ["Submit a cancellation request in account settings...", ...]
}
result = evaluate(
dataset=Dataset.from_dict(test_data),
metrics=[faithfulness, answer_relevancy, context_precision]
)
print(result)
# {'faithfulness': 0.82, 'answer_relevancy': 0.79, 'context_precision': 0.71}A context precision score below 0.7 means your retriever is pulling in too much noise. A faithfulness score below 0.8 means the LLM is generating answers that drift from what the retrieved context actually says. These are actionable numbers that tell you where to fix things.
If you don’t have this running before you make changes to your chunking strategy or embedding model, every RAG improvement you think you made is a guess. Every “improvement” is also a regression risk somewhere else.
The Vector Database Is Not the Problem
Teams spend an unusual amount of time agonizing over Pinecone vs Qdrant vs Weaviate. The reality is that for most workloads under 10 million vectors, the choice barely affects performance. What matters more is whether you can run it yourself and whether it integrates cleanly with your evaluation loop.
Pinecone is the obvious default if you want managed infrastructure and are willing to pay for it. Qdrant is open source, written in Rust and noticeably fast on filtered queries, which makes it a good fit for cost-sensitive teams. Weaviate includes hybrid search out of the box, combining vector similarity with BM25 keyword matching. For most internal RAG applications, that hybrid mode is worth the extra complexity because keyword matching catches exact product names, error codes and terminology that embedding similarity handles poorly.
The mistake is spending two weeks benchmarking databases before you’ve fixed your chunking strategy. Fix the architecture first. Pick the database second.
The Metadata You’re Not Storing
One thing missing from almost every RAG tutorial: metadata filtering is frequently more important than embedding similarity for multi-tenant knowledge bases.
If your knowledge base has documents from multiple products, departments or time periods, a pure similarity search will mix them indiscriminately. A question about Product A might retrieve a highly similar chunk from Product B. The model gets confused. The user gets a wrong answer for the wrong product.
Store structured metadata on every chunk and filter before you retrieve:
# Ingest with metadata
vectorstore.add_documents(
documents=chunks,
metadatas=[{
"product_id": "product-a",
"department": "support",
"last_updated": "2025-11-15",
"doc_type": "policy"
} for chunk in chunks]
)
# Pre-filter at retrieval time
results = vectorstore.similarity_search(
query=user_query,
k=20,
filter={"product_id": "product-a", "doc_type": "policy"}
)This pre-filter step cuts retrieval noise by an order of magnitude for multi-tenant setups. It is also one of the highest-value changes you can make with the least engineering effort. You’re trading a bit of upfront schema design for dramatically cleaner retrieval results.
Stop Blaming the Model
Here’s the pattern you’ll see if you build enough of these systems: teams stuck in a loop of model upgrades without improving retrieval quality. They switch to a newer model, see a marginal improvement, chalk it up to the better model and stop there. Three months later the same failure modes resurface.
The retrieval layer determines the majority of your output quality. The LLM is a finishing step. Give it the wrong context and even the best model produces wrong answers. Give it the right context and most models perform adequately.
The actual work is unsexy: clean your source documents before ingesting them, align your chunk boundaries to your document structure, add a reranker, store metadata and filter on it, run continuous evaluation against a real query set. None of this is as exciting as waiting for the next model release. All of it will do more for your answer quality than any model swap.
Your RAG pipeline’s failure is almost never the model’s fault.
If you’ve shipped a RAG system and haven’t run RAGAS against it yet, do that before anything else. The scores will tell you exactly what’s broken. The answer will almost certainly be in your retrieval layer, not your model selection.
A message from our Founder
Hey, Sunil here. I wanted to take a moment to thank you for reading until the end and for being a part of this community. Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? We don’t receive any funding, we do this to support the community.
If you want to show some love, please take a moment to follow me on LinkedIn, TikTok, Instagram. You can also subscribe to our weekly newsletter. And before you go, don’t forget to clap and follow the writer️!
