Ask where a RAG system is likely to be weak and most engineers point at retrieval: the embeddings aren’t good enough, the chunking splits context awkwardly, the top-k ordering puts the wrong document first. All of that is real and worth fixing. It’s also the easier half of the problem, because retrieval quality is measurable in a way the rest of the pipeline isn’t: you can hold out a labeled query, check which documents came back, and score precision and recall directly. The harder half doesn’t show up in that score at all. Even when retrieval returns exactly the right document, nothing stops the generation step from drifting away from it, adding a detail the source didn’t say, or stating something the source only implied. Retrieval quality and answer faithfulness are different axes, measured differently, and a system can be excellent on one while silently failing the other.
What investing in retrieval quality actually looks like
It’s worth seeing a real example of retrieval-quality engineering before arguing it isn’t the whole problem, because the investment is genuine and often underrated. A RAG chat platform’s reranker takes Pinecone’s raw cosine-similarity matches and re-scores them with a small trained model before they reach the generation step, instead of trusting embedding similarity order as-is:
/**
* Pure-TypeScript inference for the tiny reranker MLP trained offline in
* ml/reranker/train.py. Combines Pinecone's own cosine-similarity score
* (dense signal) with a lexical word-overlap signal to re-rank RAG matches,
* instead of trusting raw embedding-similarity order alone.
*/The model itself is a two-layer MLP trained on cosine similarity plus a lexical word-overlap feature, small enough to run as plain arithmetic in a Cloudflare Worker rather than needing a model-serving runtime in production:
function extractFeatures(query: string, cosineSimilarity: number, docText: string): number[] {
const queryWordCount = tokenize(query).length;
const docWordCount = tokenize(docText).length;
return [
cosineSimilarity,
lexicalOverlap(query, docText),
Math.min(queryWordCount, 50) / 50,
Math.min(docWordCount, 500) / 500,
];
}The premise behind reranking at all is that raw embedding similarity is a noisy proxy for relevance: two passages can be close in embedding space for reasons that have nothing to do with actually answering the query, and a second signal (here, literal word overlap) catches some of what pure cosine similarity misses. This is real, checkable engineering. You can evaluate whether the reranked order improves relevance against a held-out testset, and the project does. But notice what the entire pipeline, embedding plus reranking, is answering: which documents are most relevant to the query. It has nothing to say about whether the eventual generated answer actually stuck to what those documents contain.
Relevance and faithfulness are answering different questions
Relevance asks: did we retrieve the right material? Faithfulness asks: does every claim in the final answer trace back to that material? A system can score well on the first and badly on the second, because nothing about handing an LLM the correct documents guarantees the LLM’s response is constrained by them. The model can still generalize past what’s in front of it, fill a gap with something plausible-sounding from its training data, or state a detail more confidently than the source actually supports. Perfect retrieval upstream doesn’t propagate into a faithfulness guarantee downstream; they have to be checked independently, because they fail independently.
This is also why “faithfulness” needs its own explicit evaluation rather than being assumed as a side effect of good retrieval. One way to check it, used in a research-agent project handling a similar problem in a different pipeline, is an LLM judge given both the raw retrieved sources and the generated report, scored specifically on whether the report uses only facts present in those sources:
class EvalScore(BaseModel):
faithfulness: int = Field(
description="Score 1-5: Does the report ONLY use facts present in the raw sources?"
)
relevance: int = Field(
description="Score 1-5: Does the report actually answer the user's topic comprehensively?"
)The two fields are deliberately separate scores, not one blended number, because they can and do move independently: a report can be relevant to the topic asked while still stating something the sources never said, and a report can be scrupulously faithful to its sources while failing to actually address what the user asked. Collapsing them into a single “quality” score would hide exactly the failure mode that matters most, an answer that reads well and cites the right ballpark but says something its sources don’t support.
Where the engineering effort should actually go
None of this is an argument against reranking or better embeddings, retrieval quality is a real bottleneck and worth the investment the reranker above represents. It’s an argument against stopping there. A RAG system with excellent retrieval and no faithfulness check has a blind spot exactly where it matters most: the step between “we found the right document” and “the answer we generated is actually grounded in it” is invisible to every metric that only looks at what came back from the vector store. If you’re going to spend engineering time on one axis and treat the other as free, retrieval quality is the wrong one to treat as free, because a wrong answer built from the right sources is a failure mode retrieval metrics have no way to see.