Beyond naive RAG: the retrieval techniques that hold up in production
Why the tutorial pipeline breaks
Fixed-size chunks cut tables, clauses and procedures in half, and a chunk stripped of its document often lacks the words that would make it retrievable: a paragraph about "the revised limit" never says which policy it belongs to. Dense embeddings then retrieve passages that sound similar rather than ones that answer the question, and exact identifiers such as part numbers or regulation articles are missed entirely.
Use keyword and dense retrieval together
On the BEIR benchmark, BM25 keyword search remained a robust zero-shot baseline that many dense retrievers failed to beat outside their training domain [4]. Dense retrieval catches paraphrases; BM25 catches exact terms. Run both and merge the ranked lists with reciprocal rank fusion, a simple method that outperformed more complex fusion schemes in the original study [3].
Give every chunk its context
Anthropic's contextual retrieval prepends a short, generated description of where each chunk sits in its document before embedding and keyword-indexing it. In their tests, contextual embeddings plus contextual BM25 reduced the top-20 retrieval failure rate by 49%, and adding a reranker brought the reduction to 67% [2]. Structure-aware chunking, splitting on headings, clauses and table boundaries rather than token counts, complements this.
Rerank, then order the context deliberately
Retrieve generously (20 to 100 candidates), then let a cross-encoder reranker pick the few that go into the prompt. The reranker reads query and passage together, which embeddings cannot do.
Order matters too. "Lost in the Middle" showed that models answer best when the relevant passage is at the beginning or end of a long context and noticeably worse when it sits in the middle [5]. Put the strongest evidence first and keep the context short.
Query rewriting, used carefully
Short queries can be expanded into several reformulations, or into a hypothetical answer that is embedded instead of the query (HyDE) [6]. These help on vague questions. On narrow enterprise corpora a hypothetical answer can pull retrieval towards something plausible but wrong, so keep the original query in the mix.
Evaluate retrieval and answers separately
Build a set of a few hundred real questions with the passages that answer them. Measure retrieval (was the right passage in the top k?) separately from generation (did the answer stay faithful to the retrieved text?). Frameworks such as RAGAS provide reference-free metrics for faithfulness and relevance [7]. Re-run the set on every change to chunking, embeddings or prompts, and sample production queries for human review.
Sources
- [1]Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020) · arXiv
- [2]Introducing Contextual Retrieval · Anthropic
- [3]Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods (Cormack et al., 2009) · University of Waterloo
- [4]BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models (Thakur et al., 2021) · arXiv
- [5]Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023) · arXiv
- [6]Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE, Gao et al., 2022) · arXiv
- [7]RAGAS: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023) · arXiv
Implement this
Ready to deploy this at your organisation?
More from Solnix