Evaluating Advanced RAG Retrievers: A Practical Comparison
Evaluating Advanced RAG Retrievers: A Practical Comparison
Retrieval-Augmented Generation (RAG) systems are only as good as their retrievers. With the explosion of new retrieval strategies—vector search, BM25, multi-query, reranking, parent-document, and ensemble methods—how do you know which is best for your use case?
In this post, we walk through a real-world, metric-driven evaluation of advanced retrievers using LangChain and Ragas. We'll show you how to:
- Build and compare multiple retriever types
- Generate a synthetic "golden" test set
- Evaluate with Ragas metrics (faithfulness, context recall, answer relevancy, etc.)
- Analyze results across accuracy, latency, and cost
Experiment Setup
We used reviews from the John Wick movie franchise as our dataset. The following retrievers were implemented and evaluated:
- Naive Vector Retriever (OpenAI embeddings)
- BM25 Retriever (sparse, bag-of-words)
- Parent Document Retriever (chunking + parent context)
- Multi-Query Retriever (LLM-generated query expansion)
- Contextual Compression Retriever (Cohere reranker)
- Ensemble Retriever (reciprocal rank fusion)
💡 Try it yourself:
Explore the hands-on notebook for this workflow:
Advanced Retrieval Evaluation
A synthetic test set was generated using Ragas and Grok-3 LLM, ensuring diverse and challenging queries. Evaluation was performed using Ragas metrics and LangSmith for experiment tracking.
Metrics Used
- Answer Relevancy: Does the answer address the query?
- Context Precision: How much retrieved context is actually useful?
- Context Recall: Did the retriever find all relevant info?
- Faithfulness: Is the answer grounded in the retrieved context?
Results Overview
Accuracy Metrics
| Metric | Best Performer | Worst Performer |
|---|---|---|
| Answer Relevancy | Ensemble & Contextual (0.93) | BM25 (0.83) |
| Context Precision | Contextual & Parent Document (1.00) | BM25 (0.86) |
| Context Recall | Multi Query (0.92) | BM25 (0.35) |
| Faithfulness | Naive & Multi Query (0.84) | Parent Document (0.64) |
Latency
Latency
| Latency Type | Fastest | Slowest |
|---|---|---|
| P50 Latency | BM25 (1.26s) | Ensemble (3.19s) |
| P99 Latency | BM25 (1.86s) | Ensemble (12.20s) |
Token Usage & Cost
| Experiment | Prompt Tokens | Output Tokens | Total Tokens | Total Cost (USD) |
|---|---|---|---|---|
| Naive Retrieval | ~42K | ~2.6K | ~44K | ~$0.01 |
| BM25 Retrieval | ~15K | ~2.1K | ~17K | ~$0.01 |
| Parent Document | ~10K | ~1.8K | ~11K | ~$0.01 |
| Multi Query | ~56K | ~3.6K | ~60K | ~$0.02 |
| Contextual Compression | ~14K | ~2.0K | ~16K | ~$0.01 |
| Ensemble Retrieval | ~70K | ~3.5K | ~73K | ~$0.08 |
Summary & Recommendations
| Use Case | Recommended Approach | Justification |
|---|---|---|
| Highest Accuracy | Ensemble & Contextual Compression | Both achieved top answer relevancy (0.93) and perfect context precision (1.00) |
| Fastest Inference | BM25 | Lowest P50 (1.26s) and P99 (1.86s) latency |
| Best Cost-Performance Tradeoff | Contextual Compression | Excellent precision, low cost, and strong recall |
| Balanced Approach | Multi Query Chain | Highest context recall (0.92), good faithfulness (0.84), moderate cost |
| Low Resource Environments | Naive or BM25 Retrieval | Minimal token usage and cost, fast, reasonably accurate |
- Ensemble and Contextual Compression retrievers lead in accuracy, but Ensemble is slower and more expensive.
- BM25 is the fastest and most cost-effective, but it trails in accuracy and recall.
- Multi Query delivers the best recall and strong faithfulness, with moderate cost and latency.
- Contextual Compression stands out as a well-rounded choice for cost, speed, and accuracy.
Ultimately, your choice of retriever should align with your primary constraint—accuracy, speed, or cost. For most production scenarios, a hybrid or ensemble approach is recommended, but always validate with your own data and requirements.
Ready to take your RAG system to the next level?
Try out these strategies with your own data, experiment with different retrievers, and use metric-driven evaluation to make informed decisions. Dive into the hands-on notebook to start benchmarking today, and share your findings or questions in the comments or on social media. Your next breakthrough in retrieval performance could be just one experiment away!
How to Run Your Own Evaluation
- Prepare your dataset: Use real or synthetic data relevant to your domain.
- Implement retrievers: Try multiple strategies—don't just stick to vector search!
- Generate a test set: Use Ragas or similar tools for robust, diverse queries.
- Evaluate with Ragas: Track metrics, latency, and cost for each retriever.
- Visualize and compare: Use tables and charts to make tradeoffs clear.
💡 Try it yourself:
Explore the hands-on notebook for this workflow:
Advanced Retrieval Evaluation
Final Thoughts
No single retriever is best for every scenario. Use metric-driven evaluation to make informed choices for your RAG system. For production, consider a hybrid or ensemble approach, and always monitor cost and latency alongside accuracy.
References
LangChain Documentation: Recursive Text Splitter - For strategies on optimal document chunking approaches
Cormack, G.V., Clarke, C.L.A., Buettcher, S. (2009). Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods - The theoretical foundation behind the ensemble retriever's rank fusion approach
Semantic Chunking for RAG with LangChain - Deep dive on semantic chunking techniques for RAG systems