Ask ten people how to improve a retrieval-augmented generation (RAG) system and you’ll get ten lists: better chunking, hybrid search, reranking, query rewriting, HyDE, compression. The lists are fine. What’s usually missing is proof that any single item helped your system.
So we built a reference build to practise the boring, important part: the measurement.
What the project set out to test
The RAG optimization reference build is a tuning harness. It runs a RAG pipeline (chunker, retriever, optional reranker, optional compressor), then changes exactly one component at a time from a fixed baseline. Each change is scored on a gold set for recall, precision, keyword hits and token cost. Every change also gets a 95% bootstrap interval, so you can see whether a gain is bigger than the noise.
Important context: everything is synthetic. The corpus is 30 fictional documents, the gold set is 15 questions, the “dense” embeddings are hashed bags of words, and the reranker is simulated. The numbers show what the harness measures, not how a real embedding model would score.
What we observed
1. Hybrid search plus reranking was a clear win. Against the baseline (fixed 200-token chunks with dense retrieval), hybrid retrieval plus a reranker lifted recall@5 from 0.633 to 0.933. Keyword hit rate jumped from 0.233 to 0.833. The cost? Average context grew from about 190 to about 211 tokens, a small price for that gain.
2. Several popular tricks couldn’t be told apart from zero. Paragraph chunking and sliding-window chunking each showed +0.100 recall, but their intervals ran from -0.133 to +0.333, which includes zero. HyDE showed +0.033 with an interval that also included zero. Synonym-based query rewriting changed nothing at all. And the chunking changes nearly doubled context size, to roughly 330 tokens. On a 15-question gold set, a +0.10 bump can simply be luck.
3. The simplest method won, and that’s a warning, not a victory. Plain BM25 keyword search hit a perfect 1.000 recall@5. That’s not because keyword search beats everything. The synthetic embeddings are lexical, so they favour word-matching methods. With a real embedding model the ranking could change. The honest move is to say so rather than declare a universal winner.
4. The harness itself had a bug. Early on, the A/B test scored retrieval before reranking, so the reranker was never actually measured and always looked useless. Once metrics were taken on the final ranking, reranking on dense retrieval alone showed +0.300 recall. If your evaluation measures the wrong stage, you’ll throw away good ideas.
Why this matters for a business
Every RAG change costs something: engineering time, extra tokens, extra latency, another moving part to maintain. If you can’t show a change helped beyond noise, you’re paying for a guess. A small, honest gold set built with your domain experts, plus one-change-at-a-time testing, turns RAG tuning from opinion into evidence. It also shows token cost next to quality, so a “win” that doubles your context bill doesn’t slip through.
Takeaways
- Change one thing at a time from a fixed baseline.
- Report an interval, not just a number; small gold sets are noisy.
- Put token cost beside every quality metric.
- Check that you’re measuring the final output, not an earlier stage.
- Treat surprising winners with suspicion until real data confirms them.
- Grow your gold set from real questions your experts care about.
Reference build
Explore the harness, its limits and its test suite: AventureCloud RAG optimization. It’s a learning reference build on synthetic data, not client work.
Learn it or get it built
Want your team to evaluate RAG like this? Email trainings@aventurecloud.net about our Agentic AI trainings. Need trained AI engineers to tune retrieval on your own documents? Reach us at partnerships@aventurecloud.net.
Hashtags: #RAG #GenAI #AIEngineering
LinkedIn teaser:
We tested popular RAG tweaks one at a time. Hybrid search plus reranking lifted recall@5 from 0.633 to 0.933, three tricks couldn’t be told apart from noise, and our own test harness had a bug.
Here’s what a 15-question gold set taught us about measuring before you tune.
No responses yet