RAG SOTA: I Tested 7 Pipelines and Built SEQUOIA (Open Source)
After 20+ hours of compute time on local hardware, I benchmarked 7 RAG configurations against real-world tasks. The results surprised me — and changed how I think about retrieval architecture.
Why This Matters
RAG is everywhere in 2026. Everyone claims their pipeline is "SOTA," but most benchmarks use toy datasets. I wanted to see what actually works when you have:
- Messy real documents (not clean academic corpora)
- A local LLM (slightly weaker than GPT-4)
- Production constraints (latency, cost, accuracy tradeoffs)
The 7 Configurations Tested
| Method | Approach | Score |
|---|---|---|
| No-RAG | Direct LLM generation | Baseline |
| Classical RAG | Dense retrieval (BGE-small + FAISS) | Poor |
| Hybrid RAG | BM25 + Dense + RRF fusion + cross-encoder reranker | Moderate |
| LightRAG | Key-value extraction graph + dense hybrid | Disappointing |
| PageIndex | Two-stage hierarchical retrieval | Okay |
| GraphRAG | Entity graph + dense fallback | Complex |
| Agentic RAG | Multi-step reasoning pipeline | Slow, expensive |
| SEQUOIA | RAPTOR tree + step-back prompting | Best |
| SEQUOIA Pro | Multi-query + rerank + compression | SOTA |
What Surprised Me
LightRAG underperformed
The Twitter-hyped "graph RAG revolution" didn't hold up on real data. LightRAG produced what I call "procedural warming" — it looks sophisticated but retrieval quality was mediocre. Academic benchmarks ≠ production reality.
Step-back prompting is underrated
Most RAG systems fail because they retrieve on the literal query. Step-back prompting (rewriting the query into a more general form before retrieval) improved recall by ~15% across the board. Combined with RAPTOR tree clustering, it creates a retrieval hierarchy that actually makes sense.
Local LLMs can evaluate
I used a local model for summarization and judging. Slightly weaker than GPT-4, yes, but the relative rankings between methods stayed consistent. This means you can prototype and benchmark without burning API credits.
SEQUOIA Architecture
User Query
↓
Step-back Prompting (generalize)
↓
RAPTOR Tree Retrieval (hierarchical clusters)
↓
Rerank + Context Compression
↓
Local LLM Generation
RAPTOR = Recursive Abstractive Processing for Tree-Organized Retrieval. Cluster leaf nodes, summarize upward, retrieve at multiple levels of abstraction.
Step-back = Before searching, ask: "What is the general principle behind this specific question?"
Results
On my test set (banking documents, technical manuals, internal wikis):
| Method | Precision | Recall | Latency |
|---|---|---|---|
| Classical RAG | 0.62 | 0.58 | 120ms |
| Hybrid RAG | 0.71 | 0.65 | 340ms |
| LightRAG | 0.59 | 0.61 | 890ms |
| SEQUOIA | 0.84 | 0.79 | 450ms |
| SEQUOIA Pro | 0.87 | 0.82 | 680ms |
SEQUOIA Pro trades some latency for accuracy. SEQUOIA (basic) is the sweet spot for production.
Code & Reproducibility
Everything is open source:
🔗 github.com/Diyago/rag-benchmark
- All 7 implementations
- Evaluation dataset (anonymized)
- Configs for local LLM setup
- Notebooks for analysis
Lessons for Production
- Don't trust academic benchmarks blindly. Test on YOUR data.
- Hierarchical retrieval beats flat. RAPTOR's tree structure matches how humans actually organize knowledge.
- Query rewriting is free performance. Step-back prompting costs nothing in latency but improves retrieval significantly.
- Local evaluation is viable. You don't need GPT-4 to compare methods relatively.
What's Next
I'm extending SEQUOIA with:
- Multi-modal retrieval (images + text)
- Streaming context compression
- Adaptive depth (shallow for simple queries, deep for complex)
More AI Engineering Notes
I write about practical AI/ML from inside a bank — RAG systems, LLM deployment, team management, and what actually works vs. what's just hype.
Telegram channel (Russian, technical): AI.Insaf
Have you benchmarked RAG on real data? What surprised you? Drop a comment or reach out on Telegram.
Эта статья также опубликована в Telegram-канале AI.Insaf — про AI/ML из банковской практики, бенчмарки и управление DS-командами.
Подписывайтесь на канал для оперативных разборов и практических кейсов: https://t.me/ai_tablet
