I'm an automation tester. Usually my job is simple: the same input should give the same output, every time. Language models don't work that way. Ask the same question twice and you can get two different answers, and both can be right.
A RAG system - retrieval-augmented generation - makes it harder still. It searches your own documents and has a model write the answer from what it finds (chat with your PDF, or a support bot answering from a company's help pages). So a wrong answer has two possible causes: the search picked the wrong page, or it picked the right page and the model still got it wrong. To the user these look the same. But they're different problems with different fixes. If your tests can't tell them apart, you don't know which half to fix.
So I built a small RAG system and a test suite built to tell the two apart.
Repo: https://github.com/sbezjak/llm-rag
What it is
12 pages of pytest documentation, chunked and indexed (corpus). The system is two services in a line: a retriever that finds the most relevant chunks, and a generator (llama3.2, running locally through Ollama) that reads those chunks and writes the answer with citations.
query -> [ vector search ] + [ BM25 search ]
-> fuse (RRF) -> rerank -> top 5
-> generator -> answer + citations
The retriever has four stages: two searches run at once - vector search by meaning, BM25 by keyword - then a step that merges their rankings, then a reranker that re-reads the question against each finalist and keeps the best 5. Most bugs hide in the handoff from retriever to generator.
Then five groups of tests, one for each way the answer can go wrong:
| Test group | What it catches |
|---|---|
| Retrieval | The search never found the right page |
| Generation | Right page found, answer still wrong |
| Refusal | The answer isn't in the docs, but the model answers anyway |
| Citation | The model points at a page it didn't actually use |
| Hallucination | The model points at a page that doesn't exist |
The model answers a question it should refuse
I gave the system six questions it had no answer for and told it to say "I don't know." It refused five. The sixth was "how do I write my own pytest plugin" - a real pytest feature, just not in my 12 pages. It answered anyway - a confident 2,000-character answer built around @pytest.addoption, which isn't real pytest syntax. And it does this on every run. An off-topic question is easy to refuse. A real topic that just isn't in your documents is the hard one: the model fills the gap with something that reads like real pytest. That's the case that matters in production.
It also cited a source for the made-up answer - a page that isn't in the corpus. My hallucination test should have caught that, and it didn't, because of a bug in the test itself. To check the citations, the test first dropped any id that wasn't a real page, then looked at what was left. Left nothing - and "cited nothing" is exactly what a clean refusal looks like, so the fabricated citation passed for a refusal. That dropping step wasn't written for this test - it came from the generator, where keeping only the ids the model was actually handed is the right behaviour. I reused that function here without rechecking the assumption, and in a hallucination test the dropped id is the whole point. I only caught it by checking the citations the other way: keep every id the model wrote, real or fake, and flag any that isn't in the index. The test built to catch hallucination nearly missed the one case it was built for.
A search for a bare function name finds nothing
I typed pytest.warns into the search - just the function name, nothing around it - and all four stages missed the page that documents it, even though that page is in the index.
The interesting part is why, and it's not where you'd expect. Meaning-search needs context around a term; a function name alone gives it nothing, so it drifts to related pages ("warnings", "exceptions"). Keyword-search should catch the exact string, but pytest is in every chunk, so it adds no signal, and warns on its own isn't enough to surface the right page. So both searches fail. The merge step only combines their two lists - it can't find a chunk neither search returned. And the reranker only reorders what it's handed, so the right page never reaches it.
No better search engine fixes this. The fix is upstream: rewrite the bare query into something with context, or index identifiers with the words around them so there's something to match.
The reranker finds the right page but ranks the wrong one first
The reranker is the final step - it re-reads the question against each shortlisted chunk and reorders them. On 10 of 16 queries it put the wrong chunk first. Not missing - the right chunk almost always reaches the top 5 - just ranked behind a broad page-overview chunk, when the narrow subsection underneath is the actual answer. This was the biggest single cluster in the project, 10 of the 26 pinned failures.
A reranker scores how much a chunk is about the query. A page-overview chunk mentions more of the query's terms and covers the broader topic, so it reads as "more about this" than the narrow subsection that only answers the one specific question. The catch is that being most about a query and most useful for it aren't the same thing. The overview is more topical; the subsection is more relevant. The reranker optimizes for the first and the user needs the second, so the broad chunk wins a contest it shouldn't. You can watch it happen: ask "how do I clear the cache" and it ranks the cache page's introduction above the paragraph that documents the actual --cache-clear flag.
This isn't a bug in the code; it's how this particular off-the-shelf reranker (bge-reranker-base) behaves on documentation like this, and it matters anywhere the interface shows one best answer. There are three real fixes, in rough order of effort: show the top few candidates instead of just #1, so a slightly-misranked answer is still on screen; fine-tune the reranker on your own query-and-answer pairs so it learns that subsections beat overviews for your content; or swap it for a stronger reranker. What you can't do is trust the #1 as-is.
Meaning search buries the right page just outside the top 5
This is the case that earns the four stages. For a query like pytest -m select tests by marker, meaning-search ranked the right page 6th - in its top 10, but just outside the top 5 a user actually sees. Keyword-search caught the literal pytest -m, the merge step lifted it into the top 5, and the reranker put it first. That's the whole argument for combining the searches: the queries people actually type, full of flags and function names, are exactly the ones meaning-search ranks a little too low, in the exact window the interface shows. Recall and ranking are separate problems, fixed by separate stages - and a test that only checks the top 5 quietly passes the ranking bug, which is why each stage gets its own test.
Why the split matters
The clearest example: one answer scored 0.588 on similarity, under the pass line. It looked like the model writing a weak answer. It wasn't - a chunk still had a piece of raw documentation markup left in it, and the model copied that markup straight into its answer. The bug was in how I cleaned the documents, not in the model. Because the failing test was a generation test running on known-good context, I knew retrieval was fine and looked in the right place. Cleaning the markup moved the score from 0.588 to 0.684 - a real gain just from fixing the input. A good chunk of "the model gave a bad answer" turns out to be "the model was given bad input."
A few more findings
Three scorers, not one. Every answer gets graded by three scorers, but only one - similarity - decides pass or fail. The other two, a word-overlap score (ROUGE) and an LLM judge, are logged next to it. They're worth keeping because they disagree in useful ways: a short but correct answer scores low on similarity, since it's measured against a longer reference, so similarity fails it - while the judge reads the same answer as correct and passes it. Neither is wrong. They measure different things, which is the reason only one of them gets to decide.
Observability and drift. A day-1 baseline records latency and token use, and a drift check re-runs the same queries against it. Retrieval first looked far slower than it actually is, because it was timed while everything else was loading on the same machine.
Temperature 0 isn't deterministic. It's stable within a single run, but across sessions a borderline question flipped from citing its source on 8 of 8 reruns to 0 of 8. Temperature 0 means repeatable inside a run, not the same answer forever.
Reference answers are fixtures you tune. Trimming a reference down to just what the question asks helped one answer and hurt another - one dropped from 0.857 to 0.696 while the judge still gave it 10/10. The similarity scorer is symmetric, so a reference that says less than the answer scores just as far off as one that says more. There's no "trim to the question" rule; the reference has to match the length of the answer the model actually gives.
Checked against Ragas. Ragas is the standard library for grading RAG, so I ran it against my own scorers. On the metric they share it matched mine almost exactly, which tells me my setup is sound. It grades correctness harder than my judge does, because it scores each claim separately instead of the whole answer at once. And it has something mine don't: metrics that check whether an answer is actually backed by the source it came from. That last part is the real reason to reach for Ragas - it's the grounding check the refusal finding called for, the thing you need once you care about hallucinations, not just answer quality.
What I'd reuse
Two things. Split your tests by which part can be wrong, so failures that look identical from the outside land in separate test groups. And pin every known failure with a strict pytest.xfail and a written reason - the test stays green while documenting the limitation, and if the limitation ever disappears, the build breaks on purpose so you notice. I have 26 of these (a couple kept non-strict, for a behaviour that flips run to run); they're the record of what this system gets wrong. Both ideas carried over from the project before this one, and neither is specific to AI.
Honest limits
One corpus, one model, small query sets (16 retrieval, 10 generation, 6 out-of-corpus). Every result here is real and reproducible on this stack, but none of it is a universal claim about RAG - a bigger model or corpus might behave differently. And llama3.2 grades its own answers in one of the scorers, a known weakness I worked around rather than solved.
Run it
brew install ollama
ollama serve # leave running in its own terminal
ollama pull llama3.2 # second terminal: pull the model
uv sync
uv run pytest -m "not ollama" # fast, mocked, ~30s
uv run pytest # full suite, live model, ~23 min
Every test row in the HTML report shows the exact prompt sent to the model and the exact answer it gave back, so you can see for yourself which half went wrong.
Repo: https://github.com/sbezjak/llm-rag
This is the second of five projects on testing AI systems. The first was an eval harness (how do you score an answer when there's no single right one?). The next is red-teaming - deliberately trying to break a model.
