Last week a client pinged me. Their internal AI assistant was confidently telling employees the wrong vacation policy. Not hallucinating from nothing. Retrieving an outdated document and presenting it as current. Classic RAG failure. Here is exactly how I debugged it and what we fixed.
The symptom: assistant returns 15 days PTO for new hires. Correct answer is 20 days (policy changed 8 months ago).
First thing I always do is check what actually got retrieved.
# Add this temporarily to your retrieval pipeline
results = vectorstore.similarity_search_with_score(
query="vacation policy new hire PTO days",
k=5
)
for doc, score in results:
print(f"Score: {score:.4f} | Source: {doc.metadata.get('source')} | Date: {doc.metadata.get('last_modified')}")
print(doc.page_content[:200])
print("---")
Output showed the problem immediately:
Score: 0.8821 | Source: hr_policy_2021.pdf | Date: 2021-03-15
"New employees are entitled to 15 days PTO in their first year..."
Score: 0.8134 | Source: hr_policy_2023.pdf | Date: 2023-11-02
"Effective Q4 2023, all new hires receive 20 days PTO..."
The 2021 document was scoring higher than the 2023 document. Why? The 2021 version had clearer, more keyword-dense language. The 2023 update was buried in a longer policy revision document with more surrounding text, which diluted the semantic match.
Two problems here. Retrieval is not filtering by document freshness at all. And the index still contains the outdated document.
For the freshness problem, the fix depends on your stack. In LangChain with a Chroma backend:
from langchain.retrievers import TimeWeightedVectorStoreRetriever
retriever = TimeWeightedVectorStoreRetriever(
vectorstore=vectorstore,
decay_rate=0.01, # adjust based on how fast your docs go stale
k=5
)
This is a soft fix. It penalizes older documents but does not exclude them. For policy documents where the old version is actively wrong, you want a harder approach: metadata filtering.
results = vectorstore.similarity_search(
query=query,
k=5,
filter={"doc_type": "policy", "status": "current"}
)
This only works if you are tagging documents with status at ingestion time. We were not. So step two was fixing the ingestion pipeline:
def ingest_document(filepath, metadata={}):
# When ingesting a new version of an existing document,
# mark all previous versions as superseded
existing = vectorstore.get(
where={"source_canonical": metadata.get("source_canonical")}
)
if existing:
for doc_id in existing["ids"]:
vectorstore.update(doc_id, metadata={"status": "superseded"})
# Then ingest the new version as current
metadata["status"] = "current"
metadata["ingested_at"] = datetime.now().isoformat()
# ... rest of ingestion
Third problem: even after fixing future ingestion, the old chunks were still in the index. Delete them:
# Get all chunks from the outdated document
old_chunks = vectorstore.get(
where={"source": "hr_policy_2021.pdf"}
)
# Delete them
vectorstore.delete(ids=old_chunks["ids"])
print(f"Deleted {len(old_chunks['ids'])} chunks from outdated policy document")
After these three fixes, the retrieval order flipped. Current policy document now scores first. Assistant returns the correct 20 days.
The actual bug took about 90 minutes to find and fix. The underlying issue took two days to properly address because it was not just this one document. We found seven other policy documents in the same situation: outdated versions living in the index alongside updated ones, with no mechanism to prefer the current version.
The lesson I keep relearning is that document lifecycle management is not a nice-to-have for enterprise RAG. It is load-bearing. You cannot build a trustworthy knowledge assistant on an index that does not know which version of a document is authoritative.
