The AI agent had one job: pay approved vendor invoices, so the finance team could stop doing it by hand.
On a Tuesday morning, it picked up invoice #4471 from a freight vendor Ksh48,000, stamped Approved in the company's ERP, cleanly matched to a valid purchase order. The agent checked the things it was told to check. They all passed. It paid the invoice.
The invoice had already been paid. The previous Thursday. By a member of the finance team.
Here is what the company's systems believed that morning and none of them was wrong.
- The ERP said: Approved. Unpaid. The reconciliation job that pulls in bank activity runs overnight, and last night it had failed silently. So the ERP's picture of the world was simply four days stale.
- The bank feed said: Paid. Last Thursday. It was right. Nobody had told the ERP.
- A Slack thread said: "hold everything to this vendor they double-billed us last quarter, I'm sorting it out with their AP team." Posted by the accounts-payable lead. Three days earlier. Resolved in her head, and nowhere else.
- The vendor's own email said: "Payment well received, thank you!" referring, of course, to Thursday's payment. The agent's inbox reader had seen it that morning, then set it aside, because email ranked below the ERP and the two disagreed.
Every system was internally consistent. Every system was the authority on something. And there was no system anywhere not one that could answer the only question that actually mattered: has invoice #4471 been paid?
A human clerk would almost certainly have caught it. Not because a clerk is smarter than the model they're not. Because a clerk would have felt the friction. They'd have half-remembered cutting the check. Or scrolled past the Slack message that morning and hesitated. Or simply had the reflex to ping someone before sending $48,000 out the door. Reconciling systems that quietly disagree is most of what operations people actually do all day so much of it that nobody files it under "work." It's just judgment.
The agent had no friction to feel. It read the highest-priority system, found Approved, unpaid, and acted at machine speed, with no pause, no second source, no instinct that something was off.
It didn't break. It worked exactly as designed, against a company that had no single, trustworthy answer to give it.
This is the failure almost nobody names correctly. I call it epistemic collapse
We keep trying to fix the agent better models, better prompts, better retrieval, tighter guardrails when the agent was never the broken part. The broken part is underneath it. Companies don't have a layer that turns scattered data into one trustworthy answer. They have an ERP, and a Slack workspace, and a bank feed, and an inbox, and a spreadsheet each holding a fragment of the truth, none of them agreeing, none of them on the same clock held together by a thin film of human judgment that reconciles the whole mess silently, all day, forever.
That film finally needs a name, because we are now, for the first time, trying to build it.
3. What's actually missing.
Call it epistemic infrastructure: the layer that turns data into truth.
It's worth being precise about that, because the two words get used as if they're the same thing, and the gap between them is exactly where the agent fell in.
Data is what your systems store. Truth is what is actually the case. Most companies are drowning in the first and own nothing that reliably produces the second. The ERP had data. The bank had data. Slack had data. What no one had was a place that could take all of it and resolve it into the invoice has been paid with enough confidence to bet ksh48,000 on the answer.
Every business system you own makes the same quiet mistake: it crushes three genuinely different things into one overwritten field.
- Observation - somebody asserted X. "The ERP shows the invoice as approved and unpaid."
- Truth - X is actually the case. "The invoice is genuinely unpaid."
- History - X became true at some point, and used to be something else. "Unpaid through Wednesday. Paid since Thursday."
A database row that reads status: unpaid smashes all three together. It can't tell you who said so, when it became true, what it was before, or whether anything disagreed. The instant the field is written, every one of those distinctions is gone. The row doesn't say "the overnight job hasn't run, so this is a four-day-old observation from one source, and the bank feed disagrees." It just says unpaid, with the full, flat confidence of a fact.
Now multiply that by every system in the company each collapsing observation, truth, and history its own way, each certain about its own slice, none aware of the others and you have produced that Tuesday morning by construction. The systems weren't malfunctioning. They were doing the only thing they were ever built to do: store data and present it as truth.
Epistemic infrastructure is the missing layer that refuses to do that. Instead of overwriting a field, it records observations and keeps them who observed, from where, when it was claimed true, and what it replaced. Instead of forcing one answer when sources disagree, it represents the disagreement as a first-class fact rather than silently picking a winner. And instead of pretending the company has a single consistent state, it can tell you the honest thing: the ERP and the bank disagree about #4471, the ERP's data is stale, and a hold was placed in Slack three days ago do not pay this yet.
That last sentence is the entire product. It's the sentence no system at the company could produce, and the sentence a competent human produces effortlessly.
4. Why AI agents are the ones that broke
Here's the part that reframes everything: this problem is not new. Companies have always been a heap of disagreeing systems on mismatched clocks. The contradiction in the story has been sitting in that company for years. It never detonated for one reason.
Humans were the epistemic infrastructure.
They were the layer that knew the overnight sync is flaky, that the bank lags two days, that a hold in Slack outranks a green checkmark in the ERP, that you check with the AP lead before you pay that particular vendor. We held the company's real, reconciled truth in our heads and patched it on the fly and we never wrote it down, because it never occurred to anyone that this reconciling was a system at all. It just looked like people being good at their jobs.
AI agents are the first workers to operate on the company without that instinct. An agent takes the data at face value, because face value is all the data exposes every field arrives wearing the full confidence of truth, stripped of its provenance, its staleness, and its dissenters. Then the agent acts on it, at a scale and speed no clerk ever could.
So agents don't introduce a new failure. They're a stress test that exposes an old one. The instant you remove the humans, the layer they were silently providing goes missing and the company discovers it never had a truth layer at all. It had people standing in for one.
This is also why the usual fixes don't touch the problem:
- Better prompts make the agent reason more carefully about the inputs it's given. But every input in the story was, locally, correct. No amount of careful reasoning over Approved, unpaid recovers the fact that the data is four days stale and contradicted elsewhere. You cannot prompt your way to information the field doesn't carry.
- Better retrieval (RAG) gets the agent more of the company's data. But more data from systems that each collapse observation, truth, and history just gets you more confident contradictions, faster. RAG would have happily surfaced the ERP record and the bank record and the Slack hold as four equally authoritative snippets, with nothing to say which one reflects reality right now. Retrieval finds documents. It does not resolve truth.
- Guardrails catch the failures you already imagined. "Don't pay over 50k without approval" wouldn't have fired this was 48k and was approved. Guardrails are just a list of known traps; the whole nature of inconsistent truth is that it fails in shapes you didn't enumerate.
Each of these treats the agent as the thing to fix. None of them builds the layer underneath. You can have a perfect model reason flawlessly over a world that has no coherent notion of what's true, and it will still pay the invoice twice correctly, confidently, and on time.
The rest of this piece is about that missing layer: what it actually stores, why "store observations, never overwrite truth" is the load-bearing rule, and what it takes to build the thing humans have been quietly being all along.
Conclusion
The agent that paid invoice #4471 twice is going to be everywhere. Not that exact bug but its shape. Every company racing to put agents into procurement, support, finance, and ops is about to discover the same thing in its own dialect: the agent works, and the company can't tell it the truth.
We will spend the next few years blaming the models. We'll buy better ones, prompt them more carefully, wrap them in more guardrails, and feel the failures get rarer and stranger and more expensive. And we'll be optimizing the wrong layer the entire time. The model was never the variable. The substrate was.
AI didn't create this problem. It just removed the people who were hiding it. Your company has always run on contradiction; you only notice now because the new workers take your systems at their word. The fix isn't a better agent. It's the layer underneath one the layer that knows the difference between what a system says, what is true, and what used to be. Until that exists, every agent you deploy is one stale field away from doing the wrong thing, perfectly.
Author: Mwai Victor
One promise I made to myself is that even in the age of AI, I will keep doing what makes us human: face problems directly and solve them head-on. These articles are proof that I didn’t avoid them I worked through them.
