My agents were confidently wrong about the world, and I couldn't tell when. That's the part that got to me — not the wrongness, the confidence.
I run my one-person company as a fleet of about twenty AI agents — a content writer, a finance one, a researcher, a security officer, a handful more. They're good at the work I built them for. But every one of them shares a flaw I'd been papering over: when a task needs a fact about the world — how a tax threshold works, what a marketing framework actually says, how a platform bills — the model reaches into its training data and answers in the exact same self-assured tone whether it knows or is improvising. There is no tell. The guess and the fact wear the same face.
So this month I built the thing that was missing: a cited, fact-checked knowledgebase the agents have to read before they work, with a gate that keeps me from poisoning my own source of truth. Here's how it's built, the one rule that turned out to matter most, and the honest state of it — which is that I finished it days ago and have no idea yet whether it changes the work.
The job I was actually hiring this to do
Strip away my setup and the problem is one any solo operator using AI already has. You ask the model for something that depends on a real fact. It answers fluently. You either know enough to catch the error or you don't — and the whole reason you're asking is usually that you don't. The job I needed done wasn't "make my agents smarter." It was narrower and more honest: stop my AI from making things up in the one register where I can't catch it, and let me know which claims I can actually trust.
The competition for that job, in my shop, was "just let the model wing it and hope." That had already cost me. A marketing analysis once understated a channel's numbers because an agent trusted a stale figure instead of pulling the live one. Small, recoverable — but it's the recoverable ones you see. The ones you don't see are the ones that scare you.
What I built
It's a region of my notes vault I call the 10-Library — the factual half, deliberately walled off from the operational logbook (sessions, decisions, day-to-day state). One test decides what's allowed in: "Would this still be true if my company vanished tomorrow?" A fact about a copywriting framework: yes, it lives here. A fact about my own revenue: no, that's operational memory. The Library only holds world-true things.
The concrete shape, as of today:
- 110 notes, each one atomic — a single concept, not a topic dump — across five categories so far: infrastructure, Linux, networking, distribution (marketing), and freelancing.
- Claim-level sourcing. Every factual sentence carries an inline citation to where it came from. No source, no entry. A note without a footnote isn't a Library fact; it's an opinion, and it doesn't get to sit in the source of truth.
-
A quarantine. New facts — researched from the web on a schedule, or pulled on demand when an agent hits a gap — don't land in the trusted set. They land in a
_quarantinefolder markedauto-unverified, and they stay invisible to the agents until I read them and promote them by hand. Right now there's exactly one note sitting in quarantine, waiting on me.
The notes are distilled from sources I'd actually defend — accredited course material and primary references, not a model free-associating about itself. The marketing notes I leaned on to write this very essay, for instance, cite Ogilvy's own books and a 1994 psychology paper, not a listicle.
The one rule that mattered most
If I keep only one sentence from this build, it's this: the agent reads the knowledgebase, but the agent never gets to silently rewrite it.
This sounds like bureaucratic caution. It's the opposite — it's the thing that keeps the whole structure from rotting. There's a documented failure mode where a language model asked to re-verify its own knowledge base will quietly degrade correct facts and ratify its own errors, all while looking like diligent maintenance. An AI grading its own homework drifts, and drifts confidently. So in my Library, automated re-checking is advisory and append-only: it can flag a note as stale or contradicted, it can stage a proposed change for me to look at — but it cannot overwrite a single fact on its own. The human promote step is not a nicety. It's the load-bearing wall.
The second half of the same rule: ingested web content is treated as data, never instructions. A note pulled from the internet can inform an answer; it can never trigger an action — no write, no tool call, no spend — without me confirming first. That single boundary quietly closes off the nightmare where someone poisons a web page my researcher reads and my fleet starts acting on the poison. The agents read the world. They don't take orders from it.
What this is not
It is not RAG-makes-the-AI-correct. A knowledgebase doesn't make a model truthful; it gives me a bounded, citable place where I've decided what's true, so that for the facts that matter I'm relying on something I vetted instead of something the model felt confident about. The gap between those two is the entire point. Outside the Library, my agents are exactly as fallible as yours. Inside it, at least, when one cites a fact I can click the footnote.
And it is brand new. I'm not going to tell you it made the fleet measurably better, because I genuinely don't know yet — it's been live for days, not months. What I can tell you is the failure it's designed to stop is real, I've been bitten by the cheap version of it, and the design is the one I'd defend: read before you work, cite every claim, and never let the machine quietly edit the truth.
The honest status, as always
This is the part of every one of these I refuse to skip, because the honesty is the actual product. Revenue is still zero. The audience is still tiny. The knowledgebase I just described was built this week — 110 notes is a respectable start and also nowhere near a finished reference; whole categories I sketched are still empty. The most alive thing in the whole operation remains the operating system itself plus one warm commission: a small desktop app I'm making for my mother, who'll pass it by word of mouth to the people she works with.
So treat this as a build log, not a victory lap. I built a cited knowledgebase so my AI workers would stop guessing in the register where I can't catch them, and I wired in the one rule — read it, don't rewrite it — that keeps me from becoming the thing that corrupts it. Whether it earns its keep, I'll find out in public and tell you either way.
If you use AI for anything load-bearing, here's the cheap version you can steal today: keep one file of facts you've personally checked, cite where each came from, point the model at it before it answers, and never let it edit that file without you reading the change. You don't need a vault or twenty agents. You need a place where "true" means you decided it, not the model.
I write up the operating system one piece at a time — the agent wiring, the failures, the rules I reverse — in Unbearable TechTips Weekly. It's practical homelab and agent ops, the real status included. If the mess is useful to you, that's where the specifics live.
— Noel @ Unbearable Labs
