I'm Building a Code Security Analyzer. A Security Tool Found a Critical In It.

I'm building a tool that's supposed to help check code. I call it vibeanalyzer for now. The idea is simple: a lot of us vibe-code — we let an agent write the code, it writes it, it looks clean, the tests pass — and we have no real idea what we just let into the project. Someone close to me put it perfectly: "I don't know what's inside, but I want it to work on the outside."

Before I went all in on the analyzer, I finally did the thing I should have done long ago: check whether someone had already solved this better than me. They had. It's called Semgrep — and the first thing it did was find a Critical vulnerability in my own security analyzer.

This post is about that irony, and what follows from it.

Confession up front: I didn't know the standard tools

I'm self-taught. No CS degree, no senior leaning over my shoulder, I learn from discussions and my own mistakes. So I set off building a code analyzer without really knowing that established tools have been doing this for years — Semgrep, CodeQL, SonarQube, Snyk.

That's the part you'll probably call out in the comments first, so I'll say it myself: yes, I started building a solution without checking what already exists. So I finally did. I ran Semgrep on the vibeanalyzer repo — on the tool that's supposed to be the one guarding security.

What it found

In a few minutes:

One cosmetic issue. SHA-1 used to generate a directory name — which in my case isn't a security hole (it's not a password or a signature, just a hash choice in a non-critical spot), but it's exactly the kind of thing I'd never spot myself.

And then six findings in dependencies. This is where I sat up, because these weren't cosmetic:

• Critical in vitest — a path traversal that, with the UI server running, lets someone read, write, and execute files outside the project. And this is a direct dependency, one I pulled in myself.
• Two High — in esbuild (downloading a binary with no integrity check) and in vite (bypassing the guard that's supposed to stop .env and certificate files from being served).
• Three Medium — variants of path traversal and sensitive-data exposure across vite, esbuild, and launch-editor.

I have to be fair about how serious this actually is. Most of these are transitive — dependencies of my dependencies, not something I installed directly. They all sit in the dev toolchain, so they mostly threaten a running dev server, not necessarily production. And the real-world exploit probability (EPSS) is low across the board — one percent or less. This isn't a burning house.

But that vitest one is Direct and Critical. That's something I dragged into the project myself, directly, and I'd have had no idea.

The uncomfortable question

I'll ask it for you, because I know it's coming: How can a security-checking tool have a Critical vulnerability in itself?

The answer is uncomfortable, and it's the whole point of the project. The finding isn't in my logic — it's in the dependencies. My code can be written as honestly as I can manage and the supply chain still gets me, because the risk isn't carried by me but by what I build on. And if I didn't catch this — someone who's literally building a security tool and actively cares about the problem — what chance does a vibecoder have who just had an agent build an app and shipped it?

None. That's the reason the project makes sense at all.

What Semgrep can't do — and why I'm building anyway

I could stop here with "use Semgrep and forget your own tool." And honestly, for supply chain and known patterns of dangerous code, that holds — it does it better, deterministically, and for free. I'm not trying to beat it at its own game, and you shouldn't trust me if I claimed I was.

But I noticed something in what Semgrep didn't say. It didn't tell me whether the code does what my project actually intended. It doesn't know the boundaries I set for the project. It can't tell that some function runs fine and is safe, but solves a problem that shouldn't be there at all. Semgrep knows patterns of danger. It doesn't know intent.

That blind spot — the gap between what the code does and what it was meant to do — is what this whole series is about. I call it the intent gap. (Yes, the term gets used elsewhere, in marketing and UX; I mean it in my own narrow sense: the distance between code and its intended purpose.)

And that's exactly what I'm building vibeanalyzer on. The first thing it does is load — or ask the user to create — the project's intent and its non-goals: the guardrails that shouldn't be crossed. So that when the AI evaluates the code, it knows what the goal actually is and what's already out of bounds. That's a gap static analyzers don't fill, because they don't read intent, the idea behind the code. And it's the only piece I dare say is worth building next to the tools that already exist.

Where I am now

I've got intent and non-goal loading done. I've got a starting machine analysis of TypeScript and a basic markdown output with a mermaid graph of the folder structure. Ahead of me: the machine security layer (which, by the way, Semgrep currently does better than I do — that needs saying out loud), the skeleton of the AI layer, and the AI logic itself, which is the whole point and also the most uncertain part. I have no idea whether AI can meaningfully evaluate code against intent, or whether it'll just be a flood of false alarms. I tried it on websites, but I've yet to see how it works on a repository — and I'll write about it even if it doesn't pan out.

This series is about building in the open. Don't expect a finished product or guarantees. Expect notes from someone who finally ran the standard tools, took a hit from his own project, and decided to keep going — because the gap he wants to fill is even more obvious after today.

In the next part I'll dig into what the machine analysis of my TS code actually produces — and where it starts to diverge from what I need the AI layer to do. That's where the intent gap stops being a nice phrase and has to become working code, or fall apart trying.

By the way: after writing this, I bumped that vitest version — and it cleared all six dependency findings at once, because the others hung off it transitively. Semgrep now reports zero. Which doesn't mean the project is "safe" — it means those six known CVEs are gone. Unknown holes and my own logic bugs are still invisible to it. That's the whole reason I'm still building. And yes: I wrote a post about a hole I should have patched long ago. That's vibecoding in a nutshell — even in a tool that's supposed to police it.