Hire a Code Watchdog: Building a Claude Code Auto-Review Bot as a Quality Gate for Solo Projects (with Real Ops Logs)

When you ship alone, nobody catches your 2 AM mistakes. This article gives you a working Claude-powered review bot that runs on every pull request via GitHub Actions, posts inline findings, and — crucially — refuses to bankrupt you. By the end you can copy two files into any repo and have a guard dog that barks at real bugs and stays quiet otherwise.

I'll also walk through the failures I hit wiring this up, because the interesting part of a review bot isn't the happy path. It's the night the dog wouldn't stop barking.

Why a Claude Review Bot Beats a Linter for Solo Developers

Conclusion first: a linter checks syntax and style, but a Claude-based reviewer checks intent — and for a solo dev, intent drift is what actually breaks production.

ESLint won't tell you that your new retry wrapper swallows the original exception. ruff won't notice that you compare a Decimal to a float in a billing path. Those are semantic bugs, and they're exactly what a second pair of eyes catches in a team. When you don't have a team, you rent the eyes.

The design constraint that shapes everything below: a review bot has no human deciding when to stop. A human reviewer gets tired and goes home. An automated one will happily call the API in a loop until your card declines. So the architecture is two parts — a reviewer and a leash.

The Python Reviewer: Forcing Structured Output from Claude with Tool Use

The single biggest mistake people make is asking Claude to "review this diff" and then regex-parsing prose out of the reply. Don't. Use tool use to force a JSON schema, so the model cannot return free text where you expect structured findings.

Here is the core reviewer. It reads a unified diff on stdin, sends it to Claude with a forced tool call, and prints findings as JSON. It defaults to Sonnet 4.6 (claude-sonnet-4-6) because review volume is high and Sonnet is the cost-sane workhorse; you escalate to Opus only when you mean it.

# review.py — reads a unified diff on stdin, emits JSON findings on stdout
import os, sys, json
from anthropic import Anthropic

client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
MODEL = os.environ.get("REVIEW_MODEL", "claude-sonnet-4-6")
MAX_DIFF_CHARS = 60_000  # hard cap; see the truncation failure below

REVIEW_TOOL = {
    "name": "submit_review",
    "description": "Submit code review findings for the diff.",
    "input_schema": {
        "type": "object",
        "properties": {
            "findings": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "file": {"type": "string"},
                        "line": {"type": "integer",
                                  "description": "line number in the NEW file, from the diff hunk header"},
                        "severity": {"type": "string", "enum": ["blocker", "warning", "nit"]},
                        "title": {"type": "string"},
                        "detail": {"type": "string"},
                    },
                    "required": ["file", "line", "severity", "title", "detail"],
                },
            }
        },
        "required": ["findings"],
    },
}

SYSTEM = (
    "You are a senior reviewer for a SOLO developer with no second reviewer. "
    "Report only semantic bugs, security issues, and correctness traps. "
    "Do NOT report style or formatting — a linter handles that. "
    "If a line number is not derivable from a hunk header, omit the finding rather than guessing. "
    "An empty findings list is a valid, good outcome."
)

def main():
    diff = sys.stdin.read()
    if not diff.strip():
        print(json.dumps({"findings": []})); return
    truncated = diff[:MAX_DIFF_CHARS]
    note = "" if len(diff) <= MAX_DIFF_CHARS else "\n[diff truncated — review only what is shown]"

    resp = client.messages.create(
        model=MODEL,
        max_tokens=2000,
        system=SYSTEM,
        tools=[REVIEW_TOOL],
        tool_choice={"type": "tool", "name": "submit_review"},  # force the schema
        messages=[{"role": "user",
                   "content": f"Review this unified diff:\n\n{truncated}{note}"}],
    )
    for block in resp.content:
        if block.type == "tool_use" and block.name == "submit_review":
            print(json.dumps(block.input))
            return
    print(json.dumps({"findings": []}))

if __name__ == "__main__":
    main()

Two things earn their keep here. tool_choice with a named tool means Claude is required to call submit_review — you never parse prose. And the system prompt explicitly blesses an empty list as success. Without that line, the model invents nits to feel useful, and a guard dog that barks at the mailman gets ignored.

The GitHub Actions Workflow That Wires Claude into Pull Requests

Now the leash and the delivery. This workflow triggers on PRs, computes the diff against the base branch, runs review.py, and posts a single sticky comment. It pins max_tokens, uses concurrency to kill stale runs, and never re-reviews the bot's own comments.

# .github/workflows/claude-review.yml
name: Claude Review
on:
  pull_request:
    types: [opened, synchronize]

concurrency:                       # a new push cancels the previous review run
  group: claude-review-${{ github.event.pull_request.number }}
  cancel-in-progress: true

permissions:
  contents: read
  pull-requests: write

jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0           # need base branch history for the diff
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install anthropic
      - name: Run Claude review
        id: review
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          BASE="origin/${{ github.event.pull_request.base.ref }}"
          git diff "$BASE"...HEAD -- . ':(exclude)*.lock' ':(exclude)dist/**' \
            | python review.py > findings.json
          cat findings.json
      - name: Post sticky comment
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const data = JSON.parse(fs.readFileSync('findings.json', 'utf8'));
            const f = data.findings || [];
            const TAG = '<!-- claude-review -->';
            const body = f.length === 0
              ? `${TAG}\n🐕 Claude review: no blocking issues found.`
              : `${TAG}\n🐕 Claude review — ${f.length} finding(s):\n\n` +
                f.map(x => `- **[${x.severity}]** \`${x.file}:${x.line}\` — ${x.title}\n  ${x.detail}`).join('\n');
            const {owner, repo} = context.repo;
            const issue = context.issue.number;
            const comments = await github.rest.issues.listComments({owner, repo, issue_number: issue});
            const mine = comments.data.find(c => c.body.includes(TAG));
            if (mine) await github.rest.issues.updateComment({owner, repo, comment_id: mine.id, body});
            else await github.rest.issues.createComment({owner, repo, issue_number: issue, body});

The <!-- claude-review --> HTML marker is how the bot finds and updates its own comment instead of stacking a new wall of text on every push. The :(exclude)*.lock pathspec keeps lockfile churn out of the token bill. And cancel-in-progress matters more than it looks — see below.

The Night the Claude Bot Wouldn't Stop Barking

Here are the concrete failures, because reproducing them is how you understand the system.

Failure 1 — the retry loop that machine-gunned Opus. My first version wrapped messages.create in a naive while True retry around the SDK's RateLimitError. When the API returned 429, the code retried immediately with no backoff. Worse, my early config used Opus for every review. Under a burst of pushes, each 429 fired another full Opus request within milliseconds, which produced another 429, which fired another request. The bot wasn't reviewing anything — it was a busy loop hammering the rate limiter. The fix is exponential backoff that respects the retry-after header, and the SDK already does this for you if you stop fighting it:

client = Anthropic(max_retries=4)  # built-in exponential backoff; don't hand-roll a while True

The lesson: an unattended process plus a retry loop plus a per-call cost is a money fire. The concurrency block in the workflow is the second guardrail — without it, ten rapid pushes spawn ten concurrent review jobs, each holding its own retry loop.

Failure 2 — hallucinated line numbers from a truncated diff. Before MAX_DIFF_CHARS, a 4,000-line refactor blew past the context I budgeted, and git diff output got cut mid-hunk. Claude, trying to be helpful, reported a blocker at payments.py:812 — a line that didn't exist in the diff because the hunk header it referenced had been sliced off. The bot looked confident and was completely wrong, which is the worst kind of reviewer. The fix is the explicit truncation cap plus the system-prompt rule "if a line number is not derivable from a hunk header, omit the finding." Confident-and-wrong is worse than silent.

Failure 3 — the feedback loop on its own comments. An early iteration diffed the whole PR including review threads via a script step, and on the next synchronize event the bot ended up "reviewing" the text of its own previous comment, generating meta-findings about its own findings. Scoping the diff strictly to git diff base...HEAD -- . (code only, never comment bodies) and the sticky-comment marker killed it.

Tuning the Claude Quality Gate: When to Block the Merge

The last decision is policy: should a finding block the merge or just inform? I keep the gate soft on purpose. Making the job exit 1 on any blocker sounds disciplined, but a single false-positive blocker on a Friday teaches you to merge with --admin override, and once you've trained yourself to ignore the gate, the gate is dead.

Instead I let the comment post and only fail the check when there are blocker-severity findings and the diff touches a protected path:

BLOCKERS=$(jq '[.findings[] | select(.severity=="blocker")] | length' findings.json)
if [ "$BLOCKERS" -gt 0 ] && git diff --name-only "$BASE"...HEAD | grep -qE '(auth|payment|migrations)/'; then
  echo "Blocking: critical path has $BLOCKERS blocker(s)"; exit 1
fi

This is the watchdog philosophy in one rule: bark at everything, bite only near the things that actually hurt. The bot reviews 100% of PRs, comments on most, and blocks only when a high-severity finding lands in auth/, payment/, or migrations/.

What You Have Now, and Where to Point It Next

Copy review.py and claude-review.yml, add ANTHROPIC_API_KEY to repo secrets, and every PR gets a Claude reviewer that posts structured, line-anchored findings, updates one comment instead of spamming, respects a token cap, and won't loop your bill to the moon.

The natural extensions, in order of payoff: feed the file's full content (not just the diff) for the few files Claude flags, so it can reason about surrounding code; cache the system prompt with prompt caching to cut input cost on every run; and add a REVIEW_MODEL=claude-opus-4-8 override on PRs labeled high-risk so the expensive eyes show up only when you ask. The watchdog metaphor holds the whole way down — a good guard dog is cheap to keep, loud at the right moments, and smart enough to know the mailman.