We Benchmarked the Most Popular Code Search Tools. We Beat All of Them.

codegraph has 19,459 GitHub stars. We have zero. So we stopped talking and started measuring.

The Headline

P@10 = fraction of top-10 results that are relevant to the task. Higher is better.

knowing is 1.53x more precise than codegraph (19K stars, tree-sitter + FTS5).
knowing is 2.76x more precise than GitNexus (knowledge graph MCP).
knowing is 3.29x more precise than Gortex (Go graph engine, 256 languages).
knowing is 15.9x more precise than grep.

Why 19K Stars Means Nothing

codegraph uses tree-sitter + FTS5 + heuristic scoring (co-location bonuses, multi-term matching, CamelCase boundary matching). No graph-theoretic ranking. No random walk. No structural propagation.

knowing uses Random Walk with Restart on a content-addressed call graph. The walk propagates relevance through the actual dependency structure: "this function calls that one, which implements this interface, which is tested by those tests." Structural relevance, not string coincidence.

The result: codegraph finds symbols that contain your keywords. knowing finds symbols that are structurally relevant to your task. These are often different things.

Per-Repo Breakdown

Repos with well-structured class hierarchies and documentation perform best (Flask 0.332, Terraform 0.275, Kafka 0.253). knowing's docstring FTS directly leverages developer-written documentation as a retrieval signal, and inheritance propagation creates paths through class hierarchies. Even the weakest repo (Cargo 0.132) exceeds grep's best (0.013).

Query Latency: 500x Faster on Enterprise Repos

codegraph queries Kubernetes in about 1 second (BM25, no graph walk). knowing with its pre-computed adjacency cache: 2 milliseconds. That's 500x faster.

The cache is built once at index time and loads the entire graph in one SQLite read. RWR then runs entirely in memory. The 4,717x improvement (from 9s uncached to 2ms cached) is a structural advantage of content-addressed caching: the adjacency map is deterministic, so it never needs invalidation except on re-index.

Time-to-Consistency: New Code in 167ms

You add a function. How quickly does each system find it?

Protocol: inject validate_authentication_token() into Flask, trigger incremental reindex, query for it.

knowing's IndexFilesIncremental takes 16ms (constant, regardless of repo size). codegraph's sync rescans the entire repo (scales linearly). Aider re-parses everything on every query and still doesn't find the new function.

Why Aider fundamentally cannot find new code: A newly added function with no callers has zero in-degree, so PageRank assigns it minimal weight. It will never surface in ranked results until other code calls it. This means every time you write a new function, Aider's context is blind to it. knowing finds it via FTS keyword match, bypassing the need for graph connectivity.

Agent Efficiency: 99.9% Noise Elimination

On Kubernetes (3.5M LOC), an agent doing grep Handler gets 1,284 matches. For "Controller": 14,896 matches. The agent must read/filter all of them.

knowing returns 10 ranked results with 72% ground truth hit rate. codegraph returns 28/50. GitNexus returns 0 (can't handle k8s at all).

The advantage isn't just precision. It's that knowing delivers 10 results from 10,840 candidates. That's 99.9% noise elimination before the agent sees anything.

Determinism: Same Question, Same Answer

We ran the same task 10 times per system.

GitNexus gives a different answer almost every time you ask. Aider varies moderately. You can't regression-test a non-deterministic context system. You can't debug agent behavior if the context changes between runs.

knowing's determinism is structural: content-addressed PackRoot guarantees the same input produces the same output. Always.

The Full Competitor Landscape

We benchmarked every code retrieval tool we could install. Here's the complete picture.

GitNexus (Knowledge Graph MCP)

P@10 = 0.075. Has task-oriented retrieval but 2.76x less precise than knowing.

Fatal flaw: cannot handle enterprise repos. Killed after 60 minutes on Kubernetes (5.7GB RAM, single-threaded JavaScript). knowing indexes the same repo in 18.6 seconds at 200MB RAM.

GitNexus also gives a different answer almost every time you ask the same question (7-9 unique outputs in 10 runs). You can't trust results you can't reproduce.

Aider (~20K stars, PageRank repo-map)

P@10 = 0.050 (prior run; timed out on current full benchmark). File-level retrieval (not symbol-level). Uses tree-sitter + PageRank.

Aider's PageRank approach ranks files by how often they're referenced. This means:

• New code (no callers yet) is invisible
• Results are query-independent (same output regardless of what you ask)
• File-level granularity means you get entire files, not specific symbols

Gortex (Go graph engine, 256 languages)

The most architecturally similar competitor (Go, tree-sitter, parallel graph). P@10 = 0.063 on the 66 tasks it could complete.

Gortex extracts 23x more edges (6.3M vs 268K for k8s) but re-indexes the entire repo on every query call. This makes it impractical for benchmarking multiple tasks and unusable in interactive sessions.

Repomix (25K stars, pack entire repo)

The brute-force approach: dump the entire repo into the context window. No ranking, no intelligence.

Repomix achieves 100% recall by including everything, at 75x the token cost. Most models can't fit the output. knowing gives ranked, relevant symbols in tokens that fit any model.

codebase-memory-mcp (2.6K stars, BM25 + semantic edges)

P@10 = 0.107 on Flask. Uses tree-sitter (155 grammars) + BM25 + label boost.

codebase-memory's BM25 engine spins at 100% CPU on repos with >40K nodes. Scale:

Scale ceiling: ~150K LOC. Any enterprise codebase is unusable.

Where Each Competitor Dies

Every tool has a breaking point. Only knowing handles the full range.

CodeGraphContext (KuzuDB)

Cannot perform task-oriented retrieval. Only supports exact name search. Also: 2,159x slower indexing on Flask (215 seconds vs 0.1 seconds). A navigation tool, not a retrieval system.

Where We Lose

Honesty matters. Here's where knowing is weaker:

Dense TypeScript repos (VS Code P@10=0.137): on large TypeScript codebases with generic symbol names, keyword competition is intense (3,000+ matches for "action"). We mitigate this with density-adaptive type-seed preference (auto-enabled on graphs >40K nodes), but VS Code remains the weakest large repo.

Sparse documentation (Cargo P@10=0.132): Rust repos without /// doc comments don't benefit from docstring FTS. The vocabulary gap persists.

First-result accuracy: codegraph sometimes places the single most relevant symbol at rank 1 more often. But it fills positions 2-10 with noise, dragging precision down. If you only need the #1 result, codegraph is competitive. If you need the top-10 to be useful (which agents do), knowing wins.

No Language Server Required

We tested whether running a language server makes results better. It makes them worse.

Enrichment actually hurts P@10 (0.177 enriched vs 0.185 unenriched). The additional 42K edges from pyright dilute RWR probability mass, spreading relevance across too many paths.

The tree-sitter pipeline + docstring FTS + inheritance propagation already captures
all the connectivity RWR needs. Enrichment adds correctness for audit tools but
actively harms retrieval ranking.

This simplifies deployment: knowing is a single Go binary. No Python LSP, no TypeScript
language server, no background enrichment process. Install and query.

Feedback Compounding (Gets Smarter With Use)

Cold-start P@10 is 0.207. When an agent reports which symbols were useful, knowing records that signal and boosts those symbols in future queries for similar tasks.

The feedback anchors to content-addressed symbol hashes. It persists across sessions and expires automatically when code changes (the package's Merkle root changes, stale feedback becomes invisible). No manual curation. No embedding model. Just hash-keyed counters that decay with staleness.

Real-world impact: an agent that repeatedly works in the same area of a codebase gets progressively better context with zero configuration.

codegraph Fails on 2 Languages

codegraph could not produce results on 10/167 tasks (Spark Java, Ocelot C#). knowing
handled all 167. If your codebase includes Java or C#, codegraph gives you nothing.

The 4,717x Latency Story

Before the adjacency cache, knowing queried Kubernetes in 9 seconds (per-node SQLite lookups during graph walk). After building a compact binary cache (65 bytes/edge, one-time 973ms at index): 1.9 milliseconds. That's 4,717x.

The "500x faster than codegraph" headline understates it. The real improvement vs our own uncached baseline is 4,717x. Content-addressed caching means the adjacency map is deterministic (same edges produce same cache), so it never needs invalidation except on re-index.

Query Robustness: The Honest Negative

We rephrased the same task 5 ways and measured output overlap (Jaccard similarity):

Aider looks good here. But Aider's "stability" means it's ignoring your query. PageRank ranks by graph centrality, not task relevance. It returns the same symbols regardless of what you ask. Stable but wrong 95% of the time (P@10=0.050).

knowing's volatility is correct behavior: "add a before_request hook" SHOULD return different symbols than "implement request preprocessing" because those describe different implementation paths. Precision requires sensitivity to what you actually asked.

We Found a Catastrophic Bug in Our Own System. Here's the Fix.

During benchmarking, our P@10 dropped from 0.230 to 0.101. We traced it to a single root cause: the equivalence matching channel injected 66 noisy results that overwhelmed the 11 correct results during RRF fusion.

The fix was three lines of logic. P@10 recovered to 0.226, exceeding the pre-regression peak.

We publish this because it builds trust. We found a massive regression in our own system, diagnosed it transparently, and fixed it. The methodology caught it. If you can't find your own bugs, your numbers aren't credible.

You Can't Game These Numbers

We ran a 32-configuration parameter sweep across every tunable parameter in the pipeline: RWR restart probability, max seeds, score cutoffs, ranking weights, RRF constants, test penalties, BM25 column weights.

Result: all 32 configurations produce identical P@10. Zero variance.

P@10 is determined by graph reachability (a structural property), not parameter tuning. The only things that moved our numbers were architectural changes: inheritance propagation (+29%), docstring FTS (+5%), import resolution. Tweaking weights does nothing. You can't inflate these numbers with heuristics. The architecture is what matters.

Statistical Methodology

• 167 tasks, 9 repos, 6 languages (Go, Python, TypeScript, Rust, C#, Java)
• Hand-curated ground truth (95% achievability, validated against DB)
• Wilcoxon signed-rank test (paired, non-parametric)
• Cohen's d effect size with bootstrap confidence intervals
• Full reproduction: GOWORK=off go test ./bench/cross-system/ -v -timeout 30m

How We Tested

This isn't a demo on a cherry-picked example. It's a controlled evaluation.

Corpus: 9 public repositories covering 6 languages and the full scale range:

Tasks: 167 hand-curated fixtures across 3 difficulty tiers (easy, medium, hard).
Each task has a natural-language description ("Write a Django management command that
exports user data") and a list of ground truth symbols (the specific functions, types,
and methods a developer would need). Ground truth validated against actual database
contents (95% achievability rate). Never derived from knowing's own output.

Protocol: Each system receives the same task description and returns ranked symbols.
We measure:

• P@10: fraction of top-10 results that match ground truth (precision)
• R@10: fraction of ground truth found in top-10 (recall)
• NDCG@10: ranking quality (rewards correct results ranked higher)
• MRR: position of the first correct result

Statistics: Wilcoxon signed-rank test (paired, non-parametric, no normality
assumption). Cohen's d effect size. Bootstrap 95% confidence intervals. Significance
threshold p < 0.05.

Fairness controls:

• knowing's own repo is excluded from the corpus
• All systems get the same task descriptions (no system-specific tuning)
• Cold start: no pre-existing feedback or session state
• Each system uses its own recommended configuration
• Statistical tests are paired (same tasks, different systems)

Reproduction:

git clone https://github.com/blackwell-systems/knowing
cd knowing
./bench/cross-system/scripts/clone-repos.sh
./bench/cross-system/scripts/index-repos.sh
GOWORK=off go test ./bench/cross-system/ -run TestCrossSystem -v -timeout 30m

Every number in this post is reproducible from that command.

Try It

brew install blackwell-systems/tap/knowing
# MCP integration (auto-indexes on first query):

{ "mcpServers": { "knowing": { "command": "knowing", "args": ["mcp", "--watch"] } } }

No configuration. No manual indexing. The MCP server auto-detects your git repo and indexes on first launch.

The Complete Picture

We beat everyone who matters, on every dimension that matters, with statistical proof and honest acknowledgment of where we lose.

MIT license. Single Go binary. Open source.

github.com/blackwell-systems/knowing

Benchmark methodology: METHODOLOGY.md

Full findings: FINDINGS.md