How does an AI agent pick from 686 skills in a second?

I ran an empirical test on the "skills as semantic router" pattern for Claude Code agents. I indexed 686 randomly sampled skills from a 4,556-skill community corpus into mesh-memory, embedded them with a single sentence-transformer model, and ran a fixed set of eight task queries through it. Here are the headline numbers: strict top-1 accuracy 62.5%, top-5 cluster accuracy 87.5%, sub-second query latency, ~500 tokens loaded per task versus the ~228K tokens just to keep names + descriptions of all 4,556 skills in the system prompt (the default behavior, even with Anthropic's progressive disclosure). That is roughly a 456x context-window saving with the right skill landing in the agent's top-5 candidates seven times out of eight.

This post explains why I ran the test, how it was set up, what the results actually show, and where the pattern breaks honestly. The full source for the runner and queries is reproducible.

Why progressive disclosure is not enough at scale

Anthropic's Claude Code skills (and Cursor's equivalents, and every other agent framework's skills) ship as markdown files in a folder. Each one has a name and a short description in its frontmatter. The default loading strategy is what Anthropic calls "progressive disclosure": the agent reads every skill's name + description into its system prompt at startup, and only loads the full body when it decides to invoke one.

Progressive disclosure handles the body problem — you do not pay for skill bodies you never use. But it does not handle the index problem. Even just the names and descriptions are loaded for every skill, every session, before any work starts. At fifty skills you spend roughly 2.5K tokens on the catalog. At 200 skills the catalog eats 5% of a Claude Sonnet 200K context window before you have asked anything. The math gets ugly fast:

Even when the catalog physically fits, attention quality on a long list of similar items degrades. Past about 1,000 entries the agent starts making wrong picks on hand-eye-distinguishable cases. There is also no garbage collection: nobody removes stale skills, nobody flags duplicates, the catalog only grows.

The semantic router pattern decouples the catalog from the prompt. Each skill's name and description is stored once in an embedding index, with a tag pointing to the SKILL.md file on disk. At task time the agent runs a single semantic-search call against the task description, gets the top-5 candidates, picks one, and reads the full body only for the one it picked. Token cost per turn is constant regardless of catalog size.

That is the theory. The question is whether the search actually returns relevant skills in real conditions.

The test setup

Corpus. antigravity-awesome-skills, a public collection of Anthropic-format skills from the community. 4,556 SKILL.md files, deduplicated by directory. Each file has a YAML frontmatter (name, description, tags) and a markdown body.

Sample. 1,000 skills selected by random.shuffle(seed=42) from the sorted file list. Of those, roughly 200 were silently rejected by the mesh bulk-ingest endpoint (likely a content-validation filter), 25 failed in transit on a single wave, and 86 ended up stuck in the "pending" state after embedding — a known mesh-memory worker stall. Final indexed corpus: 686 skills.

Routing document. For each skill, the embedded text is exactly name + "\n\n" + description. The full SKILL.md body stays on disk; mesh holds only the routing signal plus a skill_path tag with the disk path. That is roughly 50-200 tokens per skill in the index.

Embedding model. intfloat/multilingual-e5-base, running locally via sentence-transformers, 768-dimensional vectors, stored in Postgres + pgvector. Ten parallel embedding workers, throughput ~38 docs/minute on a single CPU container.

Queries. Eight diverse task descriptions, written before looking at the corpus, designed to span common dev work:

1. "deploy docker production"
2. "analyze stock market data"
3. "write marketing email"
4. "optimize slow SQL query"
5. "security audit web app"
6. "set up CI CD pipeline python"
7. "debug memory leak C++"
8. "build React TypeScript component"

For each query I asked mesh for the top-5 most similar skills and inspected the names plus cosine similarity scores.

Metrics.

• Strict top-1: the first result is the kind of skill a human reviewer would pick for the task without reservation.
• Loose top-1: the first result is in the right family but not a perfect match (e.g. an Azure deploy skill for a Docker deploy query).
• Top-5 cluster: at least one of the first five results is a strong match the agent could reasonably read and use.

The results

After all 686 skills were indexed, query-by-query:

Strict top-1: 5 of 8 = 62.5%.
Top-5 cluster: 7 of 8 = 87.5%.
Cosine similarity range for top-1: 0.83-0.88.
Query latency: under 1 second across all runs.

The convergence curve

The corpus was built in waves of 100 skills, with the full query suite re-run after each wave. That gives a sense of how router quality scales with corpus depth:

*one query timeout on the run

Two observations land hard.

Top-5 cluster saturates early. By 500 indexed skills (about 11% of the full corpus) the cluster metric was already at 85% and barely moved with the next 186. For most queries the relevant family of skills was already in the index; what changed later was which member led the cluster.

Strict top-1 keeps climbing. Going from 500 to 686 indexed skills bumped top-1 from 50% to 62.5%. The improvement came from one specific skill (xvary-stock-research) finally landing in the sampled portion. Each new wave was a chance for an exact-match skill to surface for one of the eight queries.

When the router fails honestly

The SQL query never produced a real top-1. The top-1 was food-database-query (a false match on the word "query"), and the cluster contained spark-optimization and cqrs-implementation but no actual SQL-tuning skill. Looking at the unindexed portion of the corpus, sql-optimization-patterns exists — it just landed in waves 10-45 of the shuffle, beyond our 1,000-sample window.

This is the honest face of the pattern. Router accuracy is bounded by corpus depth, not by the search algorithm. Embeddings did their job perfectly: every query returned coherent clusters with similarities in the 0.83-0.88 range. When the right skill was in the index, the router found it. When it was not, no search quality compensates.

The practical consequence: this pattern wins as your skill collection grows. With under 30 skills there is no point — eager-load them and move on. Past 100, the math starts mattering. Past 1,000, it is the only sustainable option.

The token math

Per task turn:

• Default loading (names + descriptions for 4,556 skills): ~228,000 tokens in the system prompt. Will not fit in a 200K context window. In a 1M context window it consumes 23% of the budget on catalog alone. Progressive disclosure saves the bodies, not the index.
• Router: 1 search request (negligible tokens), top-5 results returned (~500 tokens), 1 full SKILL.md read for the picked skill (~500-1,500 tokens depending on size). Total per turn: under 2K tokens.

That is a 100x to 450x reduction depending on which skill ends up being read. The kicker: the router cost is constant whether the catalog holds 100 skills or 100,000.

Open issues to be honest about

Stuck-on-embed leak. Of 772 documents that made it into the mesh database, 86 (11.1%) ended up stuck in the "pending" state and never got an embedding. The mesh-memory worker code has MAX_CONSECUTIVE_ERRORS = 10, which halts a worker on a bad document and silently leaves the rest of the queue. Worth filing as an upstream issue; in the meantime, restarting the worker on a fresh container drains the backlog.

Silent bulk-ingest drops. About 200 of 1,000 documents sent never appeared in the database at all — the bulk PUT returned success but mesh stored fewer rows than were posted. Likely a content-validation filter on empty or near-empty documents. Worth investigating but did not affect the search-quality results.

The 10-worker bump. Default NUM_WORKERS = 3 is bottlenecking on a single-CPU container. Bumping to 10 quadrupled throughput (~9 в†’ ~38 docs/min) with no observed degradation. This change should be parameterizable in the next mesh release.

Query timeouts. One query in the wave-5 run timed out at the default 30s; rerunning succeeded. Likely a cold-cache warmup issue, not a search-correctness problem.

Reproduce it yourself

Everything used in this test is open source. The runner is roughly 70 lines of Python:

• Corpus: antigravity-awesome-skills
• Memory store: mesh-memory (MIT-licensed, ships with an MCP server for Claude Code and Cursor)
• Runner + queries + raw results: posted alongside this article

Drop the corpus into mesh, run a few queries against it, and look at what you get back. If the pattern fits your workflow, the wiring is trivial: a single MCP call in your claude.md that says "before working on any task, search mesh for type:skill matching the task description, then load the top result."

Related

If you have not read it yet, the companion post How to Organize Your Claude Skills Without Drowning in Files covers the versioning side of this same pattern — storing skills as memory documents with date tags, so the latest version is one query away and old versions never need to be deleted. Together the two posts cover both "how do you store skills sanely" and "how do you retrieve them sanely."

Originally published on klymentiev.com