CKP LLM: The Missing Layer Between Your AI Agent and Its Knowledge Base

Last week my AI coding agent gave me a confident, detailed answer — referencing the wrong project entirely.

The problem was not the model. It was context: the agent had loaded 20 knowledge files and picked the wrong one to answer from. The signal was buried in noise.

That bug led me to build CKP LLM — Compiled Knowledge Pattern.

The Problem

Most developers who use AI coding agents build a knowledge base: a folder of Markdown files describing projects, architecture decisions, and recurring patterns. The agent reads them at startup and uses them as memory.

It works. Until it doesn't.

Agents load everything, every time. Whether your question is about authentication or database schemas, the agent reads all 20 files before answering. Context fills up with noise. Answer quality drops — not because the LLM is bad, but because it is reading too much.

RAG solves this at scale, but for a personal or team knowledge base of 20–100 files, it is overkill. You need an embedding model, a vector store, and runtime computation on every query. Too much complexity for the problem size.

The Idea: Smarter Files

What if knowledge files could tell the agent when to load them, and what to bring along?

CKP adds five structured fields to each knowledge file. Here is what a file looks like with a CKP header,

CONCEPT:      mentat
TLDR:         Civic intelligence platform with ISTAT data, RAG chat and AI analytics for municipalities.
ANSWERS_WHEN: mentat, civic, ISTAT, territory, NestJS, security, crime, RAG, analytics
SIMILAR_HIGH: prescient:2025-05, civis:2025-05
SIMILAR_MID:  travelguidehub:2025-05
CONFIDENCE:   high
VALIDATED:    2025-05
Your normal content here. Nothing changes below the header.

The body of the file is unchanged. Everything else stays the same.

What Each Field Does

TLDR — one sentence optimised for LLM reading. If this already answers the query, the full file is never loaded.

ANSWERS_WHEN — keywords that trigger this file. The agent matches these against the query before loading anything.

SIMILAR_HIGH — files that must always load alongside this one. Direct dependencies, shared APIs, same architecture. Encoded explicitly so the agent never has to infer it.

SIMILAR_MID — files that load only if the query domain also matches them. Conditional, not automatic.

VALIDATED — a timestamp per relationship, not per file. If a related file was updated after this date, that specific relationship is flagged as potentially stale.

How It Works at Query Time

The agent keeps a small _index.md containing only the headers of all knowledge files — no body content. This index is always in context. Everything else is loaded on demand.

When a query arrives:

Read the index, match query keywords against ANSWERS_WHEN across all entries
Load the matching file plus all its SIMILAR_HIGH automatically
Load SIMILAR_MID only if their own keywords also match
If the TLDR already answers the query, do not load the full file at all

Result: 2–4 files loaded instead of 20.

The Key Innovation: Compile Time vs Runtime

Existing semantic search computes relationships at runtime — every query triggers embedding lookup, similarity calculation, retrieval. That computation runs hundreds of times a day.

CKP moves that computation to write time. When you update a knowledge file, the LLM computes relationships once and stores them in the header. At query time, the agent reads pre-computed structure.

No vector database. No embedding model at runtime. No infrastructure to maintain.

Why categorical tiers instead of decimal scores?

LLMs are significantly more consistent when classifying into categories than when assigning decimal scores. A score of 0.73 from one session may be 0.61 in another. With a fixed rubric and anchor examples, HIGH / MID / nothing produces stable, reproducible results across any LLM and any session.

HIGH — one concept requires understanding the other. Direct dependency. Max 3 entries.
MID — same domain, frequently relevant together. No direct dependency. Max 5 entries.
Nothing — loosely related. Not stored.

Benchmarks

Tested on a real NestJS codebase (Mentat — civic intelligence platform) using Claude Sonnet. 5 query types, 3 runs each, 30 total runs, two environments compared.

Token reduction by knowledge base size

3 files: 564 tokens No CKP → 522 tokens CKP → 8% reduction (below break-even)
11 files: 4,800 tokens No CKP → 1,618 tokens CKP → 66.3% reduction
30 files (projected): ~13,000 tokens No CKP → ~1,900 tokens CKP → ~85% reduction

The break-even is around 5–6 files. Above that, savings scale super-linearly.

Answer accuracy

No CKP (11 files): 9 correct out of 15 → 60% accuracy
CKP (11 files): 15 correct out of 15 → 100% accuracy

The failures in No-CKP were not random. On an authentication query, the agent mixed information from geography, ISTAT, and frontend files and produced a vague answer. On an out-of-domain query about Stripe on a civic platform codebase, it hallucinated connections between Stripe and existing NestJS modules.

CKP on that same query loaded nothing, declared out-of-domain, and answered honestly.

Reduced context is not just a cost saving. It is a hallucination risk reduction.

Cost impact

Claude Sonnet at $3 per million input tokens

1,000 queries per day: $286 saved per month
5,000 queries per day: $1,432 saved per month
10,000 queries per day: $2,865 saved per month

The AGENT.md Rule

Add this block to your AGENT.md or GEMINI.md:

`BOOT — runs unconditionally on every first message
1. Use current working directory as PROJECT_ROOT (no find/ls).
2. Read PROJECT_ROOT/memory-bank/_index.md directly.
3. If exists → ROUTING. If not → INIT.

INIT — autonomous, zero questions to user
Analyse project from package.json, README.md, src/ structure.
Classify sibling directories as HIGH/MID.
Create memory-bank/projects/[PROJECT_ID].md with full CKP header.
Create memory-bank/_index.md routing table.
Confirm with one line: [CKP LLM initialised. Proceeding.]

ROUTING
Match query keywords against ANSWERS_WHEN.
Load matched file + SIMILAR_HIGH (direct read).
Load SIMILAR_MID only if their ANSWERS_WHEN also match.
Declare in one line: [CKP: loaded X/Y files — match: file via keyword]
Never ask the user questions. Make assumptions, declare them, proceed.`

Building on Karpathy's LLM Wiki Pattern

CKP builds on Andrej Karpathy's LLM Wiki pattern, which proposed compiling knowledge into structured files loaded directly into context.

The gap in the original pattern: files are isolated. The agent knows what exists but has to infer which files relate to which, and how strongly. CKP makes those relationships explicit, pre-computed, and consistent.

From a compiled wiki to a self-routing knowledge graph, stored entirely in plain text.

Who This Is For

CKP is designed for developers and teams who:

Use AI coding agents daily and want to reduce costs
Maintain a knowledge base of 10 to 200 files
Need smarter context management without adding infrastructure
Want any LLM, any agent, any file format — no lock-in

It is not designed for millions of documents. RAG remains the right tool at that scale. CKP fills the gap between loading everything and building full RAG infrastructure.