tags: seo, analytics, webdev, ai
Most website analytics still start from the same old question: who visited the site?
That question is useful, but it is no longer enough. Modern sites are also read by search crawlers, AI crawlers, preview bots, monitoring tools, and assistants that may later send a human referral. If all of that traffic is flattened into one session stream, the operator loses the ability to understand how the machine-readable web is actually interacting with the site.
The interesting work is not just adding another bot filter. It is designing analytics so human traffic, crawler traffic, AI visibility, and referrals can be seen as different signals without turning the product into surveillance software.
The traffic model changed
A traditional analytics setup is usually optimized around pageviews, sessions, referrers, campaigns, and conversion paths. That model works for human behavior. It is weaker when the visitor is a crawler that may never execute JavaScript, may fetch only a subset of pages, may identify itself inconsistently, and may influence discovery later without creating a normal click path.
AI crawlers make this more visible. A page might be read by GPTBot, ClaudeBot, PerplexityBot, Google-Extended, or another agent-like client long before a person arrives from an AI answer. Treating those requests as noise hides a useful operational signal: which parts of the site are legible to machines, how often important pages are revisited, and whether AI-facing discovery is concentrated in the pages you actually want represented.
For operators, the question becomes less about vanity traffic and more about evidence. Did the docs get crawled after a deploy? Are product pages visible to AI systems? Are crawler spikes tied to a content change, a sitemap change, or an external mention? Those are infrastructure questions, not marketing dashboards.
Separate classification from tracking
A cleaner architecture starts by separating classification from tracking.
Tracking answers what happened. Classification answers what kind of actor produced the event. Those should not be mixed together too early. A human browser, a search bot, an AI crawler, and an uptime probe can all produce requests, but the analysis layer should not pretend they mean the same thing.
A small version of the pattern looks like this:
const AI_CRAWLERS = [
/GPTBot/i,
/ClaudeBot/i,
/PerplexityBot/i,
/Google-Extended/i,
];
export function classifyRequest(userAgent: string | null) {
const ua = userAgent ?? "";
if (AI_CRAWLERS.some((pattern) => pattern.test(ua))) {
return "ai_crawler";
}
if (/Googlebot|Bingbot|DuckDuckBot/i.test(ua)) {
return "search_bot";
}
return "human_or_unknown";
}
This is not a complete bot intelligence system. User-agent matching alone is easy to spoof and incomplete. But it shows the boundary: classification should be explicit, inspectable, and allowed to carry confidence. A mature version can add reverse DNS checks, known crawler lists, IP range validation where appropriate, edge logs, and confidence labels.
The important part is that the operator can see the decision. If the system says a request was an AI crawler, it should be able to explain why.
Privacy still matters
AI visibility should not become an excuse to rebuild invasive analytics.
You can measure a lot without fingerprinting people, setting third-party cookies, or storing raw IP addresses. First-party events, coarse request metadata, anonymized network information, respectful handling of DNT and GPC, and clear bot classification can cover a large part of the operational need.
That tradeoff matters because AI-search visibility sits close to technical SEO, content operations, and infrastructure monitoring. The goal is not to identify every person. The goal is to understand how the site is being read, by whom at a category level, and whether important surfaces are visible to the systems that now mediate discovery.
A useful analytics product should make that distinction obvious in the data model. Human behavior belongs in one lane. Bot and crawler visibility belongs in another. AI referrals belong in another. Joining them is useful; confusing them is not.
What operators should be able to prove
The practical test is simple. After shipping a change, an operator should be able to answer a few questions without guesswork:
- Which pages were visited by humans?
- Which pages were crawled by search bots?
- Which pages were read by AI crawlers?
- Which referrals came from AI assistants or AI search surfaces?
- Which events are high confidence, and which are only directional?
That is the shape WebmasterID is built around: first-party analytics, AI crawler visibility, bot intelligence, and AI referral attribution in one operator-oriented view. The point is not to invent certainty where the web does not provide it. The point is to make the uncertainty visible enough that a real operator can act on it.
Good analytics for the AI-search era should feel less like a growth hack and more like observability. It should show what happened, preserve the difference between humans and machines, and give the person responsible for the site a clear trail from signal to decision.
