A client called me last month, half panicked, after getting an email from his hosting provider. The subject line wasn’t a good one: “Your site is consuming excessive resources." The options were either to upgrade the plan or migrate to a better server.
His traffic graphs in GA4 were flat. Same visitors as the previous month. So I went into the access logs to understand what was happening.
It wasn't a DDoS. Well, it was kinda a DDoS. Made by AI bots.
The crawler invasion
Pull up the raw access log of any moderately popular site right now, and you'll see them: GPTBot. ClaudeBot. PerplexityBot. Bytespider. Google-Extended. Meta-ExternalAgent. OAI-SearchBot. CCBot. Applebot-Extended. Each one of them crawling page after page, product after product, post after post, on a schedule none of us were consulted about.
The math starts getting ugly fast. Every one of those requests fires a full WordPress bootstrap: theme, widgets, page builders, plugins, everything that makes your WordPress site run. Database queries fire. CSS and JavaScript get assembled. And then, after the server has done all that work, the response goes out: 80, 100, sometimes 200 kilobytes of HTML per page.
In the end, the bot reads about 5% of it.
So, you paid for the CPU time, the memory, the database connection, and the bandwidth, just to ship hundreds, maybe thousands of kilobytes of theme markup to a crawler that didn't want it.
"Just block them” sometimes won’t work
The first instinct of every dev who notices this is to reach for robots.txt, .htaccess, or Cloudflare bot rules, and block everything related to AI bots.
Problem solved, right?
Nope. Now you've also disappeared from the answers ChatGPT, Claude, and Perplexity give to people searching for what you sell. AI Overviews on Google? Gone. Citations in answer engines? Gone.
For most sites that depend on organic discovery, that's a worse outcome than the hosting bill. Now that Google is killing the default search engine, just blocking AI Bots is the same as killing your site to organic visitors.
AEO (Answer Engine Optimization) is the new front line of search. The companies winning at it right now are the ones whose content actually gets ingested by the major LLMs.
Also, if the marketing team is working on AEO, you might kill weeks or months of strategic work with a single setup. There has to be a better answer.
Let the bots in. Serve them just the main course
That's what ParseLess does. I built it after the call with that client, and released it on the WordPress directory a few weeks ago for public usage.
The idea is straightforward to the point of being almost embarrassing: when an AI bot hits your WordPress site, the plugin detects it by User-Agent and serves the content as clean Markdown instead of full-theme HTML. Same content. Same URL. Human visitors and Googlebot get the normal site. They don't even know the plugin exists. Only the AI crawlers get the lightweight version. To the AI Bot, this is gold.
The numbers come from measuring the same WordPress page in two ways. First, as a browser sees it: the HTML document plus every CSS file, JavaScript bundle, and image the browser has to fetch to render the page. Then, as a bot would see it through ParseLess: a single Markdown response.
| Metric | Full HTML page | Markdown response | Difference |
|---|---|---|---|
| TTFB | 459 ms | 155 ms | ~3x faster |
| HTTP requests | 16 (1 doc + 15 assets) | 1 | 94% fewer |
| Total weight | ~484 KB (doc + CSS + JS + images) | ~6 KB | ~80x smaller |
| Database queries | full bootstrap | cache read | 60–80% fewer |
The first row is the one most people focus on, but it's the third row that does the actual budget damage. When an AI crawler hits a normal HTML page, it doesn't just download the HTML. The crawler pulls the stylesheets, the JavaScript bundles, and increasingly the images too.
ParseLess collapses that to one request. No follow-up asset fetches. Images become plain Markdown links that the bot can choose to fetch or ignore, instead of being downloaded automatically with the page.
Multiply by the bot traffic you're actually receiving in a month, and the bandwidth math stops being a curiosity.
What this looks like in practice
Say you have a WordPress site with 500 posts and/or products. Say the major AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Bytespider, Google-Extended, etc.) each hit most of those pages over the course of a month. Let’s consider conservative numbers.
Before ParseLess: roughly 500 pages × 10 bots × ~484 KB = around 2.4 GB of bandwidth, and 5,000 full WordPress renders with all their cascading asset requests.
After ParseLess, with the cache warm: 6 KB per response on average, single transient reads instead of full bootstraps. Around 30 MB total bandwidth for the same set of crawls. PHP-FPM workers freed up around 3x faster per request, which means the same server handles more concurrent crawler traffic before saturating.
It's the same content being delivered through a much shorter pipe, and without making the hosting bill skyrocket.
The client whose hosting alert triggered all this saved his server upgrade. He didn't lose AEO. He kept being crawled by every model that mattered. And the crawlers stopped costing him real money.
How it works
ParseLess hooks into template_redirect and runs the request through a User-Agent check before WordPress even decides which template to load. If the bot matches a known AI crawler, the request gets routed to the Markdown converter instead of the theme.
The converter runs the_content filter, so anything Elementor, Divi, ACF, or Gutenberg blocks render into the post content is captured correctly. Then it strips the wrapping markup and converts what's left to Markdown. Tables, nested lists, code blocks with language preserved, images with alt text, links, everything is preserved. The output gets stored in a transient with a configurable TTL.
Subsequent bot hits on the same post don't run the conversion. They get a single cache read and a response in milliseconds.
The plugin respects everything you'd expect it to respect. Posts marked noindex in Yoast, Rank Math, or Genesis are excluded. Private, draft, password-protected, and trashed posts never get served. All public post types are included by default, but you can restrict them in settings.
There's also a /llms.txt endpoint at your site root. Optional, off by default if you don't want it.
Since version 0.5.0, ParseLess also publishes a /botfood-sitemap.xml, an AI-specific sitemap listing the Markdown URL of every public post. It gets advertised in robots.txt automatically so crawlers actually find it, and it can be toggled off if you'd rather not expose it.
Knowing it's actually working
The first question anyone sensible asks after installing a plugin like this is: how do I know the bots are actually hitting the Markdown version and not still pulling full HTML?
So I added an Analytics tab to the settings page. Optional logging records each Markdown request with the bot User-Agent, post URL, timestamp, bytes served, and whether the response came from cache. The IP gets stored as a salted SHA-256 hash, never as the raw address. That keeps the data useful for "how often did this bot hit me" without dragging the site into avoidable GDPR or LGPD territory.
The Analytics tab gives you a 30-day chart of bot activity, a breakdown by crawler, the top URLs being fetched, and a list of unknown User-Agents that look bot-shaped but aren't on the detection list. If something in that "unknown" list is clearly an AI crawler — happens regularly, the space moves fast — there's a one-click button to add it to the detection list.
A dashboard widget in wp-admin shows the same data in summary form, so you see it the moment you log in.
If you don't want any of this, leave logging off. The Markdown serving works fine without it. But if you're trying to justify the install to a client, or you want to know whether the new crawler that just appeared in your access logs is actually an LLM bot or something else, the data is there.
Where you will see the benefits
A short list of who actually benefits from this:
E-commerce sites running WooCommerce, especially with a lot of product pages. AI bots love to crawl product catalogs, and product pages tend to be HTML-heavy.
News sites and high-volume blogs where the content updates faster than the crawl can keep up, and the bots come back often.
Sites built on page builders. Elementor and Divi sites are the highest-impact use cases because their HTML payloads are the heaviest in the WordPress ecosystem.
Anyone on shared hosting or a tight VPS who's started getting resource warnings without being able to explain the spike in traffic.
If you're on dedicated infrastructure with headroom, the savings will still be real, but you'll feel them more in your CDN bill than in your server bill.
What it costs
Nothing. ParseLess is GPL, free, and in the official WordPress directory. Install, activate, and AI bots immediately start receiving the lightweight version. No configuration required for the default behavior to work.
Settings live under Tools → ParseLess if you want to adjust the bot list, the cache TTL, the post types, or the frontmatter options.
I'd love to hear from anyone who installs it and measures their own numbers — especially if the savings are different from what I'm seeing. Server setups vary, and the more data I have on real-world impact, the better the next version gets.
Either way, stop paying to render your theme for robots.
