From Invisible to Indexed: The Unglamorous Work of Getting Seen by AI Search

Last week I discovered that my own publication was invisible on its own domain.

Not invisible in the poetic, nobody-reads-my-blog way. Literally invisible. promptway.com, the domain every article on the site declared as its canonical home, was serving an old product prototype I had built months earlier and forgotten about. The actual publication lived on a temporary platform URL. Every AI crawler that found an article was being told "the real copy lives over there," and over there was a different product.

I had written a whole piece about optimizing your site for AI agents and LLMs. The eight layers in that article were all real and all working. And none of it mattered, because the front door had the wrong address on it.

So I spent a day fixing the unglamorous parts. This is the checklist, in plain words. None of it requires you to be a developer, though some steps need one for an hour or two. If part one was about making your site readable to machines, this is about making sure the machines ever show up.

Step 0: check what your domain actually serves

Open a private browser window and type your domain. Not your bookmark. The domain.

Then check three URLs by hand: yourdomain.com/sitemap.xml, yourdomain.com/feed.xml, and yourdomain.com/robots.txt. If any of them 404, that is your whole afternoon right there. Mine did. The publication had all three, but the domain was attached to the wrong project, so the live internet got none of them.

While you are in there, settle the www question. Pick one form of your domain, either yourdomain.com or www.yourdomain.com, and make the other one redirect to it permanently. Search engines treat them as two different sites until you do. I picked the bare domain, made www redirect to it, and also made the hosting platform's free preview URL redirect home, because a copy of your site living on a platform subdomain is a duplicate-content problem you are choosing to keep.

Step 1: let the right robots in

Your robots.txt file is the bouncer at the door. The guest list changed a lot recently, and most sites are still working from an old one.

The crawler that decides whether ChatGPT search cites you is called OAI-SearchBot, and it did not exist when most robots.txt files were written. Same story for Claude-SearchBot, Perplexity-User, and a handful of others. If your file does not mention them, you are relying on default behavior. I would rather be explicit: I allow every major AI search crawler by name, because being cited is the whole point of a publication.

This is a ten-minute edit. Ask whoever manages your site to compare your allowlist against a current guide to AI user agents and add what is missing.

Step 2: put your whole article in the feed

Here is one I had completely wrong. My RSS feed only carried summaries.

A summary-only feed feels safe, like you are protecting the full text. What it actually does is cripple every downstream channel. Feed readers show your readers a teaser and a link. Syndication platforms that import via RSS get nothing worth importing. AI systems that ingest feeds get a paragraph instead of the article.

Full-content feeds are how machines subscribe to you. The fix is technical but small: your RSS feed should carry the complete rendered article, and if you have a JSON Feed, it should too. Your developer will know this as content:encoded. The conversation takes one sentence: "make our feeds full-content."

Step 3: tell the engines when you publish, do not wait to be found

By default, publishing works like this: you post, and then you wait for a crawler to wander by. That can take days.

Two free standards flip it to a push model. IndexNow is a single notification you send when a page is new or updated, and one submission covers Bing, Yandex, and a few other engines at once. Bing matters more than its search share suggests, because most of the web results ChatGPT cites overlap heavily with Bing's top results. WebSub does the same job for feed readers: it pings a hub when your feed changes, and subscribers update within seconds instead of whenever they next poll.

Both are set-and-forget. We wired ours into a small script that runs automatically on publish, and I never think about it. If you publish through a platform like Ghost or WordPress, there is a decent chance a plugin or setting already does this. Turn it on.

Step 4: put a human name on AI-assisted work

Some of the writing on Promptway is drafted by AI personas. That is disclosed openly, profile pages and all. But disclosure alone is not enough anymore, and the sites that got hammered by Google's scaled-content crackdowns had one thing in common: nobody human was accountable for the words.

So every persona-written article on the site now carries a visible line: "Reviewed by Agnel Nieves," linking to my profile. It is the same pattern medical sites have used for years with "medically reviewed by Dr. X." The page also says it in structured data, where I am listed as the editor, and my profile is connected to my real accounts elsewhere so machines can verify I am a person who exists.

One honest note: this is an accountability signal, not a ranking cheat code. It tells readers and crawlers that a named human stands behind the work, which happens to be true. If it were not true, I would fix that first.

Step 5: never ship a blank share card

Paste one of your article links into Slack or iMessage. If the preview is a gray rectangle, you are losing clicks you already earned.

Most articles on my site have generated hero art, but a few do not, and those shipped with no preview image at all. The fix was a small template that auto-generates a clean, typographic card with the article title for any post without art. Every link now unfurls into something. Most platforms have this built in; if yours is custom, it is an afternoon of work for a developer, once, forever.

Step 6: measure the AI traffic separately

Here is the stat that convinced me to finally set up analytics: visitors arriving from AI tools like ChatGPT and Perplexity are a tiny slice of traffic, around one percent industry-wide, but they convert far better than organic search visitors. Someone who clicks through from an AI answer chose your site after the machine already summarized you. That is a warm lead, not a drive-by.

If you lump that traffic in with everything else, you will never see it. The setup is one custom channel group in Google Analytics that buckets referrals from chatgpt.com, perplexity.ai, claude.ai, and gemini.google.com. Ten minutes in the admin panel, and from then on you can answer "is any of this AI optimization work doing anything?" with a number.

The part where I admit a fleet of agents did the typing

Full transparency: I did not hand-write most of this. I described the plan, and a team of AI coding agents executed it, each sized to its task. Small models refreshed the robots file and the redirects. Bigger ones rebuilt the feeds and wrote the publish-time automation. I reviewed the output, caught a few real mistakes (one agent excluded the exact files we needed redirected, another drew a logo with the text colliding), fixed them, and shipped.

That division of labor felt right. The judgment calls, what to build, what to skip, what counted as done, stayed with me. The typing did not have to.

The whole checklist, one more time

1. Type your domain into a private window and make sure it serves the thing you want found, on one canonical host.
2. Check sitemap.xml, feed.xml, and robots.txt by hand. Fix the 404s.
3. Update your robots.txt allowlist for the 2026 crawler landscape.
4. Make your feeds full-content.
5. Wire up IndexNow and WebSub so publishing pushes instead of waiting to be pulled.
6. Put a named, linkable human reviewer on anything AI-assisted.
7. Generate a fallback share image so no link ever unfurls blank.
8. Build an AI-referrals channel group in your analytics.

None of this is clever. That is the point. The clever work, the writing, the structured data, the llms.txt file, only pays off after the plumbing is sound. Check the front door first.