Originally published at thatdevpro.com. This framework reference is part of the 14-tier Engine Optimization stack from ThatDevPro, an SDVOSB-certified veteran-owned web + AI engineering studio. You are reading the dev.to mirror; the source-of-truth canonical version with embedded validation tools lives at the link above.
Multi-Step AI Agents, Autonomous Browsing, Tool-Using AI, Site Compatibility for Agent Workflows, and the Emerging Layer Above Traditional Search
A comprehensive installation and audit reference for agentic AI search — the new layer where AI systems don't just answer queries with cited sources, but autonomously browse the web, execute multi-step research workflows, fill out forms, complete transactions, and act on behalf of users. This is a 2026 frontier requiring different optimization than traditional search or even AI engine citation.
Cross-stack implementation note: the code samples in this framework are written in plain HTML for clarity. For React, Vue, Svelte, Next.js, Nuxt, SvelteKit, Astro, Hugo, 11ty, Remix, WordPress, Shopify, and Webflow equivalents of every pattern below, see
framework-cross-stack-implementation.md. For pure client-rendered SPAs (no SSR/SSG) seeframework-react.md. For Tailwind-specific concerns (purge, dynamic classes, dark-mode CLS, focus accessibility) seeframework-tailwind.md.
1. Document Purpose
The shift from "AI assistants that cite sources" to "AI agents that take actions" is profound. Where ChatGPT in 2023 read static training data, where Perplexity in 2024 retrieved current sources, the 2026 layer of AI search is different again: agents like Claude with computer use, OpenAI's Operator, Google's Project Astra-derived systems, and emerging open agents browse the live web, execute multi-step research, evaluate options across sites, and complete tasks on behalf of users.
For SEO, this creates new optimization dimensions:
- Agent-friendly site architecture — sites agents can navigate cleanly
- Action affordances — clear interactive elements agents can identify and use
- Trust signals legible to agents — credentials and evidence agents recognize
- Form and transaction optimization — sites where agents can complete actions
- Anti-bot defenses calibrated correctly — blocking malicious bots while allowing legitimate agents
- Agent identification protocols — emerging standards for agent self-identification
- Output structures agents prefer — content patterns that work for agent consumption
This framework specifies optimization for the agentic AI layer. It complements rather than replaces:
-
framework-aicitations.md— for AI engine citation in conversational answers -
framework-technicalseo.md— for traditional crawler optimization -
framework-schema.md— for structured data foundations
The agentic layer adds requirements specific to autonomous AI workflows.
1.1 The Agent Landscape (May 2026)
current_agent_landscape:
consumer_agents:
- Claude with computer use (Anthropic)
- OpenAI Operator
- Google's Astra-derived consumer agents
- Perplexity Pages and assistants
developer_platforms:
- Anthropic Computer Use API
- OpenAI Assistants API with browsing
- LangChain-powered custom agents
- LlamaIndex agentic frameworks
enterprise_agents:
- Internal agents at major companies
- Industry-specific agents (legal research, medical research, etc.)
emerging_patterns:
- Browser-native agents (extensions, sidepanels)
- Workflow automation tools (Zapier-like with AI)
- Multi-agent systems coordinating tasks
The pace of change is rapid. Specific tool names will date quickly. The patterns this framework captures will remain.
1.2 Required Tools
- Site testing as agent: Test your site with Claude computer use, browser automation tools
- Schema validators — schema is even more important for agents
- Accessibility audits — agents struggle with poorly accessible sites
- Robots.txt — agent identification rules
- API endpoints — alternative to scraping for high-value workflows
2. Site Architecture for Agents
2.1 What Agents Need
Agents share many needs with traditional crawlers but differ in important ways:
agent_requirements:
shared_with_crawlers:
- Clean HTML structure
- Server-rendered content (or fast JS rendering)
- Schema markup for semantic understanding
- Sitemap for discovery
- HTTPS for trust
unique_to_agents:
- Action elements identifiable by purpose, not just appearance
- Form fields with proper labels and ARIA
- Multi-step workflows possible to complete
- Error states recoverable
- Confirmation states clear
- State preservable across navigation
- Anti-bot protection that distinguishes agents from malicious bots
2.2 Action Affordances
Where traditional SEO focused on content discovery, agentic optimization adds focus on actions agents can take.
Common agent actions on websites:
- Search for specific information
- Compare options across pages
- Fill out contact forms
- Schedule appointments
- Complete purchases
- Submit information requests
- Download resources
- Subscribe to newsletters
For each, the question: can an agent do this reliably?
Test methodology: Use Claude computer use or similar to attempt the workflow on your site. Observe:
- Can the agent identify the right element?
- Can it complete the action?
- Are error states recoverable?
- Is success confirmed clearly?
2.3 Form Optimization for Agents
Forms are common agent interaction points. Optimization:
<!-- Agent-friendly form -->
<form>
<label for="name">Full Name</label>
<input
type="text"
id="name"
name="name"
required
aria-required="true"
autocomplete="name"
placeholder="Enter your full name">
<label for="email">Email Address</label>
<input
type="email"
id="email"
name="email"
required
aria-required="true"
autocomplete="email"
placeholder="you@example.com">
<label for="subject">Subject</label>
<select id="subject" name="subject" required>
<option value="">Choose a subject</option>
<option value="quote">Request a quote</option>
<option value="support">Technical support</option>
<option value="general">General inquiry</option>
</select>
<label for="message">Message</label>
<textarea
id="message"
name="message"
required
minlength="10"
aria-describedby="message-help"></textarea>
<span id="message-help">Please describe your inquiry (minimum 10 characters)</span>
<button type="submit">Send Message</button>
</form>
Agent-friendly form patterns:
- Real
<label>elements (not just placeholders) - Proper
nameandidattributes -
autocompletehints for common fields -
requiredandaria-requiredfor required fields - Clear error messages
- Confirmation page or message after submission
2.4 Multi-Step Workflows
For workflows requiring multiple steps (e.g., booking, checkout):
multi_step_workflow_optimization:
state_preservation:
- Use URL parameters or session state agents can preserve
- Provide clear back navigation
- Allow editing previous steps
progress_indication:
- Clear "Step X of Y" indicators
- Progress visualization
- Estimated time remaining
error_recovery:
- Specific error messages identifying the problem
- Path back to fix
- Don't lose user input on errors
confirmation:
- Clear confirmation page after completion
- Reference number or confirmation token
- Email confirmation as backup
- Schema markup confirming the action
3. Bot Identification & Access
3.1 The Agent Identification Problem
In 2026, distinguishing between:
- Legitimate AI agents acting on user behalf
- Search engine crawlers
- Research/training bots (potentially desirable)
- Malicious scraping bots
- Spam bots
...is increasingly important and increasingly difficult.
3.2 Robots.txt for Agents
Building on framework-aicitations.md Section 4, the 2026 robots.txt landscape:
# Search crawlers — allow
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
User-agent: DuckDuckBot
Allow: /
# AI training/research bots — decide based on policy
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: CCBot
Allow: /
User-agent: Applebot-Extended
Allow: /
# Agent-specific identification (emerging in 2026)
User-agent: Claude-User # Claude acting on behalf of a user
Allow: /
User-agent: ChatGPT-User # ChatGPT acting on behalf of a user
Allow: /
User-agent: PerplexityBot-User # Perplexity acting on user behalf
Allow: /
# Malicious or unwanted scrapers
User-agent: AhrefsBot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: DotBot
Disallow: /
# Catch-all
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
The "Claude-User" / "ChatGPT-User" distinction is emerging — bots acting on behalf of specific users (rather than indexing for training) often want to be allowed even when training bots are blocked. Decisions are policy-specific.
3.3 IP-Based Identification
Some agent platforms publish IP ranges. Allowing those IP ranges while blocking other scrapers gives finer control than user-agent string alone.
3.4 Cloudflare and Similar Platforms
Modern WAF platforms have agent-specific rules:
cloudflare_agent_handling:
ai_audit_tab:
description: "Cloudflare's AI Audit shows traffic from major AI platforms"
actions: "Block, challenge, or allow per platform"
super_bot_fight_mode:
description: "Distinguishes verified bots from malicious"
settings:
- allow_verified_bots: true
- challenge_likely_bots: true
- block_known_bad_bots: true
custom_rules:
- Whitelist specific user agents
- Block specific user agents
- Rate limit per user agent
- Challenge based on behavior
3.5 Verification Mechanisms
For agents claiming to act on user behalf, emerging verification:
- Headers identifying the user authorization
- Cryptographic signatures
- Request rate consistent with user-driven action (not bulk scraping)
These standards are still emerging in 2026.
4. Content Patterns for Agent Consumption
4.1 What Agents Extract Differently
Agents process content differently than traditional crawlers:
- Crawlers index for ranking; surface concise extracts
- AI engines synthesize across sources for answer generation
- Agents extract specific information for task completion
For an agent helping a user "find the best web developer in Cassville, MO":
- Discover candidates
- Evaluate each (services, pricing, reviews, credentials)
- Compare options
- Present recommendation or take next action
The agent needs to extract specific data points from each candidate site to compare.
4.2 Structured Data Becomes Critical
For agent comparison workflows, schema is essential:
<!-- Service business with comprehensive schema for agent comparison -->
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "ProfessionalService",
"@id": "https://thatdeveloperguy.com/#organization",
"name": "ThatDeveloperGuy",
"description": "...",
"founder": {"@id": "https://thatdeveloperguy.com/about/joseph-anady/#person"},
"foundingDate": "2020",
"serviceArea": [
{"@type": "Country", "name": "United States"}
],
"knowsAbout": [
"Web Development",
"Search Engine Optimization",
"AI Search Optimization",
"WordPress Development",
"Custom Web Hosting"
],
"hasCredential": [
{
"@type": "EducationalOccupationalCredential",
"name": "Service-Disabled Veteran-Owned Small Business",
"credentialCategory": "certification",
"recognizedBy": {
"@type": "GovernmentOrganization",
"name": "U.S. Department of Veterans Affairs"
}
}
],
"offers": [
{
"@type": "Offer",
"name": "Custom Website Development",
"description": "Full-service custom website builds",
"priceSpecification": {
"@type": "PriceSpecification",
"minPrice": 597,
"maxPrice": 2997,
"priceCurrency": "USD"
}
},
{
"@type": "Offer",
"name": "Engine Optimization",
"description": "14-tier optimization across 112 individual optimizations",
"priceSpecification": {
"@type": "PriceSpecification",
"minPrice": 1497,
"maxPrice": 11997,
"priceCurrency": "USD"
}
}
],
"aggregateRating": {
"@type": "AggregateRating",
"ratingValue": "5.0",
"reviewCount": "47"
}
}
</script>
This level of structured data lets an agent compare ThatDeveloperGuy to alternatives systematically.
4.3 FAQ Patterns for Agent Q&A
When agents are answering user questions, FAQ pages with Question/Answer schema are particularly valuable:
<script type="application/ld+json">
{
"@type": "FAQPage",
"mainEntity": [
{
"@type": "Question",
"name": "What does ThatDeveloperGuy charge for a basic website?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Basic website builds start at $597 for foundation packages with about a 4-page scope."
}
},
{
"@type": "Question",
"name": "Are you SDVOSB certified?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Yes, ThatDeveloperGuy is a Service-Disabled Veteran-Owned Small Business eligible for federal contracting set-aside programs."
}
},
{
"@type": "Question",
"name": "What service area do you cover?",
"acceptedAnswer": {
"@type": "Answer",
"text": "We serve clients across the United States remotely from our Cassville, Missouri office."
}
}
]
}
</script>
4.4 Pricing Transparency
Agents trying to compare options need pricing visibility:
- Display pricing clearly on relevant pages
- Use Offer schema with prices
- Even "starting at" pricing is better than "contact for pricing"
- For complex pricing, provide pricing pages or calculators
5. API Endpoints for Agents
For high-value workflows, providing API endpoints alongside human-readable pages:
5.1 Why APIs Help
api_advantages_for_agents:
reliability:
- Structured response formats
- Consistent across requests
- Less prone to layout changes breaking
efficiency:
- Less data transferred
- Faster processing
- Lower agent compute cost
trust:
- APIs are explicitly opt-in for programmatic access
- Authentication mechanisms available
- Rate limiting controllable
5.2 Common API Patterns
Service catalog API:
GET /api/services
{
"services": [
{
"id": "web-dev",
"name": "Custom Website Development",
"description": "...",
"starting_price": {"amount": 597, "currency": "USD"},
"details_url": "https://thatdeveloperguy.com/services/web-development/"
}
]
}
Quote request API:
POST /api/quote-request
{
"name": "...",
"email": "...",
"service_interest": "web-dev",
"project_description": "...",
"budget_range": "..."
}
Availability API:
GET /api/availability
{
"next_available_consultation": "2026-05-08T14:00:00-05:00",
"booking_url": "https://thatdeveloperguy.com/book/"
}
5.3 API Documentation
If providing APIs, document them:
- Public-facing documentation
- OpenAPI/Swagger spec
- Authentication requirements
- Rate limits
- Versioning policy
Make them discoverable from main site (link from footer, mention in robots.txt comments).
6. The llms.txt File for Agents
Building on framework-aicitations.md Section 5, llms.txt helps agents understand site structure:
# {{Business Name}}
> One-paragraph description of what this site is and offers.
## Services and Pricing
- [Web Development](https://thatdeveloperguy.com/services/web-development/): Starting at $597
- [Engine Optimization](https://thatdeveloperguy.com/services/seo/): Starting at $1,497
- [Hosting](https://thatdeveloperguy.com/services/hosting/): $25/month
- [Computer Repair](https://thatcomputerdude.com/): $75-300 typical
## About
- [Company Overview](https://thatdeveloperguy.com/about/): Founder bio, credentials, business overview
- [SDVOSB Certification](https://thatdeveloperguy.com/sdvosb/): Veteran-owned small business details
## Contact
- Phone: 505-512-3662
- Email: joseph.w.anady@icloud.com
- [Booking](https://thatdeveloperguy.com/book/): Schedule consultation
## Frameworks and Resources
- [14-Tier Framework](https://thatstupidcomputer.com/14-tier-framework/): Comprehensive optimization framework
- [Blog](https://thatdeveloperguy.com/blog/): Educational articles
## Optional API
- [Service Catalog API](https://thatdeveloperguy.com/api/services): Programmatic service listings
This file provides agents with structured navigation help.
7. Trust Signals for Agents
Agents acting on behalf of users need confidence in source quality. Trust signals:
7.1 Credentials Visibly Marked
trust_signal_visibility:
business_credentials:
- SDVOSB certification badge with link to verification
- BBB rating with badge linking to BBB profile
- Industry certifications with verification links
- Awards with source attribution
individual_credentials:
- Education with school names
- Professional certifications
- Years of experience
- Notable clients (with permission)
schema_implementation:
- Person schema with hasCredential
- Organization schema with award
- LocalBusiness with appropriate certifications
7.2 Transparency Signals
transparency_for_agents:
pricing_visibility:
- Clear pricing where possible
- Service descriptions complete
- Terms of service accessible
process_visibility:
- How you work explained
- Typical timelines stated
- What's included clarified
contact_accessibility:
- Multiple contact methods
- Response time commitments
- Service hours
legitimacy_signals:
- Real address (or service area for SAB)
- Real phone number
- Real team members with bios
- Real portfolio/case studies
7.3 Anti-Patterns Agents Detect
Modern agents are increasingly sophisticated at detecting:
- Stock photo headshots represented as real team
- Fabricated testimonials
- Fake "as seen in" logos without actual coverage
- Inconsistent business information (NAP)
- Stale or template content
- Missing core business information
These hurt agent confidence in citing or recommending the source.
8. Measurement & Analytics
8.1 Tracking Agent Traffic
Distinguishing agent traffic from human traffic:
agent_traffic_identification:
user_agent_strings:
- "Claude-User/x.x"
- "ChatGPT-User/x.x"
- "PerplexityBot-User/x.x"
- Custom agent identifiers
ip_ranges:
- Major platform IP ranges (publicly documented)
- Reverse DNS lookups
behavioral_signals:
- Speed of interaction
- Pattern of pages visited
- Form submission patterns
- Lack of typical human dwell time
8.2 GA4 Custom Dimensions
Set up custom dimensions for agent identification:
// In GA4 / GTM
const isAgent = detectAgentTraffic(navigator.userAgent);
gtag('config', 'G-XXXXX', {
user_properties: {
traffic_type: isAgent ? 'agent' : 'human'
}
});
This lets you analyze:
- What pages do agents visit most?
- What conversions do agents generate?
- How do agent users vs human users behave?
8.3 Server Logs
Server logs catch agent activity GA4 can't:
# Identify agent traffic from logs
grep -E "(Claude|ChatGPT|Perplexity|Anthropic|GPT)" /var/log/nginx/access.log | wc -l
# Top pages accessed by agents
grep -E "(Claude|ChatGPT|Perplexity)" /var/log/nginx/access.log | \
awk '{print $7}' | sort | uniq -c | sort -rn | head -20
9. Audit Mode
| # | Criterion | Pass/Fail |
|---|---|---|
| AG1 | Site renders content server-side or via fast SSR | |
| AG2 | All forms have proper labels and ARIA | |
| AG3 | Multi-step workflows preserve state | |
| AG4 | Robots.txt explicitly addresses agent user-agents | |
| AG5 | Schema markup comprehensive across content types | |
| AG6 | Pricing visibility on relevant pages | |
| AG7 | FAQ schema for common questions | |
| AG8 | llms.txt file present | |
| AG9 | API endpoints provided for high-value workflows (optional) | |
| AG10 | Trust signals visible and structurable | |
| AG11 | Credentials in Person/Organization schema | |
| AG12 | Site tested with actual agent (Claude computer use, etc.) | |
| AG13 | Anti-bot configured to allow legitimate agents | |
| AG14 | Agent traffic monitoring in analytics | |
| AG15 | Form submissions completable by agents end-to-end | |
| AG16 | Confirmation states clear after agent actions |
Score: 16. World-class agent optimization: 14+/16.
10. Common Mistakes
- Heavy JavaScript without SSR — agents struggle with client-rendered content
- Forms without labels — agents can't identify fields reliably
- Aggressive anti-bot — blocks legitimate agents along with malicious ones
- Pricing hidden — agents can't compare without pricing visibility
- No schema for service offerings — agents can't extract structured info
- No FAQ markup — common questions not extractable
- No llms.txt — missed agent navigation help
- Confirmation pages with no clear success state — agents can't verify success
- Multi-step workflows that lose state — agent can't recover from errors
- Ignoring agent traffic in analytics — missing emerging traffic source
11. The Frontier (Things Likely to Change)
This is genuinely a 2026 frontier. Things likely to evolve:
- Standardized agent identification protocols — beyond user-agent strings
- Cryptographic verification — proving agents act with user consent
- Agent-specific access tiers — paid vs free agent access patterns
- Content licensing for agent consumption — beyond current AI training opt-outs
- Action confirmation protocols — verification user actually wanted action taken
- Multi-agent coordination — agents calling other agents
- Agent trust networks — reputation systems for agents
Monitor for emerging standards. Update implementation as patterns stabilize.
End of Framework Document
Document version: 1.0
Last updated: 2026-04-29
The agentic AI layer is genuinely emerging. Optimization patterns will mature over coming years. The principles in this framework — clean architecture, accessible interactions, trust signals, structured data — are durable. Specific tactics will evolve.
Companion documents:
-
framework-aicitations.md— AI engine citation in conversational answers -
framework-multimodalsearch.md— Multimodal AI understanding (related) -
framework-schema.md— Structured data foundations -
framework-technicalseo.md— Crawlability foundations
From the ThatDevPro Engine Optimization framework library. Studio: ThatDevPro (SDVOSB veteran-owned web + AI engineering). Sister property: ThatDeveloperGuy. Source: https://www.thatdevpro.com/insights/framework-agenticaisearch/.
