The ongoing large scale adoption of AI is rapidly changing the way many people interact with the web. Therefore websites and web-apps must adapt so they can continue to thrive. In this article i will go over some improvements you can make to your Symfony application to start making it agent-ready.
Since i want to cover multiple things in this article i will not be going into great detail on each of them, so consider this article more as an introduction to agent-ready-ness and these concepts. I did however write a small bundle for one of the concepts and for the last one I wrote a basic implementation prompt which can be found in the last chapter.
A great resource that will likely help you on this journey and going forward is isitagentready.com. This tool gives a good overview of the improvements you can make to your site and is in part what this article is based on.
SEO Fundamentals
Before getting into any agent specific improvements, it is best we first glance over some SEO fundamentals.
The reason for this is that most things we consider SEO improvements are generally machine- readability and discoverability improvements, and it just so happens that agents are machines too. That is why i would suggest having your SEO fundamentals on point first, since they will not only help with SEO but they're a nice baseline from which to start getting agent-ready
Now, there's a good chance you already have (most) these in place but if you don't i'd highly recommend you look them first before focusing on agents. I will not go into detail for these, if you do want more information on them you can find plenty online, or ask your agent.
- Security make sure your site is secure and served over https
-
Pages must be crawlable requests should return the correct status codes and not be
noindex -
robots.txtYou should have this setup including the sitemap direcive - Sitemap A proper XML sitemap that contains all the links you want to be crawled / indexed.
- Inform indexers Setup google search console and Bing webmaster tools and submit your sitemap. Also check these tools occasionally for any indexing warnings
-
Canonical URLs make sure each page has a specified canonical URL using
rel="canonical" - Proper titles and descriptions Make sure each page has proper title's and descriptions
-
Good semantic HTML using elements like:
nav,main,footer,h1etc. Where appropriate -
Structured data implementing structured data where it fits things like:
BreadcrumbList,Product,FAQPage - Fast, mobile-friendly and accessible your site should be fast, accessible and work on all devices.
Content Signals
Starting with content signals this is the simplest improvement you can make. content-signal is a proposed robots.txt directive, meant to communicate what you permit AI to do with content from your site. A simple example:
User-Agent: *
Content-Signal: ai-train=no, search=yes, ai-input=yes
Allow: /
What each of the categories mean:
-
ai-trainYour content may be used for training or fine-tuning AI models -
searchYour content may be used for building a search index and providing search results. This is more so related to traditional search indexes -
ai-inputYour content may be used as input for AI models. This mainly means things like AI web search tools and does not include training
Generally you'll want search and ai-input to be yes for the purpose of indexing in search and for use by AI through web search tools. I would consider these two standard for most websites, even if you don't want AI to train on your content.
ai-train is more personal and project dependent. The obvious con is that AI may copy your idea's, content or style. But in the case your site is more commercial and features your brand name in the text having AI train on it could be a positive.
It is also possible to apply specific content signals for specific URLs, more information on this is best found on contentsignals.org
Markdown Negotiation
Markdown negotiation is the ability for agents to request your pages as text/markdown instead of HTML, using the Accept: text/markdown request header.
The reason we'd want to implement this is that agents speak Markdown very well, far beter than HTML and it is also much more token efficient. In a lot of cases the HTML you would otherwise return gets turned into Markdown anyway before an AI sees it. Therefore if you create the Markdown yourself you have a lot more control over it.
There are two general ways to implement this:
- Explicit markdown responses: checking in each route if the request prefers markdown and returning an appropriate response.
- Centralized HTML-to-Markdown conversion: turning the HTML returned by routes into Markdown centrally.
These two complement each other well, generally for the most control you'll want explicit Markdown responses and then fall back to passive HTML-to-Markdown conversion for pages that are less important.
Initially in this article i was going to give a high level overview of how to implement both of these options. Instead I decided that this would be far better to just turn into a small bundle, so i present: tomvdpeet/markdown-negotiation-bundle. The bundle is mainly focused on passive HTML-to-markdown conversion, here is an example:
#[Route('/docs', name: 'docs', options: ['markdown' => true])]
public function docs(): Response
{
return $this->render('docs/index.html.twig');
}
This example will turn the rendered html in to clean Markdown when requested, simply by adding options: ['markdown' => true] to the Route attribute. The resulting Markdown will also be stripped of stuff like footer, nav, head and comparable layout/decorative HTML content.
The bundle also has some other features like supporting a dev-only ?_markdown GET parameter that will handle the request as if it prefers Markdown for easier debugging. And some tools for explicitly specified Markdown responses as well. Though the main focus is a DX-friendly and minimal overhead implementation of the centralized conversion.
API Exposure
The previous two changes we looked at were purely about machine- readability and discoverability, API Exposure on the other hand opens the door to machine-usability. So before going into API Exposure itself I want to touch on the change in perspective this goes hand in hand with.
Most websites, web-apps, platforms, etc. are user facing and support a user interface, UI. This makes sense since humans interact with them, right now. However for our systems to be agent-ready, we will also need to make them agent-usable and thus machine-usable. This change requires us to look at our systems increasingly more as an API based system rather then just a UI based system.
I believe API Exposure is a nice entry-point into making a Symfony application more agent-ready and machine-usable, since it is the way an agents discover the capabilities and API's your application provides.
API Exposure (RFC 9727) defines a standard URL, API catalog (/.well-known/api-catalog) that lists the available API's and documentation for them in a standard JSON based format.
Since i didn't want to add a full implementation tutorial in this article i did write a prompt to have an agent implement a basic setup of this for you. To help you get started with this the prompt will also setup OpenApi docs for your search page, if you have one. It does rely on the nelmio/api-doc-bundle to do this.
As mentioned API exposure is about making your API's and functionalities discoverable. I am not going into further detail on this because it it very application depended. But you should now have a stable and expandable foundation for making any API's and functionalities you add discoverable.
Conclusion
The shift to a more agentic internet will mean that we need to shift the perception of our own applications to be more and more API like. Since all of this is still very new, standards and conventions are still uncertain and changing quickly. However i believe that what I've covered here is the right direction and what internet will be moving to sooner or later. So getting a head start on implementing or at least learning about these standards will help you and your application continue to thrive in the .
original article: https://vindle.nl/insights/getting-agent-ready-with-symfony
Prompts
Api exposure prompt
Make this Symfony app more agent-ready by adding RFC 8288 Link response headers, an RFC 9727 API catalog, and OpenAPI docs with NelmioApiDocBundle.
Use the smallest pragmatic change that fits the existing codebase. Read the existing routing/controller structure first and preserve current behavior.
Tasks:
1. Install and enable NelmioApiDocBundle
- If not already installed, run:
composer require nelmio/api-doc-bundle
- Verify the bundle is registered in config/bundles.php.
- Add or update:
config/packages/nelmio_api_doc.yaml
config/routes/nelmio_api_doc.yaml
2. Expose OpenAPI docs
- Add JSON docs at:
/api/doc.json
using nelmio_api_doc.controller.swagger
- Add Swagger UI at:
/api/doc
using nelmio_api_doc.controller.swagger_ui
- Configure Nelmio metadata with the project name, description, version, and server URL.
- Keep operation details on controller route methods using OpenAPI attributes, not hard-coded YAML paths.
3. Document the search page if the project has one
- Find the existing public search route, for example /search or /zoeken.
- Add OpenAPI route-level attributes to that controller method:
- tag: Search
- query parameter: q, optional string
- 200 response with text/html
- if the app supports markdown content negotiation, also add text/markdown
- Update nelmio_api_doc.areas.path_patterns so the search route is included in generated docs, while excluding /api/doc itself.
Example attributes:
use OpenApi\Attributes as OA;
#[OA\Tag(name: 'Search')]
#[OA\Parameter(
name: 'q',
description: 'Search term.',
in: 'query',
required: false,
schema: new OA\Schema(type: 'string'),
)]
#[OA\Response(
response: 200,
description: 'Search results page.',
content: [
new OA\MediaType(
mediaType: 'text/html',
schema: new OA\Schema(type: 'string'),
),
new OA\MediaType(
mediaType: 'text/markdown',
schema: new OA\Schema(type: 'string'),
),
],
)]
4. Add an RFC 9727 API catalog
- Add a route:
/.well-known/api-catalog
- Return application/linkset+json.
- Include links to:
- the OpenAPI JSON document as service-desc
- the Swagger UI as service-doc
- the public search route as item/search-related discovery if available
- Add a Link response header on the catalog response:
Link: </.well-known/api-catalog>; rel="api-catalog"; type="application/linkset+json"
Example linkset shape:
{
"linkset": [
{
"anchor": "https://example.com/.well-known/api-catalog",
"item": [
{
"href": "https://example.com/search{?q}",
"type": "text/html",
"title": "Product search"
}
],
"service-desc": [
{
"href": "https://example.com/api/doc.json",
"type": "application/vnd.oai.openapi+json",
"title": "OpenAPI description"
}
],
"service-doc": [
{
"href": "https://example.com/api/doc",
"type": "text/html",
"title": "API documentation"
}
]
}
]
}
5. Add homepage Link discovery headers
- Add a Link response header to the homepage response.
- Prefer putting this directly in the homepage controller if it is homepage-only.
- Use one combined RFC 8288 Link header value.
Example:
Link: </.well-known/api-catalog>; rel="api-catalog"; type="application/linkset+json", </search>; rel="search"; type="text/html"; title="Site search", </api/doc.json>; rel="service-desc"; type="application/vnd.oai.openapi+json"; title="OpenAPI description", </api/doc>; rel="service-doc"; type="text/html"; title="API documentation"
6. Validate
Run the relevant checks for the project, at minimum:
- php bin/console lint:yaml config/packages/nelmio_api_doc.yaml config/routes/nelmio_api_doc.yaml
- php bin/console debug:router
- php bin/console lint:container
- php bin/console nelmio:apidoc:dump --format=json
- project tests, or targeted tests for the new API catalog/controller behavior
7. Documentation
- Update project docs to mention:
- /api/doc.json
- /api/doc
- /.well-known/api-catalog
- homepage Link headers
- OpenAPI operation details live as route-level attributes
