The Discovery Layer: What an AI Agent Actually Reads on Your Site

Most of what gets written about “preparing your site for AI agents” describes files that no vendor documents reading. I went through the official documentation from OpenAI, Anthropic, Perplexity, Google and Microsoft. The result: llms.txt — not one of these companies documents reading it from third-party sites, and Google states explicitly that its Search ignores it. sitemap.xml — not mentioned in any of the three bot documents. schema.org markup — Google writes that it is not required for its generative features. And in the IANA registry, out of 152 entries, there is no registration for mcp, ai or agent. What does exist — and what is actually worth doing — is below.

The short list of what is actually documented

There is a difference between a file that exists and a file that someone reads. Almost every guide on this subject blurs the two, so I built a table that separates “exists as an idea” from “has official vendor documentation of being read”.

llms.txt: everyone publishes one, nobody documents reading one

This file is a proposal by Jeremy Howard, published on 3 September 2024, updated to version 2 on 10 August 2026, and described on its own site as a proposal to standardise. It is not on any IETF, W3C or WHATWG standards track — a search of the IETF Datatracker returns no relevant draft. The proposal’s own site names RFC 8615 as an alternative that was considered and not adopted.

Here is the central confusion I keep running into: OpenAI, Anthropic and Perplexity do publish an llms.txt file for their own developer documentation. That is factual and easy to check. But publishing a file and reading other people’s files are two entirely different actions, and none of these companies documents the second one. I read all three official crawler pages; the string llms.txt does not appear in any of them.

Google is the only vendor that has addressed this directly. In its optimization guide for generative features, on a page updated 10 July 2026, there is a mythbusting section that names llms.txt explicitly, on the grounds that Google Search itself does not use them. In the same section Google adds that creating the file will neither harm nor help your site’s visibility or rankings.

schema.org: not required, still worth it

This is the point that surprises people most. That same Google guide from 10 July 2026 has an entire section titled “Overfocusing on structured data”, which states that structured data is not required for generative AI search and that there is no special schema.org markup you need to add. The parallel page on AI features, updated 10 December 2025, makes the same claim in different words: you do not need to create new machine readable files or dedicated markup.

And yet Google still recommends continuing to use structured data — not because of AI, but because of rich result eligibility, which is an excellent reason on its own. The additional recommendation that is directly relevant here is to make sure your structured data matches the visible text on the page. Schema that declares something the page does not say is a liability, not an advantage.

The practical conclusion: if you have valid schema, great, keep it. If you do not, add it because it helps rich results — not because someone promised it gets you into ChatGPT’s answers.

sitemap.xml: the assumption with no source

“Submit a sitemap so AI crawlers can find you” is one of the most common recommendations out there, and I could not find a single primary source for it. OpenAI’s bot page discusses robots.txt and IP ranges only. Anthropic’s crawler article discusses robots.txt and Crawl-delay. Perplexity’s crawler page does not mention sitemaps at all.

This does not mean they do not read sitemaps — it means nobody has said they do. The distinction matters, because it determines how much effort is worth spending. A sitemap.xml is table stakes for Google, so you should have a valid one regardless. But if you are building an entire workplan around the sitemap as “infrastructure for AI agents”, you are building on an assumption nobody has confirmed.

/.well-known/ — what is actually registered, and what is not

The IANA well-known URI registry is the single source of truth for what exists. As of its last update, 19 August 2026, it holds 152 entries. What matters for the agent world:

• agent-card.json — registered, permanent, under Linux Foundation change control, pointing at the A2A protocol specification. It is the only entry that concerns agents directly.
• oauth-protected-resource (RFC 9728) and oauth-authorization-server (RFC 8414) — registered, and relevant because MCP builds its authorization layer on them.
• mcp, ai, agent, ai-plugin, llm — are not registered.

As for /.well-known/ai-plugin.json from the ChatGPT plugins era: that mechanism is gone. The original documentation page now redirects to a page that does not exist, and what replaced it is MCP servers and the Apps SDK — with no well-known path at all.

MCP is not discoverable from a website — yet

This is the point that matters most if you are planning ahead. The current Model Context Protocol specification is version 2026-07-28, and it contains no mechanism for discovering an MCP server from a website. What does exist is server/discover, which asks a server which protocol versions it supports — a mechanism that assumes you already know the server’s address. The only well-known path in the specification belongs to OAuth, and it only comes into play after discovery.

In practice this means there is an open infrastructure gap between “my site is readable to an agent” and “an agent can discover what my site lets it do”. That is precisely why I am working on the AX specification — a uniform discovery layer that lets a site declare what it offers an agent. But as long as there is no agreed standard, the honest recommendation is not to invent a private path and hope somebody reads it.

IndexNow: Microsoft’s exception

The one thing a major vendor does explicitly recommend in an AI context is IndexNow. In the Bing Webmaster post from 10 February 2026 that introduced the new AI performance report, there is a dedicated section explaining that IndexNow helps keep information fresh across search and AI experiences. The protocol itself is open and free, and the implementation is a single HTTP call per updated URL.

One distinction worth noting: Bing’s developer documentation does not mention llms.txt at all. The schema recommendations in an AI-answers context appear on the Microsoft Advertising blog, which is marketing content rather than technical documentation — a different evidentiary weight.

A practical order of priority

Three lines that will tell you what your site already exposes today:

Once you have checked, this is the order I recommend. First, a correct robots.txt that separates training crawlers from answer-time fetchers — I broke that down in the 2026 bot map. Second, verifying that your content actually reaches the initial HTML, because without that nothing else matters — the test is in what an AI agent actually receives. Third, valid schema that matches the visible text. Fourth, IndexNow if your content changes. And fifth, only fifth, llms.txt — if there is time left over.

What is interesting about that order is how closely it mirrors classic technical SEO. That is not a coincidence: as of today, most of what makes a site readable to an agent is the same set of things that makes it readable to a search engine. The special files are the promise; the fundamentals are what works.

Frequently asked questions

Continue reading

Agentic Commerce in 2026: Which Protocol Should You Actually Implement? – Once an agent can discover you, the question becomes whether it can transact with you.