SEO for AI: how to get your website found in the era of agent search

A practical guide for business owners who don’t want to be invisible when customers ask an AI instead of searching on Google.

---

Search has moved

For more than twenty years, being found by customers meant showing up on Google and similar search engines. You optimized your site, climbed the rankings, and waited for the click.

Now there is a second path. More and more people open ChatGPT, Perplexity, Claude, or Gemini and ask directly: “what is the best accounting firm in São Paulo?”, “how does the MEI (Brazil’s micro-entrepreneur regime) work?”. The AI doesn’t return ten blue links. It returns one answer, often citing two to four sources.

This changes the rules of the game: instead of competing for a position in a list, you compete to be one of the sources the AI chooses to read and cite. This new work is called GEO (Generative Engine Optimization) or AEO (Answer Engine Optimization). It doesn’t replace traditional SEO. It adds to it.

The good news: much of this work is simple, cheap, and within reach of any website. The bad news: almost nobody has done it yet. In April 2026, Cloudflare analyzed the 200,000 most visited domains on the internet and concluded that the web is still very poorly prepared for AI agents. Only about 4% declared their AI preferences through Content Signals, and fewer than 4% served content in markdown to agents.

That is a window of opportunity. Let’s get to the concepts.

---

First, understand who is visiting your website

There used to be only two kinds of visitors: people and search robots (like Googlebot). Today there is a third, and it splits into two very different types:

• Training crawlers: collect content in bulk to teach AI models (examples: GPTBot, ClaudeBot, Google-Extended).
• AI agents and search engines: look up information on the spot to answer someone’s question, like Perplexity or ChatGPT’s search mode. This is where you win (or lose) citations and customers.

This distinction matters because you may want to be cited by the second group without giving your content away for free to the first. That is exactly what the concepts below make possible.

---

1. robots.txt: the front door

The robots.txt is a text file at the root of your website (yoursite.com/robots.txt) that tells robots what they can access. It has existed since the 1990s and is still the first thing any reputable AI bot checks.

The most common mistake: blocking AI robots out of habit (or because of an aggressive security plugin) and, without realizing it, making your site invisible to AI-generated answers.

A simple, healthy example:

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

If you want to block a specific robot (for example, a training crawler), you can do it individually:

User-agent: GPTBot
Disallow: /

The problem is that this model is “all or nothing” per robot. The next concept was created to solve that.

---

2. Content Signals: saying what AI can do with your content

Content Signals is an extension of robots.txt proposed by Cloudflare. Instead of only saying “you may enter” or “you may not enter”, you declare what for your content can be used. There are three signals:

• search: can be indexed to appear in search results (links and snippets);
• ai-input: can be used as a source to generate AI answers in real time;
• ai-train: can be used to train or fine-tune models.

Each one gets yes or no. A typical example for those who want to appear in AI answers but don’t want to become training material:

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

In plain English: “You may index me. You may use my content to answer questions and cite me. Do not use it to train models.”

A point of honesty: Content Signals are a declaration of preference, not a technical wall. They express your intent in a standardized way, and serious crawlers tend to respect it, but they do not replace real blocking mechanisms when strict protection is the goal.

---

3. llms.txt: a “business card” for AI

The llms.txt is a markdown file placed at the root of the site (yoursite.com/llms.txt) that introduces your business to language models in a clean and concise way.

Think of it this way: your site is built for people, full of menus, banners, pop-ups, and visual code. An AI has to discard all of that to get to what matters. The llms.txt is a shortcut: “who we are, what we do, and where the important pages are”.

The structure is simple: a title, a quoted summary, and lists of links organized by section. See the example below:

# XPTO Accounting

> Accounting firm in São Paulo (SP) specialized in MEI (Brazilian sole proprietors),
> small businesses, and self-employed professionals. In-person and online service.

## Services

- [Company registration](https://contabilidade-xpto.com.br/abertura-de-empresa): how to open a CNPJ, timelines, and costs
- [Monthly accounting](https://contabilidade-xpto.com.br/contabilidade-mensal): plans for MEI and Simples Nacional
- [Income tax](https://contabilidade-xpto.com.br/imposto-de-renda): returns for individuals

## Useful content

- [MEI guide](https://contabilidade-xpto.com.br/guia-do-mei): obligations, limits, and how to avoid fines
- [Frequently asked questions](https://contabilidade-xpto.com.br/perguntas-frequentes): common customer questions

## Contact

- [Get in touch](https://contabilidade-xpto.com.br/contato)

Be realistic about its reach. The llms.txt is a relatively recent standard proposal and adoption by AI companies is still uneven: not every agent checks it. But the cost of creating it is very low, it forces you to get your own house in order, and coding agents and documentation tools already use it. It is a cheap investment with room to grow.

---

4. Negotiated markdown: serving the lightweight version

This is a more technical concept, but easy to understand. When a browser requests a page, it receives plain HTML. Some AI agents can already ask the server: “do you have a markdown version of this page?” (through a header called Accept: text/markdown).

If your site responds with a clean version, without menus and scripts, the agent spends fewer resources reading it and understands the content better. Cloudflare itself measured the effect on its documentation: 31% fewer tokens consumed and 66% faster responses after adopting these standards.

For most small businesses, this is something the site developer or hosting platform implements once. What you need to know is that it exists and that it is worth asking your technical team.

---

5. Structured data: what was already good still is

Not everything is new. Schema.org (JSON-LD markup) remains one of the best ways to tell machines, precisely, what each page is: a local business, a product, a recipe, an FAQ, an event.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "LocalBusiness",
  "name": "XPTO Accounting",
  "url": "https://contabilidade-xpto.com.br",
  "telephone": "+55-43-0000-0000",
  "address": {
    "@type": "PostalAddress",
    "addressLocality": "São Paulo",
    "addressRegion": "SP",
    "addressCountry": "BR"
  }
}
</script>

This helps both Google and AIs understand your business without guessing.

---

6. Standards for more advanced agents

If you have an online store, a system, or an API, it is worth knowing the names that will show up on your radar:

• MCP (Model Context Protocol): a standard for AI agents to connect to tools and data in an organized way.
• Agent Skills and API Catalog: ways to announce what your service can do to agents.
• Agentic commerce protocols: emerging standards that let an AI buy or pay on the user’s behalf.

For most business websites, this is not necessary. But for those who sell online, it is something to follow closely.

---

7. Don’t forget the basics that have always worked

AIs cite sources that look trustworthy and easy to understand. This favors what good SEO has always preached:

• Clear and direct content that answers real customer questions (a well-made FAQ page adds a lot of value);
• Visible authorship and credibility: who writes, what their experience is, what the address is, how to reach you;
• A fast, secure (HTTPS), well-structured site, with logical headings and subheadings;
• Up-to-date information, with dates and revisions;
• Consistent presence elsewhere (Google Business Profile, social networks, mentions on trusted sites), because AIs also cross-check sources.

---

How do you know if your website is ready?

There is a free way to measure it. In April 2026, Cloudflare launched isitagentready.com, which scans any website and checks, in seconds, whether it adopts these standards: robots.txt, sitemap, Content Signals, llms.txt, negotiated markdown, MCP, and others. For each missing item, the tool generates ready-made text that your developer (or a coding AI) can use to implement the fix.

An important caveat: the score is not the goal. Some test items, like OAuth or payment protocols, make no sense for a simple business website. Use the report as a map of what exists and choose what really makes sense for your business.

---

Your 5-step action plan

1. Test your site at isitagentready.com and see where you stand today.
2. Review your robots.txt: make sure you are not blocking AIs by accident and add your Content Signals.
3. Create an llms.txt summarizing your business and your main pages.
4. Add structured data (Schema.org) to your most important pages.
5. Write content that answers real questions from your customers, clearly and up to date.

---

Conclusion

Search is not going away, but it is splitting: some people will keep clicking links, and others will simply ask and trust the answer they get. Those who prepare for both scenarios will have an advantage while most websites don’t even know this conversation has started.

The best time to prepare is now, while the competition is still distracted.

---

References

• Cloudflare Blog, Introducing the Agent Readiness score (April 17, 2026): analysis of the 200,000 most visited domains (78% with robots.txt, 4% with Content Signals, 3.9% with negotiated markdown, fewer than 15 with an MCP Server Card or API Catalog) and a benchmark of Cloudflare’s documentation (31% fewer tokens, 66% faster responses).
• Cloudflare Radar, Adoption of AI agent standards: chart of standards adoption, updated weekly (the figures in this article are from April 2026).
• isitagentready.com: Cloudflare’s free checking tool.
• Content Signals: specification of the search, ai-input, and ai-train signals.
• llmstxt.org: proposal for the llms.txt format, presented in September 2024.