Visibility basicsWhich AI crawlers do you allow, and what does blocking cost you?
My advice is to allow all of them. Not because the choice doesn't matter, but because blocking buys you less than you think and costs more than you think. Below is why, what the alternatives are, and a robots.txt you can copy.
That choice only makes sense once you know what these robots actually do, and that starts with how an AI answer gets made. AI search engines search first and write afterward. If you're not in that index, you're not in the answer.
What happens between the question and the answer?
Roughly three steps. The model doesn't invent the answer on its own; it gets handed material and summarizes it.
- Retrieval. Your question is turned into one or more queries against a search index. Sometimes the provider's own index, sometimes a search engine's.
- Grounding. The pages that come back are cut into passages, and the most relevant passages get passed along as context.
- Writing. The model composes an answer from those passages and attaches references.
The practical lesson is in step two: citation happens at the level of passages, not pages. So you're not competing with your whole article, but with the paragraph that answers best.
Why ordinary SEO still matters most
Because step one is a search. Whatever happens in steps two and three can only happen with sources that surfaced in step one.
That makes the order of your effort fairly clear. First make sure you're findable in ordinary search engines. Then make sure your text is easy to quote. That's a different order from the one a lot of providers are selling right now.
I don't know of a single provider that has publicly documented how sources get chosen exactly. Anybody handing you a precise recipe is filling the gap with an assumption. That's fine, as long as they say so.
Why your site has to work without JavaScript
As far as anybody knows, most crawlers run by AI providers execute little or no JavaScript. Anything put on the page by a script after loading then doesn't exist for them.
I should say honestly that no provider publishes how far their crawler goes here. So treat it as an assumption, not a fact. The good part is that you don't have to rely on it: you can check it yourself. Turn JavaScript off in your browser and reload your most important page. Whatever you can't see then is exactly the part you can't be sure counts.
This website deliberately uses no browser JavaScript at all. Everything is simply in the HTML.
Three choices, and the one I would make
There are three coherent positions. They differ not in technique but in what you gain and give up.
- Open. Allow search, user-triggered fetching and training. The best chance of being named and cited.
- Search yes, training no. Allow the search indexes and the fetchers, refuse the training crawlers. You stay citable live, your text does not go into a model.
- Closed to AI. Refuse everything with AI in the name.
I would choose open, even when your product is content itself. That is a judgement about an uncertain situation rather than a settled fact, so here is the reasoning, and you are welcome to disagree with it.
My reading is that your content gets pulled either way, whether you protect it or not. If that holds, the question isn't whether it happens but what you still get out of it while you can. And there's a competitive side that weighs more than the principle: block while your competitor doesn't, and they get named and you don't. They build recognition in exactly the channel you withdrew from.
Search yes and training no is defensible. It is also the option you have to explain to yourself the most, because you are refusing something you cannot verify was refused, and you do give something up.
Closed to AI I would not recommend to anybody who wants to be found. You are saying no to the search indexes of ChatGPT, Perplexity and Claude, and those are where more and more people start their question.
A robots.txt you can copy
For the open choice it is short. This is essentially what sits on this website.
# Every crawler is welcome.
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xmlIf you do want search yes and training no, you list the training crawlers separately. A robot that has its own group ignores the group with the asterisk, so the order below is right.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xmlTwo things go wrong here. Never block Googlebot, because you then disappear from Google and therefore from AI Overviews too. And Google-Extended isn't a crawler at all: nothing ever arrives under that name. It's a switch for what Google may do with pages Googlebot has already fetched. That doesn't make it less useful, but it does make it a different kind of thing from the rest of this list.
Which robot does what is in the table on the visibility basics, with each provider's own documentation beside it.
robots.txt is a signal, not a wall
This is the part that gets skipped most often. robots.txt is a request. Well-behaved crawlers honor it, others don't, and you have no way at all to enforce it.
That isn't a hunch. Cloudflare tested it and published the result. They set up brand new, unpublished test domains with a robots.txt forbidding all automated access, then asked Perplexity questions about those domains. What they found: alongside the declared Perplexity-User, an undeclared crawler presenting itself as ordinary Chrome on macOS, using IP addresses outside their own official ranges. Cloudflare put the scale at tens of thousands of domains and millions of requests per day.
Cloudflare's own description of robots.txt is that it expresses "website directives and preferences", so preferences rather than binding rules.
For some robots it cannot work by definition. ChatGPT-User fetches a page because a user asked for it at that moment, and requests like that don't reliably fall under your robots rules.
What does that mean for your choice? If blocking is partly a signal, and entirely dependent on the other party's goodwill, then it buys you less protection than the effort suggests. If you genuinely want something kept shut, it belongs behind a login, or it doesn't belong on the page. robots.txt is the wrong tool for that job.
You cannot opt out of AI Overviews
Not through robots.txt. This is the misconception that costs the most money, because people sit down to do it and get nothing back.
AI Overviews are built on Google's ordinary search index. That index is filled by Googlebot. To get out of AI Overviews you would have to refuse Googlebot, and then you disappear from normal search results as well. That is not a trade anybody wants to make.
Blocking Google-Extended doesn't help here: that's about Gemini, not about AI Overviews. Google writes it out themselves: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."
The only real levers are nosnippet and max-snippet, and they cut into the same flesh: they also shrink or remove your ordinary search snippet. You would be trading visibility in the search results for slightly less text in an AI answer. For almost everybody that is a bad trade.
Source: Google on its crawlers.
Being easy to lift out works in your favor
The other side of the same choice. If you decide to leave the door open, make sure there is something worth taking that stands on its own.
Citation happens per passage, not per page. A paragraph that makes sense on its own gets used. A paragraph where the answer only arrives in the fourth sentence doesn't. That is the same rule as in the basics: answer first, one claim per paragraph, with the evidence in that same paragraph.
You will see this sold as extractability. The idea underneath is right and it isn't new. Be careful with the numbers attached to it, though. For that term I could find only agencies, tool vendors and glossary sites as sources, no provider documentation and no independent research. There is a citation rate for one writing style against another going around, for instance, with no checkable measurement behind it.
That doesn't make the advice wrong. It makes the precision suspect. Do it because it is sensible and because it makes your text better for people too, not because a percentage is attached to it.
What do you do with this?
Four things, in this order.
- Look at what your
robots.txtsays right now. Often it holds something somebody put there years ago that nobody understands any more. - Pick one of the three positions and write down why. If you can't explain it, it isn't a choice, it's a setting.
- Put the answer first under every heading, so a single passage makes sense on its own.
- Keep building ordinary search visibility. That's still where you get the most.
This article belongs to the online visibility basics, where the whole foundation is laid out. If you'd rather not do it yourself, it's part of what I do under SEO and AI visibility.
One caveat that runs through the whole series: getting found is one part of a marketing strategy, not the whole of it.
Don't know what your robots.txt says?
Send me your website. I'll tell you which robots are coming in, what that earns you and what it costs you.
Dominique@framvekst.noYou'll hear back within three business days.
See my contact details