Visibility basicsWhich AI crawlers should you allow, and what does blocking cost you?
My advice: allow all of them. Not because the choice doesn't matter, but because blocking buys you less and costs you more than you'd think. Below is why, what the alternatives are and a robots.txt you can copy.
To make that choice well, you need to know what these crawlers actually do. It starts with how an AI answer gets made: first there's a search, then the writing. If you're not in the index being searched, you're not in the answer.
What happens between the question and the answer?
Three steps: search, grounding and writing. The model doesn't make up the answer on its own. It gets handed material and summarizes it.
- Search. Your question gets turned into one or more queries. Sometimes against the provider's own index, sometimes against a search engine's.
- Grounding. The pages that come back get cut into passages, and the ones that fit best get passed along as context.
- Writing. The model writes an answer from those passages and adds references.
The practical lesson is in step two. Citation happens passage by passage, not page by page. So you're not competing with your whole article. You're competing with the paragraph that answers best.
Why does ordinary SEO still matter most?
Because step one is a search. Only what surfaces there can be used in steps two and three.
That settles the order of your work. First make sure you can be found in ordinary search engines. Then make sure your text is easy to quote. A lot of providers are selling it the other way around right now.
No AI provider has publicly documented exactly how sources get chosen. Anyone handing you a precise recipe is filling that gap with an assumption. That's fine, as long as they say so.
Why does your site have to work without JavaScript?
Because as far as anyone knows, most AI crawlers run little or no JavaScript. Anything a script puts on the page after it loads doesn't exist for them.
No provider publishes how far its crawler goes here, so treat this as an assumption, not a fact. The good news is that you can check it yourself: turn off JavaScript in your browser and reload your most important page. Whatever you can't see then is the part you can't be sure counts.
This website deliberately uses no JavaScript in the browser at all. Everything is right there in the HTML.
What are your options, and which one would I pick?
Three: open, search yes and training no, or closed to AI. I'd pick open. They don't differ in technique, but in what you gain and what you give up.
| Position | What you allow | What you gain | What you give up |
|---|---|---|---|
| Open | Search indexes, user-requested fetchers and training crawlers | The best chance of being named and cited | Control over reuse of your text, which robots.txt only partly gave you anyway |
| Search yes, training no | Search indexes and user-requested fetchers; no training crawlers | AI can cite your text live, but it doesn't go into a model | You're missing from future training data, which I expect makes you steadily less relevant |
| Closed to AI | Nothing AI-related | Your text stays out of AI, as far as crawlers respect that | No more mentions in ChatGPT, Perplexity or Claude search results |
I'd pick open, training crawlers included, even if your product is text itself. That's a judgment call in uncertain times, not a settled fact, so here's the reasoning, and you're welcome to disagree.
My read is that your text gets pulled either way, whether you shield it or not. If that's true, the question isn't whether it happens but what you still get out of it while you can. What matters more to me is the competition. Block while your competitor doesn't, and they get named and you don't. They build recognition in exactly the channel you pulled out of.
Search yes and training no is defensible, but I'd advise against it. I expect it to make you steadily less relevant, because a model knows less and less about what isn't in its training data. And you're refusing something you can't even verify is being honored.
Closed to AI isn't something I'd recommend to anyone who wants to be found. You'd be saying no to the search indexes of ChatGPT, Perplexity and Claude, and that's where more and more people start looking.
A robots.txt you can copy
For the open choice, it's short. This is essentially what's on this website.
# Every crawler is welcome.
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xmlIf you go with search yes and training no, give the training crawlers their own group. A crawler looks for the group with its own name and follows only that one. If it doesn't have one, it follows the group with the asterisk. The order of the groups doesn't matter.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xmlTwo things often go wrong here. Never block Googlebot: you'd disappear from Google, and with it from AI Overviews. And Google-Extended isn't a crawler. Nothing ever arrives under that name. It's a switch for what Google may do with pages Googlebot has already fetched. Applebot-Extended is the same thing for Apple. That doesn't make them less useful, but it does make them different from the rest of this list.
Which crawler does what is in the table in the visibility basics, with each provider's documentation alongside.
Do crawlers follow your robots.txt?
The well-behaved ones do, others don't: robots.txt is a signal, not a wall. It's a request, and you have no way to enforce it.
That isn't a hunch. Cloudflare tested it and published the results. They set up brand-new test domains that hadn't been published anywhere, with a robots.txt forbidding all automated access. Then they asked Perplexity questions about those domains. Alongside the declared Perplexity-User, they found an undeclared crawler posing as ordinary Chrome on macOS, coming from IP addresses outside Perplexity's official ranges. Cloudflare put the scale at tens of thousands of domains and millions of requests a day.
Cloudflare itself describes robots.txt as expressing "website directives and preferences": preferences, not binding rules.
For some crawlers it can't work by definition. ChatGPT-User fetches a page because a user asked for it right then, and requests like that don't reliably fall under your robots rules.
What does that mean for your choice? Blocking only works with crawlers that choose to honor it, so it buys you less protection than the effort suggests. If you really want something kept private, put it behind a login, or don't put it online. robots.txt is the wrong tool for that.
Can you opt out of AI Overviews?
Not through robots.txt. It's an expensive misconception: work goes into it that gets you nothing.
AI Overviews are built on Google's ordinary search index, and Googlebot fills that index. To get out of AI Overviews, you'd have to refuse Googlebot, and then you'd disappear from the normal search results too. That's a trade nobody wants to make.
Blocking Google-Extended doesn't help either: that's about Gemini, not AI Overviews. Google says so itself: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."
The only real levers are nosnippet and max-snippet, and they hit your ordinary search result too: they shrink or remove the snippet under your link. You'd be trading visibility in the search results for slightly less text in an AI answer. For almost everyone, that's a bad trade.
Source: Google on its crawlers.
Why does text that stands on its own get used more?
Because AI tools cite passage by passage, not page by page. It's the other side of the same choice: if you leave the door open, make sure there's something worth taking.
A paragraph that makes sense on its own gets used. A paragraph where the answer only shows up in the fourth sentence doesn't. It's the same rule as in the basics: answer first, one claim per paragraph, with the evidence in the same paragraph.
You'll see this sold as extractability. The idea is right, and it isn't new. Be careful with the numbers attached to it, though. For that term I found only agencies, tool vendors and glossaries as sources: no provider documentation and no independent research. One figure making the rounds compares how often two writing styles get cited, with no measurement behind it that anyone can check.
That doesn't make the advice wrong. It makes the precision suspect. Do it because it makes sense and because it makes your text better for people too, not because there's a percentage attached.
What do you do with this?
Four things, in this order.
- Look at what's in your
robots.txtright now. It often holds something someone put there years ago that nobody understands anymore. - Pick one of the three positions and write down why. If you can't explain it, it isn't a choice. It's a setting.
- Put the answer first under every heading, so each passage makes sense on its own.
- Keep building ordinary search visibility. That's still where you get the most.
This article is part of the online visibility basics, where the whole foundation is laid out. If you'd rather not do it yourself, it falls under what I do for SEO and AI visibility.
Keep it in proportion, though. A good robots.txt makes sure you can be found. Whether people choose you after that depends on what you're offering.
Not sure what your robots.txt says?
Send me your website and what you're trying to achieve. I'll tell you whether your current setup fits that, and if it doesn't, what I'd change.
Dominique@framvekst.noYou'll hear back within three business days.
See my contact details