AI Crawlers and robots.txt: Which Bots to Allow, and Why Your Host Can Still Block Them
Written by Usama Afzal, founder and director · Published
Allowing AI crawlers takes one decision and a few lines in robots.txt, but it does not guarantee they can read your pages. The robots.txt file is a request. Your host, CDN or firewall decides whether the request is ever served. This guide covers which bots exist, what each is for, how to choose a policy, and how to test access properly. Vendor documents were read on 11 November 2026.
Which AI crawlers exist, and what is each one for?
Each AI company publishes more than one bot, and they do different jobs. Treating them as one thing is the most common mistake in robots.txt files. The table groups them by purpose, using each vendor's own wording.
| Operator | robots.txt token | Stated purpose | Does robots.txt control it? |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Surfaces websites in ChatGPT search features | Yes. Sites that opt out will not appear in ChatGPT search answers, though they can still appear as navigational links |
| OpenAI | GPTBot | Crawls content that may be used to train generative AI foundation models | Yes. Disallowing it signals the content should not be used for training |
| OpenAI | ChatGPT-User | Visits a page when a user asks ChatGPT a question or a custom GPT acts | Not reliably. OpenAI says robots.txt rules may not apply because a user triggered the visit |
| Anthropic | ClaudeBot | Collects web content that may be used to train models | Yes. Anthropic also supports the non-standard Crawl-delay line |
| Anthropic | Claude-SearchBot | Crawls content to improve the quality of search results | Yes. Blocking may reduce visibility in those results |
| Anthropic | Claude-User | Fetches pages when a Claude user's question needs them | Anthropic says its bots honour do-not-crawl rules |
| Perplexity | PerplexityBot | Finds and links sites in Perplexity search results; not used for training | Yes. Perplexity says to allow it if you want to appear |
| Perplexity | Perplexity-User | Visits a page during a user's question and links to it | Generally ignores robots.txt, because a user initiated the fetch |
| Google-Extended | A token, not a crawler: controls use of crawled content for Gemini training and grounding | Yes. Google says it does not affect inclusion in Google Search |
Two patterns matter. First, every operator separates "collect data for training" from "help a person find or read my page". Second, the bots that act on a live user request (ChatGPT-User, Perplexity-User) are designed to behave like a person with a browser, so robots.txt is the wrong tool for them. If you do not want a page seen, it must not be public.
Should you block the training bots?
That is a business decision, not a technical one, and the evidence supports three sensible policies. None of them is "block everything", which removes you from the places where buyers now ask questions.
| Policy | What you allow | What you block | Fits when |
|---|---|---|---|
| Open | Everything above | Only private paths such as admin, cart and checkout | You sell services or products and want to be found and cited. This is the default we would suggest for most small businesses |
| Search yes, training no | OAI-SearchBot, Claude-SearchBot, Claude-User, ChatGPT-User, PerplexityBot, Perplexity-User, Googlebot, Bingbot | GPTBot, ClaudeBot, Google-Extended, plus others you choose | Your content is the product (publishers, course sellers, paid research) and you want visibility without feeding training sets |
| Closed | Googlebot and Bingbot only | All AI bots | Legal or contractual limits apply. Accept that you will not appear in those assistants' answers |
Two vendor statements keep the middle policy honest. Google says Google-Extended "does not impact a site's inclusion in Google Search", so blocking it does not cost you rankings. OpenAI says a site that opts out of OAI-SearchBot will not appear in ChatGPT search answers, so blocking the search bot does cost visibility. Blocking a training bot while allowing the search bot from the same company is therefore a coherent choice. Blocking everything by habit is not.
Nothing here promises citations. Allowing a crawler makes you eligible to be read, not chosen. Our guide to GEO, SEO and AEO covers what influences selection, and how pages get cited covers the page-level side.
What does a clean robots.txt look like for each policy?
For the open policy you need very little. A single group for all crawlers, the private paths, and your sitemap:
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Sitemap: https://www.example.com/sitemap.xml
For "search yes, training no", add one group per training token. Everything not named falls back to the group above:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
There is one trap worth knowing. A crawler obeys the most specific group that names it and ignores the rest, so groups are not combined. If you list a bot by name and give it its own group, a later change you make to the * group will not apply to it. When we reviewed our own robots.txt, we found 18 named bot groups that were identical to the * group. They do no harm today, but every future edit would need repeating 19 times. List a bot separately only when its rules differ.
The robots.txt generator groups these bots by purpose and builds the file in your browser, if you would rather not write it by hand. Anthropic also states that, as of 30 October 2026, it recognises a machine-readable "no AI training" signal based on an IETF draft, and that a robots.txt rule naming ClaudeBot takes priority over that general setting.
Why can a crawler still fail when robots.txt says allow?
Because robots.txt is read by a polite crawler after it has already reached your server. Before that, your hosting layer decides whether to serve it at all. The usual causes:
- Bot-protection or "under attack" modes that show a browser check to anything that does not look like a person.
- Blocked-bot lists in a CDN or firewall that include AI user agents, by default or through a past setting someone forgot.
- Country rules. Blocking a country can block the crawler addresses that originate there.
- Rate limits that return 429 to a bot fetching several pages in a row.
- Security plugins on WordPress that block unfamiliar user agents.
Hostinger's own help article says its CDN security settings can block crawlers even when robots.txt allows them, and names Under Attack mode and blocked bots as examples. Other CDNs and hosts have similar switches. The symptom is nasty because everything you can see looks fine: your page loads in a browser, and your robots.txt is correct.
How do you test whether AI crawlers can actually reach your pages?
Work from cheap checks to better evidence. Each step answers a different question.
- Confirm robots.txt itself loads. Open
yoursite.com/robots.txtand check it returns a normal page, not an error or a challenge screen. A crawler that cannot fetch robots.txt cannot read your rules. - Request a page with a crawler's user agent. From a terminal:
curl -I -A "OAI-SearchBot" https://www.example.com/. A 200 means no rule blocks that user-agent string from your own address. A 403 or 429 is a flag worth chasing. This tests the user-agent rule only, not the crawler's real address. Hosts that check addresses will correctly refuse an imitation, so a 403 here for Googlebot or Bingbot is expected and proves nothing. Treat it as a hint, not a verdict. - Look at your access logs for the real bots. Filter for the tokens in the table and record the status codes. A real crawler comes from a published address list: OpenAI publishes separate lists for OAI-SearchBot, GPTBot and ChatGPT-User, Anthropic publishes one at claude.com/crawling/bots.json, and Perplexity publishes lists for PerplexityBot and Perplexity-User. Anything using the name from another address is an imitation.
- Use the search engines' own tests for Google and Bing. In Search Console, use URL Inspection and run a live test on your home page, a service page and robots.txt. Bing Webmaster Tools has an equivalent. This is the only way to see what the real Googlebot and Bingbot receive.
- Check your host's security settings. Look for bot-blocking lists, country rules, rate limits and any attack mode, and note when each was last changed.
- Allow time. OpenAI and Perplexity both say settings changes can take up to 24 hours to take effect.
A small log filter on a standard Apache or Nginx access log, run on the server or on a downloaded copy, gives a count by status code for those bots:
grep -Ei "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User" access.log | awk '{print $9}' | sort | uniq -c
If this prints nothing, either those crawlers have not visited, or your host does not keep logs for that period. Both are useful facts. If it prints a column of 403 or 429, you have found a block that robots.txt will never fix.
What can robots.txt not do?
- Secure anything. It is public and advisory. Use passwords or authentication for private pages.
- Remove a page from search. A blocked URL can still appear if other sites link to it. Use
noindexon pages you want out of results. - Control user-triggered fetches. ChatGPT-User and Perplexity-User may ignore it by design.
- Undo past collection. It affects future crawling, not content already gathered.
Google adds a separate point for its AI features: its guidance says a page needs to be indexed and eligible to show a snippet, and that there are no additional requirements to appear in AI Overviews or AI Mode. Preview controls such as nosnippet and max-snippet are what limit use there, not robots.txt for AI tokens.
A 15-minute checklist
- Decide the policy: open, search yes and training no, or closed.
- Write one
*group, add named groups only where rules differ, and add your sitemap line. - Load robots.txt in a browser and check it returns a plain text file.
- Run the user-agent request for OAI-SearchBot, Claude-SearchBot and PerplexityBot and note the status codes.
- Run the live test in Search Console for three URLs.
- Open your host's security panel and note any attack mode, bot list or country rule.
- Put a calendar reminder to run the log filter next month and compare.
If you want an expert to run this across your site, the technical SEO audit includes an AI crawler access check, and our GEO and AI search visibility work covers what to do once access is confirmed. Our AI search readiness checklist is a free self-check, and the llms.txt article explains why that file is optional.
Not sure whether AI crawlers can reach your site?
Send us the address. We check robots.txt, crawler access and the pages themselves, then tell you what is worth fixing first.
Ask for a crawler access check →Frequently asked questions
Should I allow GPTBot?
GPTBot is OpenAI's training crawler. Allowing it is a content-use decision, not a visibility one. If you want to appear in ChatGPT search answers, the token that matters is OAI-SearchBot, which you can allow independently.
If I block GPTBot, will ChatGPT stop mentioning my business?
Not on that basis alone. OpenAI documents GPTBot as the training crawler and OAI-SearchBot as the one that controls ChatGPT search appearance. Mentions can also come from other sources the model has seen.
Does blocking Google-Extended hurt my Google rankings?
Google says it does not impact inclusion in Google Search and is not a ranking signal. It limits use of content for Gemini training and grounding.
Do all AI bots obey robots.txt?
The documented search and training bots from OpenAI, Anthropic and Perplexity say they do. The user-triggered ones, such as ChatGPT-User and Perplexity-User, may not. Unknown scrapers may ignore it entirely, which is why robots.txt is not security.
How do I know if a request really came from OpenAI or Anthropic?
Compare the request's IP address with the vendor's published list. The user-agent string alone can be copied by anyone.
How long until a change takes effect?
OpenAI and Perplexity both say up to about 24 hours. Allow a few days before deciding that a change did nothing.
Sources
Read on 11 November 2026. Vendor documentation changes, so check the current page before acting on a detail.
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity: Perplexity crawlers
- Google Search Central: Google's common crawlers
- Google Search Central: AI features and your website
- Hostinger: How to allow or block AI crawlers on your Hostinger website
How we work: refund policy · terms of service · about Website Prism
Keep reading