Free robots.txt generator with AI bot controls
Choose what to block, add your sitemap and decide how to treat AI crawlers that search, fetch pages for users or collect training data. Copy the finished robots.txt into the root of your site.
Free. No sign-up. Runs in your browser: what you type is not sent to us.
Typical: admin area, cart, checkout, internal search. Leave empty to allow everything.
Bot names are taken from each vendor's published documentation and can change. robots.txt is a request, not a lock.
What robots.txt can and cannot do
It asks well-behaved crawlers not to fetch certain paths. It does not remove a page from search results and it is not a security measure: a blocked URL can still be listed if other sites link to it. To keep a page out of results, use a noindex tag or password protection.
AI crawlers have different jobs
Several AI companies run more than one crawler. Some build a search index, some fetch a page when a user asks a question and some collect data for training. You can allow the search and user-triggered ones and block the training one, or the reverse. This tool groups them by purpose so the choice is clearer. Each vendor documents its own bot names; check their pages if a name changes.
Never block the crawlers you rely on
A blanket disallow for all user agents takes your whole site out of search. If you are unsure, start with the safe default of allowing everything and only block specific admin, cart or internal search paths.
Test after you publish
Open yourdomain.com/robots.txt, then use Search Console's tools to confirm important pages are fetchable.
Questions people ask
Will blocking an AI bot stop my content being used?
It asks that vendor's crawler not to fetch your pages. Compliance is voluntary, and it does not affect content already collected or content shown by other sources.
Should I block GPTBot or ClaudeBot?
That is a business decision. Blocking training crawlers is a common choice. Blocking search crawlers may reduce the chance your pages are cited in those products' answers.
Is the file generated on your server?
No. It is built in your browser and nothing is sent to us.
Want this done for you?
Tools show you the gaps. If you would rather have someone fix them, tell us what you need and we reply with questions or a written quotation. The closest service is Technical SEO audit.