robots.txt generator for AI crawlers
Pick a policy, flip the crawlers you care about, and get a robots.txt block that says what you actually meant. Retrieval crawlers and training crawlers are handled separately, because blocking one is a content decision and blocking the other takes you out of AI answers.
Your robots.txt block
If you already have a robots.txt, append this block rather than replacing the file. Keep any existing Sitemap: line — this generator only writes one if you give it a URL.
What the block will do
Retrieval and training are not the same decision
A retrieval crawler fetches your page because somebody just asked a question and the model needs sources to answer it. Block it and your page is not a candidate — not ranked low, not outranked, simply absent from the set the model is allowed to look at. There is no way to recover from that inside the answer.
A training crawler collects text to build future models. Block it and your content stays out of a training corpus; nothing about tomorrow's answer changes. That is a licensing decision about your work, and it is a legitimate one — a lot of publishers make it. It is just not the same decision as the first one.
Most AI-visibility checkers collapse both into one switch labelled <em>AI crawlers</em>, then report a single red light. That is why their advice so often reads as “you are blocking AI” when the site is only blocking CCBot, which costs it nothing at all.
What we found in 48 sources
The two numbers that matter are 5 and 7. Five sources had cut themselves out of the answer pool by blocking a retrieval crawler. Seven had blocked something, which means two of them blocked a training crawler only — the careful version, the one that costs nothing. Reporting a single “AI crawlers blocked” figure would have made those two look like the same mistake as the five.
Five sources could not be judged at all: no robots.txt, a robots.txt that would not parse, or a fetch that never completed. We count those as unknown rather than open, because “we could not tell” and “we checked and it is fine” are different findings.
The mistake that looks careful
One site in our sample, citegrade.com, blocks CCBot and allows GPTBot and OAI-SearchBot. Read that as a policy and it is coherent: no free training corpus, full participation in retrieval. Read it as a checkbox and it looks like a site that is half-blocking AI, which is how a single-switch checker will report it.
The version of this mistake that actually costs citations is the reverse: a site reads “block AI crawlers” somewhere, adds <code>User-agent: *</code> with <code>Disallow: /</code> under an AI heading, and takes out OAI-SearchBot along with CCBot. Nothing in the file looks wrong. The site simply stops appearing, and nobody can tell from the outside why.
# Training crawlers — blocking these costs you no citations User-agent: CCBot Disallow: / # Retrieval crawlers — these decide whether you enter the answer pool User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: /
If you are not sure, read the logs
Any list of user-agents, including the one on this page, is a snapshot. The reliable test is in your server log: if a crawler appears within seconds of a real user question and fetches one page, it is retrieval. If it appears on a schedule and walks the whole site when nobody is asking anything, it is collection.
That test also tells you which entries on a list matter to you. A crawler that has never visited your site is a line in someone else's blog post, not a decision you have to make today.
Questions about AI crawlers and robots.txt
Which AI crawlers should I allow?
The retrieval ones: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot and Claude-User. Those are the fetches that put your page in front of a model at the moment somebody asks a question. Blocking one of them removes you from consideration before the ranking question even starts.
Is blocking CCBot a mistake?
No. CCBot collects for Common Crawl, a public dataset used to train models — it is not what decides whether you appear in an answer tomorrow. Blocking it is a legitimate content decision. Blocking OAI-SearchBot is a different decision with a different consequence, and conflating the two is the single most common mistake in this category.
Will allowing AI crawlers get me cited?
It is necessary, not sufficient. In our 48-source experiment, pages that allowed retrieval crawlers were cited 31.9% of the time against 7.1% for pages that blocked them. That is the largest gap we measured with a physical mechanism behind it — a crawler that cannot fetch you cannot quote you — but 31.9% is still not a promise, and the sample is 48 pages.
Should I block GPTBot?
That depends on whether you want your text in OpenAI's training data, and it has nothing to do with whether you get cited. Our recommendation is to decide it as a content-licensing question, not an SEO one. If you block GPTBot, do not also block OAI-SearchBot — they are separate user-agents on purpose.
I already have a robots.txt. Do I replace it or append?
Append. Replacing the file deletes your sitemap line and any rules for other crawlers, and a missing Sitemap directive is a real loss. Paste the AI-crawler block at the end, keep everything above it, and check that the file still starts with your existing rules.
The other tools
llms.txt generator
Write llms.txt and llms-full.txt, checked against the four rules the format has.
FREE Open →JSON-LD generator
Organization, WebSite, Article, FAQPage and BreadcrumbList, validated as you type.
FREE Open →AI crawler reference
Every user-agent that decides whether your page enters an AI answer, and what blocking it costs.
FREE Open →Fix generator — all four files at once
If you want llms.txt, robots.txt and JSON-LD in one pass instead of one at a time.
FREE Open →AI readability check
Eight checks on a pasted page, including whether the two files above are even reachable.
FREE Open →