AI crawlers: the full list, and which ones decide citations
Thirteen user-agents that matter, what each one is for, and what blocking it actually costs you. Six of them can remove you from AI answers. The other seven cannot — and the difference is the whole point.
What your file says
Retrieval crawlers — these six decide whether you appear
Training crawlers — these seven cost you nothing
Why the two lists are separate
The retrieval column is mechanical. A crawler that cannot fetch your page cannot put it in front of a model, and no amount of good writing changes that. In our 48-source experiment, pages that allowed retrieval crawlers were cited 31.9% of the time against 7.1% for pages that blocked them. That is the largest gap we measured with a physical mechanism behind it.
The training column is a policy. Blocking GPTBot does not make you less citable tomorrow; it keeps your text out of a corpus that trains a future model. Reasonable publishers land on both sides of that. What is not reasonable is treating the two columns as one switch, which is what most tools in this category do.
What 48 sources actually did
Two of the seven that blocked something blocked only a training crawler. Read as one number — “14.6% of sources block AI” — those two look identical to the five that cut themselves out of the answer pool. They are not identical. One group made a licensing decision and kept its citations; the other lost them.
The test that beats any list
Every list, this one included, is a snapshot of names that existed on the day it was written. New crawlers appear and old ones get renamed. The durable test is the shape of the request in your log: a fetch that arrives seconds after a real user question, and asks for one page, is retrieval. A fetch that arrives on a schedule, walks the whole site, and happens while nobody is asking anything is collection.
That is also why we do not publish a definitive list with a green tick next to each row. The row you need is the one that shows up in your traffic, and the only place that is written down is your server.
Questions about AI crawlers
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects text for training. OAI-SearchBot fetches pages to index for ChatGPT search. Blocking the first keeps your content out of a training corpus; blocking the second keeps you out of answers. They are separate user-agents precisely so that publishers can make those two decisions separately — and most checkers throw them into one bucket anyway.
Does blocking AI crawlers hurt my Google rankings?
Not through Googlebot, which is a separate user-agent and is not on this list. Google-Extended is the one that matters here, and it controls Gemini training only — Google's own documentation says it does not affect Search. Blocking Googlebot itself is a different thing entirely and would remove you from Search.
What is CCBot?
Common Crawl's crawler. Common Crawl is a public dataset of web pages that many organisations, including model trainers, download. Blocking CCBot is the standard way to keep your text out of that dataset, and it does not remove you from AI answers.
How do I find out which AI crawlers actually visit my site?
Your server log. Search it for the user-agent strings on this page. The timestamp tells you which kind it is: a fetch seconds after a real question is retrieval, a scheduled sweep when nobody is asking anything is collection. No published list can tell you what your own traffic looks like.
Can I just copy a block into robots.txt?
Yes — the <a href="../tools/robots-txt-generator.html">robots.txt generator</a> writes one with retrieval and training kept apart, and you can flip individual crawlers before copying. Append it rather than replacing your existing file, and keep your Sitemap line.
The other tools
llms.txt generator
Write llms.txt and llms-full.txt, checked against the four rules the format has.
FREE Open →robots.txt generator for AI crawlers
Choose a policy and get the AI-crawler block, with retrieval and training kept apart.
FREE Open →JSON-LD generator
Organization, WebSite, Article, FAQPage and BreadcrumbList, validated as you type.
FREE Open →Fix generator — all four files at once
If you want llms.txt, robots.txt and JSON-LD in one pass instead of one at a time.
FREE Open →AI readability check
Eight checks on a pasted page, including whether the two files above are even reachable.
FREE Open →