robots.txt allow ai crawler gptbot
This is one of the five queries behind this directory. We sent it to a web search, kept every source the results returned, then asked a model the same question 5 times and recorded which sources it actually cited. 4 of the 5 sources were cited at least once.
What we measured
Measured 2026-10-16 · gpt-oss-20b via text.pollinations.ai (alias "openai") · read the page body · 5 of 6 runs usable for this query
csdn.net
2026年指南：robots.txt如何放行AI爬虫抓取
blog.csdn.net
AI crawler robots.txt 2026 gptbot claudebot perplexity
baijiahao.baidu.com
网站GEO优化实战基础：robots.txt 的 AI 爬虫友好配置
robotstxtlab.net
AI Crawler robots.txt Generator & Checker | Robots.txt Lab
blog.csdn.net
robots.txt 精准屏蔽 GPT 及其他 AI 爬虫（附实战指南）
Titles and domains come from the pages themselves, so you can find them. The citation count, the rank, the word count and the retrieval access are our measurement, and the method is published in full.
What this query shows
The most-cited source in this pool was csdn.net — named in 5 of the 5 runs we could measure.
The lowest-ranked source that still got cited was robotstxtlab.net (#5 on entry, cited 1 of 5 times) — search position and citation are not the same thing.
1 of the 5 sources in this pool was never cited in any run. A zero here is a result we publish, not a result we hide.
1 source was cited in every single run: csdn.net.
1 source here turns away the retrieval crawlers outright — OAI-SearchBot, ChatGPT-User or PerplexityBot. That is a closed door, not a content problem, and no amount of writing fixes it.
Across the 5 runs, the set of cited sources overlapped by 0.700 on average (range 0.250–1.000). That is partly settled — there is a stable core, and room around it.
Where this query sits
Of the 5 queries we measured, this one is the most settled — the same pages keep winning. It sits last on our list of the queries that are still open.
The other four queries
Questions this query answers
Does blocking GPTBot stop ChatGPT from citing my page?
Those are two different crawlers. GPTBot is OpenAI's training crawler; OAI-SearchBot is the one that fetches pages for search. Blocking GPTBot opts you out of training, and blocking OAI-SearchBot takes you out of the pool that ChatGPT search draws on. Treating them as one switch is the most common mistake we see.
How many pages in your pool block AI crawlers?
7 of 48. We record this per page rather than estimating it, and the blocked ones stay in the pool: being blocked is a finding, not a reason to drop a source.
Should I just allow every bot?
That is a business decision, not an SEO one. What we can say is that allowing everything and blocking everything are both cruder than deciding per crawler, because the two OpenAI crawlers have different jobs.