AI crawler policies: GPTBot, ClaudeBot, CCBot, and Google-Extended

AI crawler policies are robots.txt rules aimed at the user agents that fetch content for AI training, AI search, or a user’s live request. They are not a separate protocol: the same RFC 9309 matching applies. The difference is that a site may want to allow AI search citations while blocking model training, or the reverse.

This page is research material, not legal advice. It describes the published user agents and how to write rules; it does not decide what your site should allow.

The main user agents

Vendors change names and scopes. Check the operator’s current documentation before relying on a token. The table is a starting point, not a permanent contract.

Allow search, block training

A common policy is to allow search/answer crawlers and block training crawlers. In robots.txt:

User-agent: *
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

This is a policy example, not a recommendation. It blocks the named training crawlers and allows the named search/answer crawlers. A crawler that does not match any group falls back to *, which allows everything.

Allow everything

If the goal is maximum visibility, allow all:

User-agent: *
Allow: /

Sitemap: https://www.crawlerbros.com/sitemap.xml

This is the policy this site uses. The owner chose maximum visibility for search, answer, and training crawlers. The decision is disclosed here and in robots.txt.

Block everything AI

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Google-Extended
User-agent: CCBot
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /

Blocking OAI-SearchBot or Claude-SearchBot can reduce your visibility in those products’ search/answer features. Blocking training crawlers does not affect Googlebot’s ordinary search indexing, but Google-Extended is separate from Googlebot; read Google’s documentation before assuming otherwise.

What robots.txt cannot do

• It cannot stop a crawler that ignores it.
• It cannot prevent a page from being linked or quoted by a user.
• It cannot override a contract or a licence.
• It cannot make a blocked page invisible if other sites copy it.
• It cannot distinguish “training” from “search” for a crawler that uses one user agent for both.

If a vendor uses one user agent for multiple purposes, a robots.txt rule cannot separate them. Check the vendor’s documentation and, if the distinction matters, use a contractual or technical control.

Payload and server load

AI crawlers can request many pages. If you allow them, monitor:

• request volume by user agent;
• response time and error rate;
• cache hit rate;
• bandwidth.

You can rate-limit at the edge, but be careful not to block legitimate search crawlers by IP range. User-agent rules are not authentication; a malicious client can spoof them.

A policy template

1. Decide the goal: maximum citation, maximum control, or a split.
2. Write explicit groups for the user agents you care about.
3. Keep a * group as the default.
4. Declare your sitemap.
5. Test the rules with the robots.txt tester.
6. Document the decision and the date.
7. Re-check vendor documentation periodically; user agents and scopes change.

The robots.txt study shows how the top Tranco domains handle these user agents as of November 2026.

Sources

• OpenAI — crawler overview
• OpenAI — crawler overview
• Anthropic — crawler documentation
• Google — crawlers
• Common Crawl — CCBot