On October 30, 2026, Google announced plans to support a shared AI training control in robots.txt: one rule that tells multiple AI providers at once not to train on your content (Google Search Central Blog). OpenAI published its own post the same day, "Making it easier for websites to express AI training preferences." But Google's support is planned for early 2027, so what works today is still the per-company rules. This guide explains the new signal and what to put in robots.txt right now, using only official documentation.
The short answer: write per-company rules now, add the shared signal in 2027
To stop AI training today, disallow GPTBot (OpenAI), Google-Extended (Google), and ClaudeBot (Anthropic) individually. Google's crawler documentation says it intends to support the shared signal, written as user-agent: __usage_ai-training, in early 2027 (Google's common crawlers). Until then, assume nobody reads it.
The other key point: training crawlers and search crawlers are different bots. If you block only the training ones, you can still appear in ChatGPT and Claude search. Block every AI bot without thinking and you lose AI search traffic too.
What was announced on October 30
Until now, a site that wanted to opt out of AI training had to look up each provider's crawler name and add rules one by one. Under the new approach, a shared "policy token" in robots.txt sets your training preference for every participating AI provider at once. It builds on work by the IETF AIPREF working group, and Google's post links to the draft "draft-kelley-aipref-training-preference-signal."
Item | What the official sources say |
|---|---|
Syntax | Uses |
Google timeline | Early 2027 (crawler list); "aim to support this new control along with other AI providers in 2027" (blog) |
How Google treats it | If a site opts out via the token and Google-Extended is not explicitly specified, Google will treat it as a disallow of Google-Extended |
Existing controls | Company-specific controls such as Google-Extended keep working alongside the shared signal |
Status | A draft proposal, to be standardized through the IETF. Google says it is consistent with the EU Code of Practice for General Purpose AI model providers, which recognizes robots.txt as a primary way to express preferences |
Note that this covers training only. Google says the mechanism brings in the IETF's AI training definition while discussion continues on the rest of the AIPREF vocabulary. It is not a switch that also stops AI search from citing or summarizing your pages.
What works today: major AI crawlers and their roles
robots.txt works per published crawler name (User-agent). Even within one company, training, search, and user-triggered fetching are separate, so check the role before you block.
Provider | User-agent | Role | If you block it |
|---|---|---|---|
OpenAI | GPTBot | Training | Signals your content should not be used to train generative AI foundation models |
OpenAI | OAI-SearchBot | ChatGPT search | Not shown in ChatGPT search answers (may still appear as navigational links) |
OpenAI | ChatGPT-User | User-initiated fetches | Not automatic crawling; OpenAI says robots.txt rules may not apply |
Google-Extended | Gemini training and grounding | Not used for Gemini training or for grounding in Gemini Apps and Vertex AI. No effect on Google Search inclusion or ranking | |
Anthropic | ClaudeBot | Training | Future materials excluded from training datasets |
Anthropic | Claude-SearchBot | Claude search | Not indexed for search; may reduce visibility in search results |
Anthropic | Claude-User | User-initiated fetches | Claude cannot retrieve your pages in response to user questions |
Sources: Overview of OpenAI Crawlers / Google's common crawlers / Anthropic's crawlers
robots.txt by goal
Pattern 1: Block training, stay in AI search (most business sites)
For a company or service site, being recommended in AI search is itself a source of leads. Block only the three training crawlers and leave search crawlers alone.
User-agent: GPTBotDisallow: /User-agent: Google-ExtendedDisallow: /User-agent: ClaudeBotDisallow: /
One judgment call on Google-Extended: Google says "websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps." If you want referral traffic from Gemini Apps, you may choose to leave Google-Extended allowed.
Pattern 2: Keep AI out as much as possible (paywalled or proprietary content)
On top of Pattern 1, block the search crawlers (OAI-SearchBot, Claude-SearchBot) and Claude-User. You will lose traffic from ChatGPT and Claude search. Because OpenAI says robots.txt rules may not apply to ChatGPT-User, robots.txt alone cannot fully stop AI access.
Pattern 3: Add the shared signal in 2027
Following Google's note, the shared signal looks like this. It is still a draft, so check the syntax again when support goes live. Keep the per-company rules in place so your intent is clear before and after the switch.
User-agent: __usage_ai-trainingDisallow: /
Google's post says website owners will be able "to opt out across all providers, or to pick and choose." The exact precedence rules are best confirmed once the draft is finalized.
Five things to know before you change robots.txt
- It does not undo the past. robots.txt states a preference for future crawling. It is not a deletion mechanism.
- Each subdomain needs its own file. Anthropic asks you to add the rule for every subdomain you want to opt out.
- IP blocking is not recommended. Anthropic says blocking IP addresses may not work correctly. Use robots.txt first.
- Changes take time. OpenAI says search results can take about 24 hours to adjust.
- Google Search rankings do not change. Google-Extended does not affect Search inclusion and is not a ranking signal.
Our take: for most small business sites, block training only
In our website work we hear both "I don't want AI using our content" and "I want AI search to recommend us." If the site exists to be found (company profile, services, hiring), block training crawlers and keep search crawlers. That loses the least. Sites whose product is the content itself, such as paid articles or proprietary data, have a real reason to block search crawlers too.
Many visitors from AI assistants land on our free tools. If you want to see how they work, try the free online clock and timer.
We build websites and help with SEO and AI search settings after launch. If you'd like a second opinion on how your robots.txt should look, feel free to reach out.
Summary
- On October 30, 2026, Google announced plans to support a shared robots.txt signal (
__usage_ai-training) for opting out of AI training across providers, targeted for early 2027. - What works today is per-company rules. Training crawlers: GPTBot, Google-Extended, ClaudeBot.
- Blocking search crawlers (OAI-SearchBot, Claude-SearchBot) removes you from AI search. Treat them separately from training.
- Blocking Google-Extended does not affect Google Search, but your content will not be used for grounding in Gemini Apps.
- robots.txt covers future crawling only and must be set per subdomain.
FAQ
Will blocking AI training in robots.txt hurt my Google rankings?
No. Google's documentation states that Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. Blocking it does mean your content will not be used for grounding in Gemini Apps.
Does writing __usage_ai-training in robots.txt do anything today?
Google says it intends to support the shared signal in early 2027, so assume it is not read yet. It does no harm to add it now, but keep the per-company rules (GPTBot, Google-Extended, ClaudeBot) alongside it. Google says company-specific controls will keep working after the shared signal launches.
How do I block AI training but stay visible in ChatGPT and Claude search?
Disallow only the training crawlers (GPTBot, ClaudeBot) and leave the search crawlers (OAI-SearchBot, Claude-SearchBot) allowed. OpenAI says each setting is independent, and sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers.
Does robots.txt remove content that was already collected?
No provider says it does. robots.txt expresses your preference for future crawling. Anthropic, for example, says blocking ClaudeBot signals that the site's future materials should be excluded from training datasets.
How long does a robots.txt change take to apply?
OpenAI says search results can take about 24 hours to adjust after a robots.txt update. Other providers do not publish a timeline. Remember that each subdomain needs its own robots.txt.