One Token, Four AI Labs: robots.txt Gets a Shared AI-Training Switch

Diagram titled One Token, Four AI Labs: a robots.txt snippet with User-Agent __usage_ai-training, Disallow / and Allow /press-kit/ on the left, and on the right four provider rows — Anthropic in progress, OpenAI in the coming months, Microsoft from 1 February 2027, Google in 2027 — with a bottom panel noting that a User-Agent wildcard group must not be read as matching the policy token and that a file with no policy-token line expresses no preference — Original graphic by Guruji Corporation. Timelines as stated by each provider on 30 October 2026 and in IETF draft-kelley-aipref-training-preference-signal-00.

For twenty years a website said "no" to a crawler by naming it. On 30 October 2026 the four largest Western model developers — OpenAI, Google, Microsoft and Anthropic — said they will read one shared line instead. A single User-Agent group now speaks for all four, and two of them have published the date they start obeying it.

The mechanism is small. The part worth your time is what it does not cover, because most Indian business websites are about to assume they are opted out when the file they have says nothing at all.

What actually happened

The mechanism adds no new syntax — deliberately. The vehicle is draft-kelley-aipref-training-preference-signal-00, "Machine-Readable Training Preference Signal", submitted on 30 October 2026 by Kevin Kelley of Anthropic and announced on the IETF's ai-control mailing list the same day. Its abstract is plain about it: the draft "introduces no new directive names and no new grammar". The group structure and longest-match precedence of the Robots Exclusion Protocol, RFC 9309, are untouched. What is new is a policy token — a product token in the User-Agent line that names a purpose rather than a crawler:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Read it as robots.txt has always been read: a preference against AI training everywhere except /press-kit/, where Allow expresses a preference in favour. The leading double underscore exists so the token cannot collide with a real crawler name.

Three rules decide whether your existing file says anything. First, wildcard groups do not count: the draft says a User-Agent: * group "MUST NOT be interpreted as matching the policy token". A site that blocks every crawler with a wildcard has expressed no training preference whatsoever. Second, if no group contains the __usage_ai-training line, "no preference has been expressed" — silence is not consent, but it is also not a refusal. Third, and most easily misread, "this policy token is not a directive for crawling behavior": a provider may still fetch a page it will not train on. Blocking the fetch still requires naming that crawler's own product token, as it always did.

Four labs, four dates. OpenAI said it will honour the common signal and "aim to implement the signal in the coming months", with its existing crawler controls continuing to apply. Google published its plan on Search Central: it aims to support the control "along with other AI providers in 2027", and where a site opts out via the token without naming Google-Extended, Google plans to read that as if the publisher had disallowed Google-Extended. Microsoft was the most specific: from 1 February 2027 it honours AI Training preferences as it crawls, at site, path and page level. And Anthropic is furthest along — Kelley's message states plainly that "Anthropic has already started implementing support for this draft", alongside its existing crawler controls.

Microsoft went further than the shared token. It splits the question in two. Training is the shared robots.txt preference, extended down to the page with <meta name="Bingbot" content="ai-training:disallow"> or X-Robots-Tag: Bingbot: ai-training:disallow. Grounding — using your page as input to a generated answer — is a separate, page-level preference Microsoft defines itself, with three values: Allow, DisallowExtended (Microsoft-operated experiences only, excluding third-party grounding services) and Disallow. Both default to allow when unset. Microsoft states that neither choice "guarantees inclusion, traffic, or payment" and that eligibility for ordinary Bing search results is unchanged. Its NoArchive and NoCache tags are retired for these purposes, with a temporary Bing Webmaster Tools bridge opening 15 December 2026 and the migration deadline on 1 March 2027.

What "AI training" means is now defined, not inferred. The token borrows its meaning from draft-ietf-aipref-vocab-08: "using an asset to modify the learned parameters of a generative AI model", excluding models used only for non-generative tasks. That document still notes its contents do not reflect working-group consensus in whole or part.

What is not settled. The instrument is an individual Internet-Draft — the IETF marks it "not endorsed by the IETF" with "no formal standing", and it expires on 3 May 2027. It is a preference, not a licence and not an enforcement mechanism: the vocabulary draft says outright that "preferences are not a security mechanism", and private agreements may override your robots.txt.

Why this matters for Indian SMBs and mid-market teams

Your robots.txt probably says nothing, and you think it says no. Thousands of Indian sites — brochure sites, D2C catalogues, clinic and institute pages — carry User-Agent: * with Disallow: /admin/ and believe they have opted out of AI training. Under this mechanism that file expresses no training preference at all. Writing the four-line group is a ten-minute change with a checkable outcome; assuming you are covered is free and wrong.

One rule replaces a list that keeps growing. The old routine was to enumerate GPTBot, Google-Extended, ClaudeBot, bingbot and whatever launched last quarter. A single policy-token group is the first version of this problem that survives a vendor reshuffle. Keep the vendor-specific rules too: every provider has said existing controls keep working, and until each implementation ships, the named crawler is still the only rule that binds.

Training and grounding are different questions, and only one of them affects your leads. Opting out of training is about future model weights. Grounding is about whether your product page becomes the citation in someone's AI answer today — closer to a marketing decision than a legal one, and Microsoft is now the only provider offering it as a separate switch.

Put three dates in a calendar now. Microsoft starts honouring the preferences on 1 February 2027, the grounding transition for sites still using NoArchive/NoCache closes on 1 March 2027, and Google lands in 2027 with OpenAI somewhere in the coming months. Anything you publish is only read on the next crawl, so a change made in February is not retroactive. For agencies, auditing client robots.txt before that date is short, defensible, billable work.

A preference is not a wall. If your concern is a competitor's crawler hoovering your catalogue, robots.txt is still a request, and this draft does not change that. Rate limits, WAF rules, authenticated content and legal terms remain the controls that hold.

Where Guruji Corporation fits in

We do not write internet standards and we do not speak for any of these vendors. What we do is the unglamorous part that follows an announcement like this: read the primary documents instead of the coverage, work out what applies to a specific Indian organisation's site, make the change in version control with a rollback, and write it down so the next audit starts from a record.

If nobody on your team can say whether your content is opted out of training, that is a short piece of work with a written handover — not a strategy project. Start with a short project brief.

Sources

One Token, Four AI Labs: robots.txt Gets a Shared AI-Training Switch

What actually happened

Why this matters for Indian SMBs and mid-market teams

Where Guruji Corporation fits in

Want to see how we can build your project?