AI training opt-out token: four backers, little enforcement

By Elena Vasquez

Four AI giants back a robots.txt opt-out for AI training, but the proposed standard has no enforcement teeth.

The new “policy token” lets publishers tell participating AI companies whether their content may train models.

Publishers have tried blocking bots, litigation and licensing deals. A shared signal reduces clutter, but not the compliance problem.

What does the policy token actually do?

Publishers would add instructions to robots.txt, the file websites use to tell web crawlers what they may access. The token identifies an AI-training crawler category and gives participating companies a broad permission instruction.

• The line User-Agent: __usage_ai-training names the AI-training crawler category.
• A disallow or allow instruction sets the preference for participating AI and technology companies.
• An allow line can include exceptions, such as permission to scrape a press kit.

Publishers normally list every crawler separately and grant or withhold access. The token would replace that repetitive work with one preference for participating companies.

Takeaway: Publishers get a convenient switch, not a promise that AI companies will obey it.

Who is backing the draft, and when?

Kevin Kelley, Anthropic’s technical lead for standards, wrote the proposal and submitted it to the Internet Engineering Task Force on Oct. 30. It remains a draft.

The IETF launched an AI Preferences, or AIPREF, effort to develop standards for website owners’ content-use preferences. The four companies participate.

Alex Springer, technical lead at publisher coalition SPUR, said backing an unreviewed draft contradicts the IETF’s public-consensus process:

“The way that the platforms have approached this by announcing their unilateral support of a ‘standard’ that is an unreviewed draft is the antithesis of that.”

Springer said the companies appear to be forcing a de facto standard that falls short of content owners’ requirements.

• Anthropic: Says it is rolling out adoption now.
• OpenAI: Plans to honor the token in coming months, without replacing its existing crawler controls.
• Google: Aims to support it in 2027. The approach builds on Google-Extended, which lets publishers block AI-training access through robots.txt; Google introduced that crawler control in 2023.
• Microsoft: Will comply and add an AI-grounding opt-out. From Feb. 1, 2027, the new preferences replace NoArchive for training and NoCache for grounding.
• Meta: Has not committed to honor it. Spokesman Andy Stone said Meta supports publisher preferences, wants a common signal and will work with other AI companies on implementation.

Support from four major firms makes this unusual among proposed web standards, but promises are not compliance.

Takeaway: The rollout has momentum on paper, while the IETF review still has the final say.

Will it stop AI scraping?

Robots.txt is advice, not an enforceable rule. A crawler can ignore it, and the protocol cannot force obedience.

AI visibility firm ScalePost tracked more than 50 publisher properties. Crawlers from the four backing companies made roughly 1 billion requests over 30 days, while just 0.15% requested robots.txt files.

The figures suggest crawler compliance is unsettled. AI companies also face copyright lawsuits, regulatory pressure and proposed crawler legislation, giving them reasons to project publisher-friendly behavior.

Raptive chief strategy officer Paul Bannister called the token “window dressing” and said: “I think this is useless and is marketing, and far, far, far too late… It does nothing and doesn’t solve the problem at all.”

Publishers can still combine the token with CDN-level crawler blocks or tools such as TollBit. Bannister compared it to locking a front door against some thieves, not every burglar. He called the token one arrow in a quiver of about 1,000 measures publishers need.

Takeaway: The token adds a signal, not a lock on the supply of content.

What does blocking cost in visibility?

The visibility cost remains mostly unclear. Publishers are improving their presence in AI tools, and some sell GEO products to brands seeking more mentions and citations in large language models.

Microsoft, the only backer to address visibility directly, warns that disallowing AI-grounding crawlers will reduce brand visibility in AI experiences.

That helps explain caution toward Google’s AI opt-outs: excluding content from AI-powered search can cost visibility and traffic publishers cannot afford to lose.

Takeaway: Opting out may protect content while reducing the exposure publishers bought through AI products.

What remains unanswered?

The token covers training only. It does not govern AI inference, or AI usage, when a model generates responses or retrieves and processes publisher content in real time. Inference now anchors recent AI licensing deals.

Compliance also would not stop a provider obtaining the same information through other crawlers, third-party suppliers or open-source crawls such as Common Crawl. No participating company has promised to avoid buying third-party scraped data or using such archives, Bannister said.

Springer argued the draft normalizes opt-in-unless-opted-out, allowing presumed training permission unless a publisher objects. He said owners should decide through direct permission, not accept a default.

Takeaway: One switch covers one route to the data; the wider consent problem remains.