AI policy token wins support, but enforcement stays unresolved

Anthropic says it is rolling out adoption now, OpenAI plans to honor the signal in the coming months, Google targets 2027, and Microsoft will replace two crawler preferences on Feb. 1, 2027. The proposal, submitted to the Internet Engineering Task Force on Oct. 30, still lacks formal approval.

The immediate limitation is enforcement. ScalePost tracked more than 50 publisher properties; crawlers operated by the four backers made roughly 1 billion requests over 30 days, but only 0.15% were for robots.txt files. The data suggests crawlers rarely check publisher instructions before scraping.

The draft adds an AI-training instruction to robots.txt, the file websites use to control crawling. Its proposed entry starts User-Agent: __usage_ai-training, followed by disallow or allow.

What does the policy token change for publishers?

Instead of listing every crawler and granting or denying access, publishers would set one broad preference for participating AI and technology companies. They can allow exceptions, such as press-kit access, without separate instructions for each participating crawler.

The token covers model training, not AI inference, where a model retrieves or processes publisher content in real time to generate answers or complete tasks. Inference sits at the core of most recent AI licensing deals, while publishers have otherwise tried bot blocking, legal action and licensing negotiations.

Draft status and official commitments

Kevin Kelley, Anthropic's technical lead for standards, submitted the proposal to the IETF on Oct. 30. The IETF's AI Preferences, or AIPREF, effort is developing standards for website owners, and Anthropic, Google, Microsoft and OpenAI participate; the draft still requires IETF deliberation.

Alex Springer, technical lead at publisher coalition SPUR, said unilateral support for an unreviewed draft conflicts with the IETF's public consensus process. “They appear to have decided to unilaterally wield their collective weight in a concerted effort to force a new de facto standard in place,” he said. He said the draft falls far short of what content owners require.

• Anthropic says it is rolling out adoption now.
• OpenAI will honor the token in the coming months; the mechanism will not replace its existing crawler controls.
• Google aims to support the token in 2027, building on Google-Extended, a robots.txt-controlled crawler for AI training access introduced in 2023.
• Microsoft will comply and offer an AI-grounding opt-out. Its preferences will replace NoArchive for training and NoCache for grounding starting Feb. 1, 2027.
• Meta supports a common signal for publisher preferences but did not commit to honoring the token. Spokesperson Andy Stone said Meta will work with other AI companies on standards and industry implementation.

The four commitments make this one of the first proposed standards with support across multiple major AI and search companies. They do not make the draft an approved IETF standard or guarantee compliance.

Enforcement remains the central gap

Like robots.txt, the token offers no mechanism to ensure that companies honor publisher preferences. It expresses intent rather than securing an enforceable right to block use.

AI providers face copyright infringement lawsuits, pressure from government regulators and proposed legislation targeting crawlers, creating reasons to appear responsive to publishers. Paul Bannister, chief strategy officer at Raptive, called the proposal window dressing and said: “I think this is useless and is marketing, and far, far, far too late.” He concluded that it does nothing and fails to solve the problem.

The token could still add one layer alongside CDN-level crawler blocking and tools such as TollBit. Bannister compared that limited protection with a front-door lock that deters some thieves, and described the proposal as one more measure among roughly 1,000 tools or actions publishers need.

Visibility and consent remain unresolved

For most providers, the effect on publisher reach remains unclear. Microsoft alone addressed the trade-off, warning that disallowing AI grounding, where crawlers scrape sites for real-time information, will reduce brand visibility in AI experiences.

Publishers have worked to improve their AI visibility, and some sell GEO products to brands seeking more mentions and citations in large language models. Publishers who use Google's existing AI opt-outs can exclude content from AI-powered search features but may sacrifice valuable visibility and traffic.

Even if an AI crawler honors the token, its provider may obtain the same material through another crawler or another source. Bannister said none of the four has promised not to buy data from third-party scrapers or use open crawls such as Common Crawl.

Springer said the draft appears to reinforce training as the default unless publishers explicitly object. He argued that this risks normalizing presumed permission, while content owners should retain control through direct permission.

The next test is practical: whether providers honor the signal before training crawlers collect content, rather than merely supporting its language. Publishers will need evidence that the control reduces unwanted use without reducing their mentions, citations and traffic in AI products.