OpenAI, Google, Microsoft and Anthropic have agreed to support a shared mechanism that will allow website owners and publishers to communicate whether their content can be used to train artificial intelligence models. The proposal aims to replace the separate instructions that websites currently need to maintain for different AI companies.
The mechanism will use a common policy signal inside a website’s robots.txt file, which communicates instructions and preferences to automated web crawlers. Instead of identifying individual AI companies and configuring separate restrictions, publishers could use one shared instruction to express their preference across participating providers.
The development was announced on October 30, 2026, as publishers continue to raise concerns about AI companies collecting online articles, images and other material to develop generative models. Anthropic has already adopted the proposed signal, while OpenAI, Google and Microsoft have outlined their respective implementation plans.
One Shared Signal Could Simplify AI Training Restrictions
The proposal introduces a policy token called __usage_ai-training. Website operators would place it in a separate user-agent group inside their robots.txt file to indicate whether content should be used for AI training.
For example, a publisher could use the instruction Disallow: / under this policy token to express a preference against using the entire website for AI training. It could also allow selected sections, such as a press-information page, while restricting other content.
The proposal was submitted to the Internet Engineering Task Force (IETF) by Kevin Kelley, an Anthropic technical lead for standards. It is part of the organisation’s AI Preferences working group, which is examining ways to make website usage preferences easier for digital services to interpret.
Until now, publishers have generally had to manage separate crawler instructions for different companies. OpenAI, Google and Anthropic, for instance, have their own crawler identification systems. A common signal could reduce the administrative work involved in maintaining these restrictions as more AI services enter the market.
However, the proposal is not yet an adopted internet standard. The IETF’s document tracker classifies it as an individual Internet-Draft and explicitly states that it has no formal standing in the organisation’s standards process. The draft may still be revised as discussions continue.
Companies Plan Different Implementation Timelines
The four companies have expressed support for the shared mechanism, but their implementation schedules differ.
OpenAI said it intends to honour the common signal in robots.txt and plans to implement it in the coming months. The company added that its existing crawler controls will continue to operate alongside the new mechanism.
Google said it plans to support the shared control in 2027. The company already offers Google-Extended, which allows website owners to communicate restrictions concerning the use of their content for certain AI purposes. The new signal would provide an additional, common way to express training preferences across participating providers.
Microsoft has outlined a broader set of publisher controls. Alongside the shared AI-training preference, it is introducing separate controls for AI grounding, the process through which an AI system uses website information to help generate an answer to a user’s question.
Microsoft said its new preferences will begin taking effect as its crawlers discover them from February 1, 2027. It also said publishers will remain eligible for traditional Bing search results even if they restrict the use of their content for AI training or grounding.
Anthropic has already adopted the shared training signal. Meta has expressed support for the idea of a common mechanism, but has not confirmed that it will honour the proposed signal.
AI Training And Search Visibility Are Not The Same
One important aspect of the proposal is the distinction between allowing a website to be crawled and allowing its content to be used for AI training.
The new policy token communicates a preference about how content may be used. It does not automatically prevent every crawler from accessing a website, nor does it replace the separate instructions that publishers use to manage search-engine crawling.
This distinction matters because website owners may want their articles to remain discoverable through Google Search or Bing while restricting their use in training generative AI models.
AI grounding is another separate issue. A website may want its content to remain available to an AI service when it answers a question and cites the original article, while preventing the same material from being used to train a model’s underlying parameters.
Microsoft’s announcement specifically addresses these separate choices through additional page-level controls. Google has also said its existing search visibility arrangements will continue alongside the proposed training signal.
The proposed mechanism cannot reverse data collection that has already taken place. If an AI company has previously collected website content, adding the new instruction does not automatically remove that material from its existing datasets or models.
What This Means For Publishers And News Organisations
The agreement could give digital publishers a simpler way to communicate their preferences to major AI developers without maintaining a growing list of company-specific restrictions.
For news organisations, this is particularly relevant because their reporting, photographs and archives are valuable sources of information for generative AI systems. Publishers have increasingly questioned whether AI companies should be able to use such material without separate licensing arrangements or clearer consent mechanisms.
However, the proposed signal does not establish a payment system, guarantee compensation or resolve disputes over copyright and fair use. It is primarily a common technical method for communicating usage preferences.
Its effectiveness will depend on whether AI providers consistently honour the signal and how the IETF’s standards discussions progress. The development marks a step towards giving website owners more control over how their content is used, but it does not yet settle the wider dispute between publishers and AI companies over access to online material.
