OpenAI and Tech Giants Back New AI Scraping Opt-Out Standard
OpenAI, Google, Microsoft, and Anthropic have agreed to support a single robots.txt signal to help publishers opt out of AI scraping.
Major technology companies, including OpenAI, Google, Microsoft, and Anthropic, announced support for a new shared standard that allows website publishers to opt out of having their content scraped for artificial intelligence models. The initiative aims to simplify how websites manage their data preferences.
How does the new AI opt-out standard work?
Currently, website owners must block AI web crawlers individually by updating their site files for every single tech company. Web crawlers are automated programs that scan the internet to index content or collect data.
The new system introduces a shared policy token placed inside a website's robots.txt file, which is a standard text file used to guide web crawlers. Proposed by Kevin Kelley, a technical lead for standards at Anthropic, through the Internet Engineering Task Force, this single signal communicates a site's AI training preferences to all participating AI companies at once.
Which companies are supporting the new setting?
Several leading AI developers have committed to honoring the new shared token when their crawlers encounter it on a website:
Meta has also expressed support for the shared signal through communications head Andy Stone, though Meta has not yet explicitly stated whether it will honor the setting.
- Anthropic has already adopted the new signal.
- OpenAI plans to implement the token in the coming months.
- Google and Microsoft plan to add support for the signal next year.
How does this impact search engine visibility?
Publishers do not have to worry about losing their search engine rankings when opting out of AI model training.
Microsoft stated it will offer granular controls letting publishers opt out of AI training or AI grounding, which is the practice of linking AI models to factual sources to reduce errors. Microsoft confirmed that websites will still appear in Bing search results regardless of their choice.
Similarly, Google previously created controls allowing publishers to opt out of AI search features or AI training without affecting their visibility in standard search results.
Why it matters for businesses and publishers
As AI companies scrape vast amounts of online content to train their systems, publishers face challenges controlling how their intellectual property is used. Other efforts have emerged to address scraping, such as Cloudflare blocking AI crawlers by default and the Really Simple Licensing standard, which allows sites to charge crawlers.
The new shared token gives business owners and content creators a straightforward, unified way to protect their content without managing complex rules for every individual AI provider.
Comments 0
No comments yet. Start the conversation.