OpenAI, Google, Microsoft and Anthropic say they will adopt a new standard that lets publishers opt out of being scraped by AI crawlers, a shift that would allow site owners to state their preferences once instead of company by company.
How would a shared AI opt-out signal work?
The shared signal, called a “policy token,” was proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force. It would live inside a website’s robots.txt file, the long-used text file that has signaled which web crawlers can and cannot scrape a site. Those crawlers might be indexing pages for search, archiving them or gathering material for AI training.
Right now, sites have to communicate their AI training preferences to each company separately. The new token is meant to change that by letting publishers apply one set of rules across several AI companies at the same time.
The move comes as publishers deal with a steady stream of AI crawlers pulling their content into AI models. Some companies and organizations have already pushed back. The Really Simple Licensing (RSL) standard gives publishers a way to charge crawlers that visit their sites, and Cloudflare blocks AI crawlers by default.
Which AI companies will honor the policy token?
OpenAI, Google, Microsoft and Anthropic say they will restrict their AI crawlers from accessing a site when they encounter the shared token. The companies are not on the same timeline, however. Anthropic has already adopted the signal. Google and Microsoft plan to support it next year. OpenAI says it will implement it in the “coming months.”
Can publishers opt out of AI training and stay visible in search?
Microsoft also plans to offer more granular controls. Publishers will be able to opt out of AI training or AI grounding, the process of pointing an AI model to a verified source of information to reduce hallucinations. Microsoft says websites will still appear in Bing’s search results regardless of their preferences. Google has already rolled out similar controls for AI search optimization that let sites decide whether their content appears in AI search features or is used for AI training, without affecting their visibility in search.
Meta has also weighed in. Andy Stone, the company’s communications head, signaled support for the new setting but did not say whether Meta will honor it.
“We believe creating a common signal that conveys such preferences to multiple providers at once is an important step,” Stone said. “And we look forward to working with other AI companies on the standards and their implementation across the industry.”
The signal’s success will depend on whether the companies follow through on their commitments and how quickly the rollout happens across the industry. For now, publishers have a proposed common language for telling AI companies what they can use, and a commitment from several of the largest AI developers to listen.
Questo articolo è stato scritto con l'assistenza dell'IA.
News Factory APP - notizie agentiche per potenziare il tuo SEO e AEO.