Google joins OpenAI and Anthropic in promising new scraping controls

The 30-second read

Major technology firms including Google, OpenAI, and Anthropic have pledged support for a unified robots.txt signaling standard. This initiative allows publishers to explicitly block data scraping specifically for artificial intelligence model training.

• 01Google joins OpenAI and Anthropic in adopting a new common signal for web crawling controls
• 02The standardized protocol enables websites to opt out of AI training data collection
• 03Microsoft and other industry leaders are supporting the unified robots.txt update
• 04New controls aim to address publisher concerns regarding unauthorized content use by AI developers

AI-generated summary · Verify at source