Yesterday, Google confirmed it will support a new shared control that lets any website tell the biggest AI companies at once - OpenAI, Google, Microsoft and Anthropic are the first backers - to stop training on its content with a single line in its robots.txt file. Anthropic has already adopted it. Google and Microsoft plan to support it in 2027, and OpenAI says it will follow in the coming months.
If you own a business website, this is the first time you will be able to set one AI policy across participating providers instead of chasing a dozen different crawlers - though rollout is staged through 2027 and not every company has committed. Here is what was announced, and how to think about the decision.
What the new standard is
The mechanism is a shared "policy token" inside robots.txt - the same plain-text file that has told search engines what they may crawl for decades. The proposal comes from the Internet Engineering Task Force (IETF), authored by Kevin Kelley, a technical lead for standards at Anthropic.
Until now, opting out of AI training meant writing separate rules for every AI company's crawler - GPTBot for OpenAI, Google-Extended for Google, ClaudeBot for Anthropic, and so on. Most small business sites we audit have either none of these or a half-finished set copied from a blog post. The shared token replaces that patchwork with one instruction, and company-specific rules keep working alongside it for exceptions.
Who is honoring it, and when
From the companies' own announcements and reporting by The Verge: Anthropic is already honoring the signal. Google and Microsoft plan support in 2027 - Google says it will treat the shared opt-out as equivalent to disallowing Google-Extended, its existing control from 2023. OpenAI says it will implement it "in the coming months." Meta has signaled support for a common signal but has not committed to honoring it - a gap worth noting.
The distinction that matters: training vs grounding
This is the part most commentary will get wrong. "AI training" - using your content to build future models - is not the same as "grounding," when an AI assistant quotes your site to answer a question and links back to you. Microsoft's announcement explicitly separates the two: you can opt out of training while staying available for grounding, and your pages stay in Bing search results either way. Google already works this way: sites that allow Google-Extended remain eligible to appear in Gemini answers and receive clicks.
In other words, this is not a choice between "let AI use my site" and "disappear from AI." It is a choice about whether your content becomes free raw material for future models.
Should your business opt out?
Our honest take for most Indian small businesses: probably not - though the right answer depends on what your content is worth, and the exceptions below matter.
The websites with a genuine case for blocking AI training are those whose content is the product - news publishers, original research, paid libraries. Reddit just forced its platform behind a login partly for this reason, after losing a million weekly Google referrals to AI answers (we covered that yesterday). Its leverage is licensing deals: OpenAI and Google pay for access to Reddit's data.
A dental clinic in Noida, a coaching institute in Ghaziabad, or a furniture manufacturer in Gurugram is in the opposite position. Your content exists to be found. If an AI assistant learns from your service pages and later recommends a business like yours, that is closer to advertising than theft. The realistic risk is not that AI trains on your pages - it is that AI never mentions you at all.
There are narrow exceptions: proprietary price lists, course content you sell, original photography portfolios. A training opt-out for those sections is reasonable, and the standard supports exactly that kind of partial rule.
What to actually do this week
- Look at your current file (yourdomain.com/robots.txt). If it is empty or a template leftover, fix that regardless of AI.
- Decide deliberately: allow, block training but allow grounding, or block specific sections - and write the decision down so future site changes do not silently undo it.
- Do not block AI crawlers hoping to "protect your rankings." Google has stated these preferences do not affect search visibility.
- If your developer set a blanket block last year "to be safe," review it. We routinely find sites accidentally opting out of the very AI citations they want.
Taken together with Reddit's logged-in pivot and Google's ads arriving inside AI answers in India, the direction is clear: the web is sorting itself into content that AI pays for, content that AI cites, and content that AI ignores. Small businesses should aim squarely at the second category.
Not sure what your robots.txt currently tells AI companies, or whether your site is visible to AI search at all? VAIR IT Technologies will check it as part of a free consultation and tell you plainly what to change. Reach us at vairit.com/contact.
