For two years a publisher who wanted to refuse AI training had one real option: disappear from Google. The same bot did both jobs, so the same rule caught both.

The deadline Cloudflare set on 1 July arrived on 15 September, and mixed use crawlers are now blocked by default on pages that carry ads. A mixed use crawler is one bot doing two jobs, search and AI training, which is how Googlebot works. On the same day Cloudflare published the escape hatch: an Accountable designation with four requirements, worked out with Apple, Google, Microsoft, Amazon, Anthropic, Meta and OpenAI. An accountable operator honours a training opt out in robots.txt, offers a separate opt out of AI summaries, reports per URL whether a page was used for training, and guarantees that refusing training does not lower a site's search ranking. A new Disallow AI Training setting keeps accountable crawlers allowed for search and blocks every other training crawler. The older Block AI Bots switch and managed robots.txt are on the way out.

Why this one is different

Every previous answer to this was a demand or a lawsuit. This one is a definition, and the companies being asked to meet it helped write it. The fourth requirement is the one that matters: the reason the opt out was unusable was never technical, it was the fear that refusing training would quietly cost you the traffic. A written guarantee from the search engine is the only thing that removes that fear, and it is now attached to a label a publisher can check.

The opt out was always available. It just cost you your traffic.

How we got here

1. 1 Jul 2026Cloudflare announces the September default and replaces Pay Per Crawl with Pay Per Use, which pays when content is used in an answer rather than when a page is fetched.
2. 16 Jul 2026The EU orders Google to open Android to rival assistants under the Digital Markets Act.
3. 2 Sep 2026The US government tells a court that training on copyrighted text is fair use.
4. 15 Sep 2026The default takes effect, and the Accountable designation is published the same day.

What it does and does not mean

None of this is law. Accountable is a designation one company grants, against criteria that company wrote with the crawler operators, and it can be changed without anybody voting. The new defaults apply to domains onboarded after 15 September, so an established site on an existing plan sees nothing change until somebody clicks something. And the numbers say the fight is smaller than it sounds: 17% of sites block training, under 1% block search, which is a minority position with an enormous amount written about it. What changed is that the two questions came apart. A publisher can now answer them separately, and for the first time the answer to one does not decide the other.