Hexa Tiger 
Scale My Publication 
SEO for Public Figures 
A web scraping company’s analysis indicates that Cloudflare‘s recent changes have removed AI training opt-out instructions from the robots.txt files of 118 websites. The findings from HasData, a web scraping API provider, suggest that these changes, implemented around September 15, 2026, have made it more difficult for AI models to identify websites that do not wish to have their content used for training.
HasData’s AI Crawler Block Index, updated on September 29, 2026, compared data from July 2026 with a re-run on September 16, 2026. The study examined 10,894 websites, including those from the Tranco top 10,000 list and news homepages. The analysis revealed that many sites declare one policy in their public rules file but behave differently in practice.
Cloudflare’s Changes Impact AI Opt-Outs
The core of the HasData finding centers on Cloudflare’s September 15 update. This change stripped machine-readable “no-training” lines from the robots.txt files of 118 sites that had previously included them. These lines, such as `ai-train=no`, provided a clear signal to AI crawlers about a site’s training preferences. Without these directives, the public rules file becomes a less reliable guide for AI systems.
Roman Milyushkevich, co-founder and chief technology officer at HasData, noted that “robots.txt is a suggestion.” The study found that prior to Cloudflare’s update, 39.5% of sites that disallowed GPTBot in their robots.txt file still served it pages. This indicates a broader trend of declared policy diverging from actual server behavior.
Publishers More Cautious Than Open Web
The HasData report highlights a significant difference in how news publishers and the broader open web approach AI crawlers. In July 2026, 56.4% of news publishers blocked at least one AI crawler in their robots.txt files, compared to 10.3% of top web domains. This gap widened for specific crawlers like GPTBot, with 50.5% of publishers blocking it versus 7.9% of top web sites.
HasData attributes this disparity to business models. Publishers often hold content that AI models seek for training and citation, and they also offer subscription revenue that competes with AI-generated summaries. Most e-commerce, blog, and documentation sites, however, do not perceive AI training as a direct risk.
Enforcement Varies by Identity and CDN
The study also explored how sites enforce their policies. When tested from a datacenter IP address, publisher sites served the GPTBot request only 54.2% of the time, while a standard Chrome browser received a page 83.8% of the time, a gap of about 30 points. This suggests enforcement is often identity-based, relying on user-agent strings rather than IP ranges.
Content delivery networks (CDNs) also play a role in how requests are handled. Cloudflare’s approach was characterized as “Challenge-based,” with 51.3% of requests served, 16.9% hard-blocked, and 24.7% met with a challenge. This contrasts with CDNs like Fastly, which had a 61.5% hard-block rate.
Impact on AI Training and Future Considerations
While Cloudflare’s September 15 update removed the machine-readable opt-out lines, HasData found that for 16 out of 21 affected sites tested, GPTBot was still blocked or challenged by the server. This suggests that the removal of the robots.txt directive did not automatically open the door for AI training on these sites. However, the machine-readable signal, crucial for crawlers to decide whether to fetch a page, was lost.
The report notes that the long-term implications depend on future developments, including the completion of Cloudflare’s migration for existing customers and the rollout of Microsoft’s support for a robots.txt no-training signal in early 2027. Google’s planned page-level controls for its search generative AI features, due March 3, 2027, may also influence how AI models access and use web content.
