Report That Cloudflare AI Bot Blocking Prevents Googlebot From Indexing Sites via @sejournal, @martinibuster
Webinar How Freshpet Earned AI's Trust The GEO playbook behind Freshpet's AI Overview citations, plus checks to run on your own brand. Free, live Aug 20. Guide Local Google Visibility Guide + Cheat Sheet Track how your business appears across Google Search, Maps, and Gemini. Listings, reviews, and competitor signals all in one view. Webinar How Freshpet Earned AI's Trust The GEO playbook behind Freshpet's AI Overview citations, plus checks to run on your own brand. Free, live Aug 20. Rundown Why Your Brand Drops Out of AI Answers Five articles on the signals behind citation fluctuation, where models source citations, and how to audit your content for AI visibility gap Webinar New AI Search & SEO KPIs: 4 Real Signals AI mentions and citations are benchmarks, not decisions. Get 4 traffic-predictive signals drawn from real bot data across hundreds of sites. 🔥SEJ Pro Course: Own Your Brand’s Promo Code & Coupon Search Results Before Parasites Do Anecdotal report claims that blocking AI training bots with Cloudflare prevents Googlebot from indexing websites. SEJ STAFF Roger Montti 4 seconds ago ⋅ 3 min read SEJ STAFF Roger Montti Owner - Martinibuster.com at Martinibuster.com Bio Follow There’s a report that blocking AI training bots with Cloudflare is also blocking legitimate Googlebot and Bingbot from indexing a website. Cloudflare recently announced that this would happen beginning in September. It’s unclear whether this is a fluke or user error but it does serve as a reminder to be aware of the pending change and to adjust settings accordingly. A Redditor in the r/SEO community asked if anyone else has experienced Cloudflare blocking Googlebot as an AI trainer. “Has anyone else run into Cloudflare AI Crawlers blocking Googlebot/Bingbot? I’m testing Cloudflare’s new AI Crawlers & Scrapers feature and noticed something unexpected. When I set AI Training = Block, both Googlebot and Bingbot start receiving HTTP 403 responses when trying to fetch my sitemap. As soon as I disable the AI Training block, the sitemap is accessible again. From what I understand, Cloudflare now classifies Googlebot and Bingbot as “Search + Training” bots, so enabling the training block appears to affect them as well. Google’s John Mueller responded to ask for a DM to take a closer look. Another Redditor asked if maybe Cloudflare was blocking rogue bots that were spoofing Googlebot’s user agent. The original poster replied that wasn’t the case and offered more information: “I understand that Cloudflare blocks fake Googlebots, but in my case I’m referring to what Cloudflare itself shows in the dashboard. When I enable Bot Fight Mode, uploading my sitemap returns a 403 error. As soon as I disable it, the 403 disappears. Also, in the AI Crawlers section, Cloudflare shows Googlebot and Bingbot as blocked automatically. That’s why I’m wondering if Bot Fight Mode is interfering with legitimate crawlers or if the dashboard is reporting something else.” In it’s official announcement states that beginning September 15 Cloudflare will begin blocking mixed-usage bots, which are bots like Googlebot that simultaneously index websites for search and collect data AI training. The effect is that Cloudflare will block Googlebot. “On September 15, 2026, Cloudflare will set updated defaults for new domains: bots classified as Training or as Agent will be blocked on pages that display ads, and Search will remain allowed. Mixed-purpose crawlers that combine Search and Training will also be blocked by all configurations to block AI training, including the legacy “Block AI bots” option. Before September 15, all customers can opt out of these new defaults.” That kind of blocking is supposed to go into effect in September so it’s curious that someone is reporting that this is happening right now. Read Full Bio SEJ STAFF Roger Montti Owner - Martinibuster.com at Martinibuster.com I have 25 years hands-on experience in SEO, evolving along with the search engines by keeping up with the latest ... Learn how to connect search, AI, and PPC into one unstoppable strategy. Why Google’s Spam Problem Is Getting Worse Why Do Budgets Overspend Even With A Target ROAS or CPA? – Ask A PPC SEO Pulse: Google Updates Console, Maps & AI Mode Flow Join 75,000+ Digital Leaders. Learn how to connect search, AI, and PPC into one unstoppable strategy. Learn how to connect search, AI, and PPC into one unstoppable strategy. In a world ruled by algorithms, SEJ brings timely, relevant information for SEOs, marketers, and entrepreneurs to optimize and grow their businesses -- and careers.
Source: Search Engine Journal
This article has been carefully curated and reformatted for educational and informational purposes. Full credit goes to the original publisher.
📚 Visit more helpful articles on Joab Peters Blog
No comments