ChatGPT Now Crawls 3.6x More Than Googlebot

This publish was sponsored by Alli AI. The opinions expressed on this article are the sponsor’s personal. 

Everybody assumes Googlebot is the dominant crawler hitting their web site. That assumption is now fallacious.

We analyzed 24,411,048 proxy requests throughout 78,000+ pages on 69 buyer web sites on Alli AI’s crawler enablement platform over a 55-day interval (January to March 2026). OpenAI’s ChatGPT-Consumer crawler made 3.6x extra requests than Googlebot throughout our information pattern. And that’s not even counting GPTBot, OpenAI’s separate coaching crawler.

A observe on methodology: Crawler identification used consumer agent string matching, verified towards revealed IP ranges. Request metrics are measured on the proxy/CDN layer. The dataset covers 69 web sites throughout quite a lot of industries and sizes, predominantly WordPress-based. Full methodology is detailed on the finish.

Discovering 1: AI Crawlers Now Outpace Google 3.6x & ChatGPT Leads the Pack

Once we ranked each recognized crawler by request quantity, the outcomes have been unambiguous:

ChatGPT-Consumer made extra requests than Googlebot, Amazonbot, and Bingbot mixed.

Grouped by goal, AI-related crawlers (ChatGPT-Consumer, GPTBot, ClaudeBot, Amazonbot, Applebot, Bytespider, PerplexityBot, CCBot) made 213,477 requests versus 59,353 for conventional search crawlers (Googlebot, Bingbot, YandexBot). AI crawlers at the moment are making 3.6x extra requests than conventional search crawlers throughout our community.

Discovering 2: OpenAI Makes use of 2 Crawlers (And Most Websites Don’t Know the Distinction)

OpenAI operates two distinct crawlers with very totally different functions.

ChatGPT-Consumer is the retrieval crawler. It fetches pages in actual time when customers ask ChatGPT questions that require up-to-date internet info. This determines whether or not your content material seems in ChatGPT’s solutions.

GPTBot is the coaching crawler. It collects information to enhance OpenAI’s fashions. Many websites block GPTBot through robots.txt however not ChatGPT-Consumer, or vice versa, with out understanding the distinct penalties of every.

Mixed, OpenAI’s crawlers made 142,225 requests: 3.8x Googlebot’s quantity.

The robots.txt directives are separate:

Consumer-agent: GPTBot      # Coaching crawler — feeds OpenAI's fashions
Consumer-agent: ChatGPT-Consumer # Retrieval crawler — fetches pages for ChatGPT solutions

Discovering 3: AI Crawlers Are Quicker & Extra Dependable, However Their Quantity Provides Up

AI crawlers are considerably extra environment friendly per request:

Two probably causes. First, AI retrieval crawlers are fetching particular pages in response to consumer queries, not exhaustively discovering website structure. They know what they need, they seize it, they usually go away. Second, whereas all crawlers on our infrastructure obtain pre-rendered responses, Googlebot’s broader crawl sample means it requests a wider vary of URLs, together with stale paths from sitemaps and its personal legacy index, which provides latency from redirect chains and error dealing with that retrieval crawlers keep away from solely.

However there’s a catch: whereas every particular person request is light-weight, the sheer quantity means mixture server load is substantial. ChatGPT-Consumer at 11ms × 133,361 requests remains to be an actual infrastructure price, simply distributed in another way than Googlebot’s fewer, heavier requests.

Discovering 4: Googlebot Sees a Totally different (Worse) Model of Your Web site

Googlebot’s 96.3% success price versus near-perfect charges for AI crawlers reveals an essential structural distinction.

Googlebot acquired 624 blocked responses (403) and 480 not discovered errors (404), accounting for 3% of its requests. In the meantime, ChatGPT-Consumer achieved 99.99% success. PerplexityBot hit an ideal 100%.

Why the hole? The almost certainly clarification is index age and crawl habits, not website misconfiguration.

Googlebot maintains an enormous legacy index constructed over years of steady crawling. It routinely re-requests URLs it already is aware of about — together with pages which have since been deleted (404s) or restructured (403s). That is regular habits for a search engine sustaining an index of this scale, but it surely means a significant share of Googlebot’s requests are directed at URLs that not exist.

AI crawlers don’t carry that baggage. ChatGPT-Consumer fetches particular pages in response to real-time consumer queries, concentrating on content material that’s at the moment related and linked. That’s a structural benefit that produces near-perfect success charges.

Business Studies Affirm AI Crawling Surged 15x in 2025

These findings align with broader business traits. Cloudflare’s 2025 evaluation reported ChatGPT-Consumer requests surging 2,825% YoY, with AI “consumer motion” crawling rising greater than 15x over the course of 2025. Akamai recognized OpenAI as the one largest AI bot operator, accounting for 42.4% of all AI bot requests. Vercel’s evaluation of nextjs.org confirmed that not one of the main AI crawlers at the moment render JavaScript.

Our information exhibits this crossover might already be occurring on the website stage for properties that actively allow AI crawler entry.

Your New web optimization Technique: How To Audit, Clear Up & Optimize For AI Crawlers

1. Audit your robots.txt for AI crawlers immediately

Most robots.txt recordsdata have been written for a Googlebot-first world. At minimal, have express directives for ChatGPT-Consumer, GPTBot, ClaudeBot, Amazonbot, PerplexityBot, Applebot, Bytespider, CCBot, and Google-Prolonged.

Our advice: Most companies profit from permitting each retrieval crawlers (ChatGPT-Consumer, PerplexityBot, ClaudeBot) and coaching crawlers (GPTBot, CCBot, Bytespider), coaching information is what teaches these fashions about your model, merchandise, and experience. Blocking coaching crawlers immediately means AI fashions be taught much less about you tomorrow, which reduces your possibilities of being cited in AI-generated solutions down the road.

The exception: when you’ve got content material you particularly want to guard from mannequin coaching (proprietary analysis, gated content material), use granular Disallow guidelines for these paths moderately than blanket blocks.

2. Clear up stale URLs in Google Search Console

Our information exhibits Googlebot hits a 3% error price, principally 403s and 404s, whereas AI crawlers obtain near-perfect success charges. That hole probably displays Googlebot re-crawling legacy URLs that not exist. However these failed requests nonetheless eat the crawl funds.

Audit your GSC crawl stats for recurring 404s and 403s. Arrange correct redirects for restructured URLs and submit up to date sitemaps.

3. Deal with AI crawler accessibility as a definite web optimization channel

Rating in ChatGPT’s solutions, Perplexity’s outcomes, and Claude’s responses is rising as a definite visibility channel. In case your content material isn’t accessible to those crawlers, significantly when you’re operating JavaScript-heavy frameworks, you’re invisible in AI search.

We’ve revealed a stay dashboard displaying how AI crawler visitors breaks down throughout an actual website: which platforms are visiting, how typically, and their share of whole visitors; if you wish to see what this appears like in apply.

4. Plan for quantity, not simply particular person request weight

AI crawlers ship gentle, quick requests, however they ship many of them. ChatGPT-Consumer alone accounted for greater than 133,000 requests in 55 days. The combination server load from AI crawlers is now probably exceeding your Googlebot load. Make sure that your internet hosting and CDN can deal with it, the low per request response occasions in our information replicate the truth that Alli AI serves pre-rendered static HTML from the CDN edge, which is strictly the form of structure that absorbs this quantity with out taxing your origin server.

Methodology

This evaluation is predicated on 24,411,048 HTTP proxy requests processed by Alli AI’s crawler enablement platform between January 14 and March 9, 2026, protecting 69 buyer web sites.

Crawler identification used consumer agent string matching, verified towards revealed IP ranges. For OpenAI crawlers particularly, each request was cross-referenced towards OpenAI’s revealed CIDR ranges. This confirmed 100% of GPTBot requests and 99.76% of ChatGPT-Consumer requests originated from OpenAI’s infrastructure. The remaining 0.24% (requests from spoofed consumer brokers) have been excluded.

Limitations: The dataset is scoped to Alli AI clients who’ve opted into crawler enablement. Crawlers that don’t self-identify through consumer agent are usually not captured. Response time measurements are on the proxy layer, not the origin server.

About Alli AI

Alli AI gives server-side rendering infrastructure for AI and search engine crawlers. This evaluation was produced utilizing information from our proxy infrastructure to assist the web optimization group higher perceive the evolving crawler panorama.

Need to see this information in motion? See the breakdown firsthand by visiting our AI visibility dashboard.

Picture Credit

Featured Picture: Picture by Alli AI. Used with permission.

In-Put up Iamges: Pictures by Alli AI. Used with permission.