Cloudflare revealed its sixth annual 12 months in Assessment, providing a complete seems to be at Web visitors, safety, and AI crawler exercise throughout 2025.
The report attracts on knowledge from Cloudflare’s community, which spans greater than 330 cities throughout 125 nations and handles over 81 million HTTP requests per second on common.
The AI crawler findings stand out. Googlebot crawled much more internet pages than another AI bot, reflecting Google’s dual-purpose strategy to crawling for each search indexing and AI coaching.
Googlebot High AI Crawler Visitors
Cloudflare analyzed profitable requests for HTML content material from main AI crawlers throughout October and November 2025. The outcomes confirmed Googlebot reached 11.6% of distinctive internet pages within the pattern.
That’s greater than 3 occasions the pages seen by OpenAI’s GPTBot at 3.6%. It’s practically 200 occasions greater than PerplexityBot, which crawled simply 0.06% of pages.
Bingbot got here in third at 2.6%, adopted by Meta-ExternalAgent and ClaudeBot at 2.4% every.
The report famous that as a result of Googlebot crawls for each search indexing and AI mannequin coaching, internet publishers face a tough selection. Blocking Googlebot’s AI coaching means risking search discoverability.
Cloudflare wrote:
“As a result of Googlebot is used to crawl content material for each search indexing and AI mannequin coaching, and due to Google’s long-established dominance in search, Website online operators are basically unable to dam Googlebot’s AI coaching with out risking search discoverability.”
Associated: Full Crawler Listing For AI Consumer-Brokers
AI Bots Now Account For 4.2% of HTML Requests
All through 2025, AI bots (excluding Googlebot) averaged 4.2% of HTML requests throughout Cloudflare’s buyer base. The share fluctuated between 2.4% in early April and 6.4% in late June.
Googlebot alone accounted for 4.5% of HTML requests, barely greater than all different AI bots mixed.
The share of human-generated HTML visitors began 2025 at seven share factors beneath non-AI bot visitors. By September, human visitors started exceeding non-AI bot visitors on some days. As of December 2, people generated 47% of HTML requests whereas non-AI bots generated 44%.
Crawl-to-Refer Ratios Present Broad Variation
Cloudflare tracks how usually AI and search platforms ship visitors to websites relative to how usually they crawl. A excessive ratio means heavy crawling with out sending customers again to supply websites.
Anthropic had the very best ratios amongst AI platforms, starting from roughly 25,000:1 to 100,000:1 in the course of the second half of the yr after stabilizing from earlier volatility.
OpenAI’s ratios reached as excessive as 3,700:1 in March. Perplexity maintained the bottom ratios amongst main AI platforms, typically beneath 400:1 and below 200:1 from September onward.
For comparability, Google’s search crawl-to-refer ratio stayed a lot decrease, typically between 3:1 and 30:1 all year long.
Consumer-Motion Crawling Grew Over 20X
Not all AI crawling is for mannequin coaching. “Consumer motion” crawling happens when bots go to websites in response to consumer questions posed to chatbots.
This class noticed the quickest development in 2025. Consumer-action crawling quantity elevated greater than 15 occasions from January by means of early December. The pattern intently matched the visitors sample for OpenAI’s ChatGPT-Consumer bot, which visits pages when customers ask ChatGPT questions.
The expansion confirmed a weekly utilization sample beginning in mid-February, suggesting elevated use in colleges and workplaces. Exercise dropped throughout June by means of August when college students have been on break and professionals took holidays.
AI Crawlers Most Blocked In Robots.txt
Cloudflare analyzed robots.txt information throughout practically 3,900 of the highest 10,000 domains. AI crawlers have been essentially the most ceaselessly blocked consumer brokers.
GPTBot, ClaudeBot, and CCBot had the very best variety of full disallow directives. These directives inform crawlers to keep away from complete websites.
Googlebot and Bingbot confirmed a special sample. Their disallow directives leaned closely towards partial blocks, possible centered on login endpoints and non-content areas fairly than full web site blocking.
Civil Society Turned Most-Attacked Sector
For the primary time, organizations within the “Folks and Society” vertical have been essentially the most focused by assaults. This class contains non secular establishments, nonprofits, civic organizations, and libraries.
The sector acquired 4.4% of world mitigated visitors, up from below 2% initially of the yr. Assault share jumped to over 17% in late March and peaked at 23.2% in early July.
Many of those organizations are protected by Cloudflare’s Mission Galileo.
Playing and video games, the most-attacked vertical in 2024, noticed its share drop by greater than half to 2.6%.
Different Key Findings
Cloudflare’s report included a number of extra findings throughout visitors, safety, and connectivity.
International Web visitors grew 19% year-over-year. Progress stayed comparatively flat by means of mid-April, then accelerated after mid-August.
Submit-quantum encryption now secures 52% of human visitors to Cloudflare, practically double the 29% share initially of the yr.
ChatGPT remained the highest generative AI service globally. Google Gemini, Windsurf AI, Grok/xAI, and DeepSeek have been new entrants to the highest 10.
Starlink visitors doubled in 2025, with service launching in additional than 20 new nations.
Practically half of the 174 main Web outages noticed globally have been brought on by government-directed shutdowns. Cable minimize outages dropped practically 50%, whereas energy failure outages doubled.
European nations dominated Web high quality metrics. Spain topped the record for total Web high quality, with common obtain speeds above 300 Mbps.
Why This Issues
The AI crawler knowledge ought to impacts how you concentrate on bot entry and visitors.
Google’s dual-purpose crawler creates a aggressive benefit. You possibly can block different AI crawlers whereas preserving Googlebot entry for search visibility, however you may’t separate Google’s search crawling from its AI coaching crawling.
The crawl-to-refer ratios assist quantify what publishers already suspected. AI platforms crawl closely however ship little visitors again. The hole between crawling and referring varies broadly by platform.
The civil society assault knowledge issues in the event you work with nonprofits or advocacy organizations. These teams now face the very best charge of assaults.
Trying Forward
Cloudflare expects AI metrics to alter because the house continues to evolve. The corporate added a number of new AI-related datasets to this yr’s report that weren’t obtainable in earlier editions.
The crawl-to-refer ratios could change as AI platforms alter their search options and referral habits. OpenAI’s ratios already confirmed some decline by means of the yr as ChatGPT search utilization grew.
For robots.txt administration, the information reveals most publishers are selecting partial blocks for main search crawlers whereas absolutely blocking AI-only crawlers. The year-end state of those directives offers a baseline for monitoring how writer insurance policies evolve in 2026.
Featured Picture: Mamun_Sheikh/Shutterstock
