A brand new evaluation of 858,457 websites hosted on the Duda platform exhibits how AI crawlers are interacting with web sites at scale. The info gives a clearer view of how crawling exercise is rising and what SEOs and companies ought to do to extend visitors from AI search.
AI Crawling Has Already Reached Scale
AI crawling is rising shortly, with extra requests tied to real-time solutions and most of that exercise coming from a single supplier. The info creates a sample that exhibits which internet sites are being crawled and extra importantly, why.
Yr-Over-Yr Development In LLM Referrals
LLM referral visitors has elevated sharply over the previous yr, with a number of platforms displaying significant features from very totally different beginning factors.
AI Referral Site visitors Patterns
- Whole LLM referrals: 93,484 to 161,469 (+72.7%)
- ChatGPT: 81,652 to 136,095 (+66.7%)
- Claude: 106 to 2,488 (23x progress)
- Copilot: 22 to 9,560 (from near-zero)
- Perplexity: 11,533 to 13,157 (+14.1%)
Development will not be occurring evenly, however throughout the board, referral visitors from AI methods is growing. That makes AI-generated discovery a rising supply of visitors, not a marginal one.
Crawlers Are More and more Fetching Content material To Floor Solutions
AI crawlers are not used primarily for indexing, with most exercise now tied to retrieving content material in actual time to generate solutions for customers.
Most crawling is now occurring in response to consumer queries relatively than for constructing an index, which adjustments how content material is accessed and used.
- Person Fetch (real-time solutions): 56.9% of all crawler exercise, pushed nearly fully by ChatGPT
- Coaching (mannequin studying): 28.8%, cut up throughout GPTBot and different mannequin crawlers
- Discovery (content material indexing): 14.3%, distributed throughout a number of methods
- ChatGPT Person Fetch quantity: ~39.8 million visits
The tendencies are largely pushed by ChatGPT, which is answerable for practically all real-time retrieval exercise. Meaning the transfer towards answer-based crawling will not be evenly distributed, however concentrated in a single platform shaping how content material is accessed. This development could change with Google’s new Google-Agent crawler.
Market Focus In AI Crawling
AI crawler exercise is closely concentrated, with OpenAI answerable for the overwhelming majority of requests, reflecting its place as the first instrument customers depend on to search out and retrieve info.
- OpenAI: 55.8 million visits (81.0%)
- Anthropic (Claude): 11.5 million (16.6%)
- Perplexity: 1.3 million (1.8%)
- Google (Gemini): 380,000 (0.6%)
Most AI crawling exercise comes from OpenAI, which aligns with ChatGPT’s position as a major instrument for locating and retrieving info. Claude follows at a a lot smaller share, suggesting a special utilization sample, whereas the remainder of the market accounts for a minimal portion of crawler exercise.
Scale And What That Really Means
AI crawling is already working throughout a big portion of the online, reaching lots of of 1000’s of web sites and producing tens of tens of millions of requests in a single month.
Greater than half of all websites within the dataset obtained at the very least one AI crawler go to, displaying that this exercise will not be restricted to a small subset of internet sites.
- Whole websites analyzed: 858,457
- Websites with at the very least one AI crawler go to: 506,910 (59%)
- Whole AI crawler visits (Feb 2026): 68.9 million
AI crawling will not be remoted to high-profile or closely trafficked websites. It’s already widespread, with constant exercise throughout a majority of the online.
The Relationship Between Crawling and Actual Site visitors
Websites that permit AI methods to crawl them persistently present stronger engagement throughout a number of metrics.
What the information really exhibits is:
- Websites that permit AI crawling obtain considerably extra human visitors
- Increased-traffic websites usually tend to be crawled
Websites that permit crawling by AI methods obtain considerably extra human visitors, averaging 527.7 classes in comparison with 164.9 for websites that aren’t crawled. This doesn’t set up causation, nevertheless it exhibits a transparent alignment between websites that appeal to human guests and the way usually AI methods revisit them.
- Common human visitors (AI-crawled vs not): 527.7 vs 164.9 (3.2x larger)
- Common type completions: 4.17 vs 1.57 (2.7x larger)
- Averageclick-to-call: 8.62 vs 3.46 (2.5x larger)
- Websites with 10K+ classes: 90.5% crawl price
AI methods will not be discovering weak or inactive websites and lifting them up. They’re returning to websites that already appeal to human guests. For entrepreneurs, that shifts the main focus away from making an attempt to “get crawled” and towards constructing actual viewers demand, since visibility in AI methods seems to observe it.
What Correlates With Extra Crawling
The analysis in contrast websites that embody particular third-party integrations, structured options, and content material depth with these that don’t and located which of them mattered most for AI crawler exercise and referrals.
Throughout the dataset, 59% of web sites obtained at the very least one AI crawler go to in February 2026. Websites which are crawled extra usually have a tendency to mix three forms of alerts: exterior integrations, structured enterprise information, and content material depth.
1. Exterior Integrations
These integrations join the location to exterior methods that validate and distribute enterprise info.
- Yext integration: 97.1% crawl price vs ~58% with out (+38.9pp)
- Evaluations integrations: 89.8% crawl price vs 58.8% with out, 376.9 common crawler visits
Websites which are related to exterior information and assessment methods are crawled extra usually and extra often, indicating that AI methods depend on these integrations as alerts {that a} enterprise is actual, verifiable, and value revisiting.
2. Structured Web site Options And Enterprise Knowledge
These are constructed into the location and assist AI methods perceive and confirm enterprise identification.
- Google Enterprise Profile sync: 92.8% crawl price vs 58.9% with out, 415.6 common crawler visits
- Native schema: 72.3% vs 55.2% (+17.1pp), 22.3% adoption
- Dynamic pages: 69.4% vs 58.2% (+11.2pp)
- Ecommerce: 54.2% vs 59.2% (-5.0pp)
Websites that clearly outline their enterprise identification and construction their info in a machine-readable means are crawled extra usually, displaying that AI methods favor websites they will simply interpret, confirm, and extract info from.
3. Content material Depth (Quantity Of Usable Knowledge)
Websites with extra content material present extra alternatives for AI methods to retrieve, reference, and reuse info in responses.
- Websites with 50+ weblog posts: 1,373.7 common crawler visits vs 41.6 with no weblog (~33x larger)
Websites with extra content material are crawled much more usually, indicating that AI methods could return to sources that provide a bigger provide of usable info to attract from when producing solutions.
Native Enterprise Schema Completeness = Extra Crawling
This a part of the analysis focuses particularly on native enterprise schema, evaluating how the completeness of schema implementation for speaking enterprise particulars pertains to AI crawler exercise. The fields measured embody enterprise title, telephone quantity, tackle, hours, and social profiles.
- No native schema fields: 55.2% crawl price
- 10–11 accomplished schema fields: 82% crawl price
- Websites with extra full native schema present a 26.8 proportion level larger crawl price (82% vs 55.2%)
Websites that present extra full native enterprise info in structured type are crawled extra usually and obtain extra crawler visits. As extra of those fields are stuffed in, each crawl price and crawl frequency improve.
The info exhibits that clearly outlined native enterprise information makes a web site simpler for AI methods to establish, confirm, and subsequently revisit, all of the stipulations for receiving visitors from AI search.
Takeaways
AI crawling is a parallel methodology for content material discovery and the analysis exhibits clear patterns for websites which are visited by crawlers most frequently.
- AI crawling operates alongside conventional search, altering how content material is accessed and reused
- Websites with structured native alerts, deeper content material, and extra full schema are crawled extra usually
- A number of reinforcing alerts seem collectively on the identical websites, not in isolation
- The info exhibits course, not causation, however the patterns are constant
The info exhibits that websites that make it straightforward for AI crawlers to index and revisit the them are inclined to carry out higher. Apparently, websites that current clear, structured, and verifiable info, whereas persevering with to construct actual viewers demand, usually tend to be revisited by AI methods and profit from visitors generated via AI search.
Learn the analysis: Duda examine finds AI-optimized web sites drive 320% extra visitors to native companies
Featured Picture by Shutterstock/Preaapluem
