AI visibility performs a vital function for SEOs, and this begins with controlling AI crawlers. If AI crawlers can’t entry your pages, you’re invisible to AI discovery engines.
On the flip aspect, unmonitored AI crawlers can overwhelm servers with extreme requests, inflicting crashes and sudden internet hosting payments.
Person-agent strings are important for controlling which AI crawlers can entry your web site, however official documentation is usually outdated, incomplete, or lacking fully. So, we curated a verified checklist of AI crawlers from our precise server logs as a helpful reference.
Each user-agent is validated towards official IP lists when out there, making certain accuracy. We are going to keep and replace this checklist to catch new crawlers and modifications to present ones.
The Full Verified AI Crawler Listing (December 2025)
| Identify | Objective | Crawl Price of SEJ (pages/hour) | Verified IP Listing | Robots.txt disallow | Full Person Agent |
|---|---|---|---|---|---|
| GPTBot | AI coaching knowledge assortment for GPT fashions (ChatGPT, GPT-4o) | 100 | Official IP Listing | Person-agent: GPTBot Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; GPTBot/1.3; +https://openai.com/gptbot) |
| ChatGPT-Person | AI agent for real-time net looking when customers work together with ChatGPT | 2400 | Official IP Listing | Person-agent: ChatGPT-Person Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); suitable; ChatGPT-Person/1.0; +https://openai.com/bot |
| OAI-SearchBot | AI search indexing for ChatGPT search options (not for coaching) | 150 | Official IP Listing | Person-agent: OAI-SearchBot Permit: / Disallow: /private-folder |
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; suitable; OAI-SearchBot/1.3; +https://openai.com/searchbot |
| ClaudeBot | AI coaching knowledge assortment for Claude fashions | 500 | Official IP Listing | Person-agent: ClaudeBot Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; ClaudeBot/1.0; +claudebot@anthropic.com) |
| Claude-Person | AI agent for real-time net entry when Claude customers browse | Not out there | Person-agent: Claude-Person Disallow: /sample-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; Claude-Person/1.0; +Claude-Person@anthropic.com) | |
| Claude-SearchBot | AI search indexing for Claude search capabilities | Not out there | Person-agent: Claude-SearchBot Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; Claude-SearchBot/1.0; +https://www.anthropic.com) | |
| Google-CloudVertexBot | AI agent for Vertex AI Agent Builder (web site house owners’ request solely) | Official IP Listing | Person-agent: Google-CloudVertexBot Permit: / Disallow: /private-folder |
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Construct/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.7390.122 Cellular Safari/537.36 (suitable; Google-CloudVertexBot; +https://cloud.google.com/enterprise-search) | |
| Google-Prolonged | Token controlling AI coaching utilization of Googlebot-crawled content material. | Person-agent: Google-Prolonged Permit: / Disallow: /private-folder |
|||
| Gemini-Deep-Analysis | AI analysis agent for Google Gemini’s Deep Analysis characteristic | Official IP Listing | Person-agent: Gemini-Deep-Analysis Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; Gemini-Deep-Analysis; +https://gemini.google/overview/deep-research/) Chrome/135.0.0.0 Safari/537.36 | |
| Gemini’s chat when a consumer asks to open a webpage | |||||
| Bingbot | Powers Bing Search and Bing Chat (Copilot) AI solutions | 1300 | Official IP Listing | Person-agent: BingBot Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36 |
| Applebot-Prolonged | Doesn’t crawl however controls how Apple makes use of Applebot knowledge. | Official IP Listing | Person-agent: Applebot-Prolonged Permit: / Disallow: /private-folder |
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Model/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot) | |
| PerplexityBot | AI search indexing for Perplexity’s reply engine | 150 | Official IP Listing | Person-agent: PerplexityBot Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) |
| Perplexity-Person | AI agent for real-time looking when Perplexity customers request data | Official IP Listing | Person-agent: Perplexity-Person Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; Perplexity-Person/1.0; +https://perplexity.ai/perplexity-user) | |
| Meta-ExternalAgent | AI coaching knowledge assortment for Meta’s LLMs (Llama, and many others.) | 1100 | Not out there | Person-agent: meta-externalagent Permit: / Disallow: /private-folder |
meta-externalagent/1.1 (+https://builders.fb.com/docs/sharing/site owners/crawler) |
| Meta-WebIndexer | Used to enhance Meta AI search. | Not out there | Person-agent: Meta-WebIndexer Permit: / Disallow: /private-folder |
meta-webindexer/1.1 (+https://builders.fb.com/docs/sharing/site owners/crawler) | |
| Bytespider | AI coaching knowledge for ByteDance’s LLMs for merchandise like TikTok | Not out there | Person-agent: Bytespider Permit: / Disallow: /private-folder |
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Cellular Safari/537.36 (suitable; Bytespider; https://zhanzhang.toutiao.com/) | |
| Amazonbot | AI coaching for Alexa and different Amazon AI providers | 1050 | Not out there | Person-agent: Amazonbot Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; Amazonbot/0.1; +https://developer.amazon.com/help/amazonbot) Chrome/119.0.6045.214 Safari/537.36 |
| DuckAssistBot | AI search indexing for DuckDuckGo search engine | 20 | Official IP Listing | Person-agent: DuckAssistBot Permit: / Disallow: /private-folder |
DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html) |
| MistralAI-Person | Mistral’s real-time quotation fetcher for “Le Chat” assistant | Not out there | Person-agent: MistralAI-Person Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; MistralAI-Person/1.0; +https://docs.mistral.ai/robots) | |
| Webz.io | Knowledge extraction and net scraping utilized by different AI coaching firms. Previously often known as Omgili. | Not out there | Person-agent: webzio Permit: / Disallow: /private-folder |
webzio (+https://webz.io/bot.html) | |
| Diffbot | Knowledge extraction and net scraping utilized by firms everywhere in the world. | Not out there | Person-agent: Diffbot Permit: / Disallow: /private-folder |
Mozilla/5.0 (Home windows; U; Home windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com) | |
| ICC-Crawler | AI and machine studying knowledge assortment | Not out there | Person-agent: ICC-Crawler Permit: / Disallow: /private-folder |
ICC-Crawler/3.0 (Mozilla-compatible; ; https://ucri.nict.go.jp/en/icccrawler.html) | |
| CCBot | Open-source net archive used as coaching knowledge by a number of AI firms | Official IP Listing | Person-agent: CCBot Permit: / Disallow: /private-folder |
CCBot/2.0 (https://commoncrawl.org/faq/) |
The user-agent strings above have all been verified towards Search Engine Journal server logs.
Widespread AI Agent Crawlers With Unidentifiable Person Agent
We’ve discovered that the next didn’t establish themselves:
- you.com.
- ChatGPT’s agent Operator.
- Bing’s Copilot chat.
- Grok.
- DeepSeek.
There is no such thing as a technique to monitor this crawler from accessing webpages aside from by figuring out the express IP.
We arrange a entice web page (e.g., /specific-page-for-you-com/) and used the on-page chat to immediate you.com to go to it, permitting us to find the corresponding go to document and IP deal with in our server logs. Beneath is the screenshot:
What About Agentic AI Browsers?
Sadly, AI browsers resembling Comet or ChatGPT’s Atlas don’t differentiate themselves within the consumer agent string, and you may’t establish them in server logs and mix with regular customers’ visits.
That is disappointing for SEOs as a result of monitoring agentic browser visits to a web site is vital for reporting POV.
How To Examine What’s Crawling Your Server
Some internet hosting firms supply a consumer interface (UI) that makes it straightforward to entry and have a look at server logs, relying on what internet hosting service you’re utilizing.
In case your internet hosting doesn’t supply this, you will get server log information (often positioned /var/log/apache2/entry.log in Linux-based servers) by way of FTP or request it out of your server help to ship it to you.
After getting the log file, you’ll be able to view and analyze it in both Google Sheets (if the file is in CSV format), Screaming Frog’s log analyzer, or, in case your log file is lower than 100 MB, you’ll be able to attempt analyzing it with Gemini AI.
How To Confirm Reliable Vs. Faux Bots
Faux crawlers can spoof official consumer brokers to bypass restrictions and scrape content material aggressively. For instance, anybody can impersonate ClaudeBot from their laptop computer and provoke crawl request from the terminal. In your server log, you will notice it as Claudebot is crawling it:
curl -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; ClaudeBot/1.0; +claudebot@anthropic.com)' https://instance.comVerification may help to avoid wasting server bandwidth and stop harvesting content material illegally. Essentially the most dependable verification technique you’ll be able to apply is checking the request IP.
Examine all IPs and scan to match if it’s one of many formally declared IPs listed above. If that’s the case, you’ll be able to enable the request; in any other case, block.
Numerous varieties of firewalls may help you with this by way of allowlist verified IPs (which permits official bot requests to cross by way of), and all different requests impersonating AI crawlers of their consumer agent strings are blocked.
For instance, in WordPress, you need to use Wordfence free plugin to allowlist official IPs from the official lists (as above) and add blocking customized guidelines as beneath:
The allowlist rule is superior, and it’ll let official crawlers cross by way of and block any impersonation request which comes from completely different IPs.
Nevertheless, please observe that it’s attainable to spoof an IP deal with, and in that case, when bot consumer agent and IPs are spoofed, you received’t be capable to block it.
Conclusion: Keep In Management Of AI Crawlers For Dependable AI Visibility
AI crawlers are actually a part of our net ecosystem, and the bots listed right here symbolize the main AI platforms at present indexing the online, though this checklist is prone to develop.
Examine your server logs often to see what’s really hitting your web site and be sure you inadvertently don’t block AI crawlers if visibility in AI engines like google is vital for your small business. In case you don’t need AI crawlers to entry your content material, block them by way of robots.txt utilizing the user-agent title.
We’ll hold this checklist up to date as new crawlers emerge and replace present ones, so we advocate you bookmark this URL, or revisit this text regularly to maintain your AI crawler checklist updated.
Extra Sources:
Featured Picture: BestForBest/Shutterstock
