Perplexity revealed a response to Cloudflare’s claims that it disrespects robots.txt and engages in stealth crawling. Perplexity argues that Cloudflare is mischaracterizing AI Assistants as net crawlers, saying that they shouldn’t be topic to the identical restrictions since they’re user-initiated assistants.
Perplexity AI Assistants Fetch On Demand
In accordance with Perplexity, its system doesn’t retailer or index content material forward of time. As an alternative, it fetches webpages solely in response to particular person questions. For instance, when a person asks for current restaurant evaluations, the assistant retrieves and summarizes related content material on demand. This, the corporate says, contrasts with how conventional crawlers function, systematically indexing huge parts of the net with out regard to speedy person intent.
Perplexity in contrast this on-demand fetching to Google’s user-triggered fetches. Though that’s not an apples-to-apples comparability as a result of Google’s user-triggered fetches are within the service of studying textual content aloud or web site verification, it’s nonetheless an instance of user-triggered fetching that bypasses robots.txt restrictions.
In the identical approach, Perplexity argues that its AI operates as an extension of a person’s request, not as an autonomous bot crawling indiscriminately. The corporate states that it doesn’t retain or use the fetched content material for coaching its fashions.
Criticizes Cloudflare’s Infrastructure
Perplexity additionally criticized Cloudflare’s infrastructure for failing to tell apart between malicious scraping and bonafide, user-initiated visitors, suggesting that Cloudflare’s strategy to bot administration dangers overblocking providers which are appearing responsibly. Perplexity argues {that a} platform’s incapacity to distinguish between useful AI assistants and dangerous bots causes misclassification of authentic net visitors.
Perplexity makes a robust case for the declare that Cloudflare is obstructing authentic bot visitors and says that Cloudflare’s determination to dam its visitors was based mostly on a misunderstanding of how its know-how works.
Learn Perplexity’s response:
Brokers or Bots? Making Sense of AI on the Open Internet
