Cloudflare, an internet infrastructure provider, claims to have identified questionable indexing practices on the part of Perplexity, to feed what it calls “its conversational response engine”. According to a rapport Published on its official blog, the start-up would use boots pretending to google Chrome on MacOS, in order to access content explicitly prohibited to its declared crawlers.
Cloudflare says he has received customer complaints who, although having specifically blocked the Perplexity crawlers via files robots.txt Or firewall rules (WAF), had found that the company always had access to their content.
He decided to carry out a series of tests and, for this purpose, created new sites and implemented the same access restrictions for the official Bots of Perplexity. Just saved, these sites were not indexed by any search engine. Despite this, Perplexity was able to provide him with detailed information concerning the accommodated content.
Cloudflare indicates that it has observed that when Perplexitybot and Perplexity-Use were blocked, the platform adapted its methods: modification of the user agent (identification chain sent to indicate to the website which it is), rotation of IP addresses and ASN change (identification number of an autonomous system) to bypass the blocking measurements.
The company specifies that the IP addresses used did not appear in the beach officially communicated by Perplexity, adding that “This activity was observed on tens of thousands of areas and millions of requests a day”.
Recalling that the operation of the web is based on confidence, so she decided to withdraw Perplexity from her list of checked bots and reinforced her protections to block stealth crawlers.
The latter denies the accusations of furtive collection or bypassing robots.txtand claims that, unlike conventional crawlers, his agents operate solely at the request of the user, without indexing or data storage. According to her, Cloudflare’s analysis is based on technical confusion between its various services and a deep misunderstanding of the functioning of AI agents, questioning its ability to judge legitimate traffic.
