Therefore,
Cloudflare denounces stealth crawling ia:
Cloudflare accuses the IA Perplexity engine of using stealth techniques to bypass the rules imposed by websites. Consequently, access protected content. Furthermore, Faced with these practices. Therefore, the company strengthens its tools to better control and monetize access by IA crawlers to online data.
Cloudflare reveals that Perplexity. Furthermore, the Genai -based search engine, bypassing the classic Crawling websites of websites thanks to unsuccessful stealth robots. Furthermore, Despite the blockages of official bots. Similarly, Perplexity would use obfuscation techniques, such as agents disguised as conventional browsers and the rotation of not affiliated IP addresses, to ignore the directives of the Robots.txt files and the firewall rules. Furthermore, Faced with this observation, Cloudflare strengthens its tools to detect, block and monetize access by IA crawlers.
In -depth tests. Therefore, detected bypassing methods – Cloudflare denounces stealth crawling ia
Cloudflare has received several complaints from customers who have explicitly prohibited from Crawler cloudflare denounces stealth crawling ia Perplexity their sites via robot.txt files and firewall rules (WAF) blocking official BOTS Perplexitybot and Perplexity-Urgee. Despite these measures, Perplexity still managed to access content. To understand this behavior. Cloudflare has created several recently acquired tests, not indexed and protected by strict guidelines prohibiting all automatic access. According to Cloudflare. by asking questions via the Perplexity interface on these areas, the AI engine provided detailed answers on their content, which indicates a bypass of the protections put in place.
The firm also observed that Perplexity not only uses its declared user-agents. but also stealth user-agents, notably imitating a Google Chrome browser on MacOS, to hide its crawling activity. In addition. the engine alternates between several IP addresses not listed in its official beach, as well as various ASN numbers (Autonomous System Number), in order to escape automated blockages. This IP rotation behavior. network identity theft has been detected on cloudflare denounces stealth crawling ia tens of thousands of areas, totaling millions of daily requests.
When the stealth crawler is blocked. Perplexity then tries to rebuild its responses from other sources, which results in information less precise and less specific to the original content, testifying to the partial efficiency of the blocking measures.
Conversely, Cloudflare stresses that other major players, such as Openai, strictly respect robot.txt files and stop their crawlers when blocked, demonstrating responsible behavior in accordance with web standards.
Cloudflare measures to fight crawling
Faced with these behaviors. Cloudflare claims to have integrated specific rules into its bots management system to detect and block this stealth crawling. Using advanced automatic learning and network analysis techniques, the company was able to identify and neutralize these crawlers. These protections are available for all its customers, including those benefiting from the free offer.
Already last month. Cloudflare had strengthened its measures to control access cloudflare denounces stealth crawling ia by IA crawlers to online content by launching an “Pay Per Crawl” model, which allows publishers to authorize, block or monetize this access.
Perplexity reaction
Faced with the accusations of Cloudflare, Perplexity published a statement refuting the allegations of stealth crawling. The company explains that the millions of requests identified by Cloudflare do not come from its own crawlers. but from a third -party service called BROWSERBASE, used occasionally for specific tasks.
Perplexity challenges the Cloudflare methodology. believing that the data has not been properly allocated and that the information provided is insufficient for an independent verification. According to the company, this confusion could hamper legitimate access to web information.
The firm underlines the need to clearly distinguish user agents – who meet specific requests without storing. driving models with the data collected – traditional bots which massively collect data. It emphasizes the importance of differentiation in BOTS management cloudflare denounces stealth crawling ia policies, in order to prevent legitimate user agents from being penalized.
Further reading: The Hubble space telescope revisits a galaxy with a mysterious “zombie” star! – An Early Bird offer is to be seized on the Revolutionary Gaming Tablet Redmagic Astra – One UI 8 update arrives on new Samsung smartphones – The Motorola Razr 60 Swarovski Edition will be launched in China on August 7, at the same time as the Moto Buds Loop – Long -term care robots in Japan: a slow revolution despite the emergency.
