Cloudflare wants to charge Genai crawlers

Cloudflare now blocks IA crawlers who access online content without authorization or financial compensation. Moreover, The cybersecurity company wishes to leave the choice to websites: authorize access, block it or require remuneration. For example, For this last option, Cloudflare is currently testing a new “Pay Per Crawl” model.

The ia “crawlers”. For example, these bots that collect data to cause artificial intelligence models, often access the contents of websites without authorization. Similarly, The IA exploration robots collect the content (such as texts. Consequently, articles and images) with the purpose of generating answers, but without redirecting visitors to the initial source. Result: publishers. creators of content and site operators lose not only income, but no longer know how frequency or for what their content is used, explains Cloudflare as part of the announcement of a new measure to counter this phenomenon.

Cloudflare now blocks cloudflare wants charge genai crawlers IA crawlers who access content without financial authorization or compensation. Website operators can thus decide themselves if they authorize access and under what conditions, as Cloudflare writes. The type of use by AI models can also be restricted.

According to the company’s announcement. operators have three options:

• Authorize free access to IA crawlers
• Completely block access
• Require remuneration for the use of content

Pay Per Crawl: an alternative to blocking – Cloudflare wants charge genai crawlers

For this last option, Cloudflare introduces the “Pay Per Crawl” model, a payment by request, currently in the private beta phase. This system allows operators to regain total control over the monetization of their content.

According to Cloudflare, this payment model fits into the existing web infrastructure. Technically, it is based on HTTP status codes and standardized authentication mechanisms.

Here is how it works: when an IA indexing robot requests content. it indicates via the header of the request that cloudflare wants charge genai crawlers he wishes to pay and obtains access. Otherwise, it receives a “payment required” type response with the information relating to the price.

With this measure, Cloudflare continues his offensive against the suction of content by generative AI models. Last March. the popular service provider dedicated to the performance and safety of websites had already unveiled AI Labyrinth, a system aimed at trapping exploration robots that do not respect rules like robots.txt. Where the default blocking acts upstream to refuse access. AI Labyrinth works as a finer defense, diverting persistent crawlers towards a labyrinth of endless links, in order to slow down and identify abusive behaviors.