Cloudflare launches toolset to empower small publishers against AI scraper bots

Marketplace to enable small websites to licence content due next year

Cloudflare announced yesterday the launch of a series of tools to stop AI bots scraping websites without consent. It’s the first part of a larger plan.

The plans culminate with the proposed launch of a marketplace next year which will enable website owners to sell access to AI model providers.

The marketplace is the final step of Cloudflare CEO Matthew Prince’s longer game of trying to wrest control back from the AI companies who are scraping sites and return that control to publishers.

“AI will dramatically change content online, and we must all decide together what its future will look like,” Prince said yesterday.

"Content creators and website owners of all sizes deserve to own and have control over their content. If they don’t, the quality of online information will deteriorate or be locked exclusively behind paywalls.”

The first steps are the free observability tools for Cloudflare customers, called AI Audit. Website owners will get a dashboard to view analytics on just how often AI bots are crawling their sites, and where they come from.

Customers will be able to block bots entirely, or partially if they want to deal with a particular AI company. The point is – it’s the website owner in the driving seat.

With these tools Cloudflare is trying to address a problem that the industry has been trying to negotiate since LLMs really broke through, which is how smaller publishers survive when there is essentially no longer any need to visit their sites because Google or ChatGPT will provide the answers the searcher needs.

The potential plight facing smaller publishers who will see their traffic plummet has been made more acute by the fact that bigger players like Vox, Reddit, TIME, and most recently Condé Nast have, one-by-one, signed deals with OpenAI to licence their content.

Smaller publishers are left with nothing, as their content is fed into the LLMs that will reduce their traffic in the future.

Even now, heavy bot scraping on your site can increase traffic to such an extent that legitimate users experience latency and cloud bills increase.

But Cloudflare’s marketplace, due to be launched next year aims to empower small publishers to strike deals with the makers of LLMs.

The advanced analytics that will constitute part of the Cloudflare service will help customers get a handle on the metrics that are commonly used in negotiations with the likes of OpenAI. Cloudflare will also model terms of use that every content creator can add to their sites to protect the rights to their own content.

Matthew Prince said: “With Cloudflare's scale and global infrastructure, we believe we can provide the tools and set the standards to give websites, publishers, and content creators control and fair compensation for their contribution to the Internet, while still enabling AI model providers to innovate."