Cloudflare is allowing websites to block AI from scraping them and can even make bots pay for access
If you own or run a website and don't want AI bots crawling through your work to train its dataset—and who does?—Cloudflare has launched the ability to "block all AI bots in one click", with even more interesting features coming down the pipeline in the future.
According to a new Cloudflare blog, as spotted by Ars Technica, the cloud-based content delivery and management service has enabled a whole host of tools to better manage concerns around AI from the sites it supports.
The new AI functions, labelled "AI Audit", are split into three main actions. The first is that Cloudflare users can control bots' access to websites. In the blog post, it said: "Many small sites don’t have the skills or bandwidth to manually block AI bots. The ability to block all AI bots in one click puts content creators back in control."
This new system can not only do a better job at differentiating between 'good' and 'bad' uses of AI bots but also gives site owners the ability to curate which bots they block. Secondly, site owners now have a better feed of analytics from AI, like being able to spot how many are crawling a site and which uses of AI source their page.
As a result of these in-depth analytics, users can better negotiate with AI bot owners for access to the site.
Finally, though this isn't live with the rest of the tools, website owners can now directly sell access to AI crawlers. It can set prices for those accessing the site en masse, and get paid when they are cited or trained on. The first two actions can be monitored right now if you use Cloudflare, but the price setting is set to be supported at some point in the future, with the ability to sign up to a waiting list now.
What is artificial general intelligence?: We dive into the lingo of AI and what the terms actually mean.
The Cloudflare CEO and co-founder Matthew Prince makes an interesting point in the blog for the inclusion of these AI monitoring tools. As sites are increasingly worried about their work being lifted without credit, and then boosted above them in search results, Prince talks about the effects this can have on both the reader and creator.
"Content creators and website owners of all sizes deserve to own and have control over their content," Prince says. "If they don’t, the quality of online information will deteriorate or be locked exclusively behind paywalls."
If only AI bots are paying for those paywalls, it might disincentivise the land grab effects of mass generative AI theft on the online landscape. AI Audit has only just rolled out but it could offer better protections for site owners, especially when the price-setting tool arrives.