After spending over 15 years creating content online, I have seen a lot change. From basic blogs to advanced AI, the internet has evolved fast. But there has always been one issue: creators and publishers do the hard work, and others, especially AI companies, get benefit from it without giving anything back to creators or publishers.
Now, Cloudflare is finally doing something about it.
The company has announced a new feature called Pay Per Crawl. It will let content creators and website owners charge AI crawlers for access. So, if an AI wants to use your content, it has to pay first.
Until now, we had only two options: either let all bots in for free or block them completely.
I have often struggled with this myself. Blocking crawlers affects SEO. Keeping content open means anyone, even big AI companies, can freely scrape years of work without asking. Now, with Pay Per Crawl, there is a third option. You can allow access, but only if the crawler is willing to pay.
How It Works
The feature integrates with existing infrastructure to create a framework for paid content access. Cloudflare is bringing back an old HTTP status code – 402 Payment Required – and finally putting it to the right use. If a crawler tries to access your content, it will either pay and get the content (HTTP 200 OK), or get a 402 error saying payment is required.
The system uses secure headers to make sure only verified crawlers can access the content. And yes, Cloudflare handles billing. You just set your price and decide which bots to allow, block, or charge.
Cloudflare will act as the Merchant of Record, which means it handles all billing and payment processing between the AI crawler and the content owner.
Every time an AI bot makes a request and agrees to pay, that request is logged. If the content is served successfully, the payment is recorded. Cloudflare collects the payments from the crawler and then pays out the earnings to the content owner.
There is no need for creators to set up their own payment systems. Everything is managed through Cloudflare’s platform, and all billing details are tracked inside your Cloudflare account.
AI tools have grown by using content we all have published online. Articles, blog posts, forums, reviews, they used everything without asking. No credit. No share of profits. Just silence. With Pay Per Crawl, that changes. It gives creators a way to protect their work and even earn from it.
This does not mean every AI bot is bad. But if someone is using our content to train a tool or power a product, paying a small fee should not be a problem.
This is just the beginning. In the future, we might see smart agents or AI tools that can pay on the fly to access the best content, whether it is research, news, or reviews.
Imagine asking your AI assistant to find the best DSLR camera reviews, and it pays a few rupees to access the most trusted content. And that money goes directly to the creators who wrote those reviews. That is the kind of future we need. Creators must be rewarded for their hard work.
Cloudflare’s Pay Per Crawl is a big move to give control back to content creators. It is still in private beta, but it is a step in the right direction. I hope more platforms follow this model.
If you run a blog or a news site, it is time to think about how your content is being used — and how you can protect it.
