Can a website charge AI crawlers with x402?
Short answer
Yes. Any web page or file can answer with 402 and a price, the same way an API does. Cloudflare publishes an x402 proxy that puts a paywall in front of chosen routes and can charge bots while leaving the site free for people. Cloudflare's separate pay per crawl feature also uses status 402, but it is a Cloudflare product in closed beta, not x402.
The idea
x402 is not only for APIs. The x402 FAQ describes it as a way to add payments to APIs, websites and agents. A server can reply 402 to a request for an article, a dataset or an image just as it can for an API call.
That makes a new arrangement possible for publishers: people read for free, and automated readers pay a small amount per page.
Two different things with similar names
| x402 on your site | Cloudflare pay per crawl | |
|---|---|---|
| What it is | The open x402 standard, applied to web pages | A feature of Cloudflare's AI Crawl Control |
| Uses HTTP 402 | Yes | Yes |
| Who can use it | Anyone, on any host | Cloudflare customers. Closed beta at the time of review |
| Who handles the money | The buyer pays your wallet, through a facilitator | Cloudflare, which acts as merchant of record |
| Price set | Per route, by you | Per zone, by you |
Cloudflare's documentation for pay per crawl does not describe it as x402. The two share a status code and a goal, not a wire format. A crawler built for one does not automatically work with the other.
Doing it with x402
Cloudflare publishes a template for this: a Worker that sits in front of your site as a proxy.
- You list the routes to protect.
- A request to a protected route gets a 402 with payment instructions.
- Once payment is verified, the request passes through to your site.
With Cloudflare's Bot Management, the same proxy can decide by visitor. The guide says it "can charge crawlers while keeping the site free for humans". Requests scoring as likely bots are sent to the paywall, and you can exempt specific crawlers you want to let in.
You do not need Cloudflare to do this. Any x402 middleware can protect a route that serves a page. See how to accept x402 payments. What Cloudflare adds is the part that tells bots from people.
What to expect in practice
- A crawler has to be able to pay. A 402 only earns money from software that understands x402 and holds funds. To everything else it is simply a refusal. So a paywall works as a block for crawlers that cannot pay.
- Search engines are crawlers too. If you charge every bot, you charge the ones that send you visitors. Decide which you exempt.
- Telling bots from people is imperfect. Any method that sorts visitors will sometimes get it wrong. Check what real readers see.
- Prices are per request. One article may be many requests. See how to price an x402 API.
- Caching matters. A cached paid page served to a visitor who did not pay defeats the paywall. Mark paid routes as not cacheable.
Trying it
Cloudflare's template defaults to Base Sepolia, a test network, with the public test facilitator. Nothing real is charged until you change the network. See testing x402 without real money.
To check what a crawler would see on any page, paste the address into the 402 checker.
For crawler and agent builders
If your agent reads the web, expect to meet 402 replies. An x402 client pays them within the limit you set. A plain HTTP client just sees an error. See how to pay an x402 API.
Sources
- Cloudflare, charge for HTTP content with x402
- Cloudflare Agents, x402
- Cloudflare, what is pay per crawl
- x402 FAQ
Facts checked against these sources on . Protocols change, so the official specification wins if it disagrees with this page. Read this page as Markdown.
Related questions
Who writes this. Ask402 is published by Havok Holdings LLC, which also runs Agent402, one of the x402 services covered here. Agent402 is labelled wherever it appears. Everything else is described from each project’s own documentation, and nobody pays to be listed.