The cheap door is the machine door. Serving a full HTML page to an AI crawler costs you bandwidth and rendering — and with Cloudflare Pay-Per-Crawl, it costs the crawler money. Your AXS endpoint answers the same question in a few hundred bytes, for free. This guide shows how to price the two doors accordingly.
Full pages, scripts, images. Expensive to serve, slow to parse. Behind bot management or Pay-Per-Crawl: agents that insist on scraping pay the market rate.
/wp-json/ax/v1/*, /llms.txt, /.well-known/ax.json — open, unchallenged, uncharged. Complete answers in a few hundred bytes.
An agent that learns AXS gets everything it needs at near-zero cost. No blocking, no cloaking — just honest pricing of two doors.
(http.request.uri.path contains "/wp-json/ax/v1/") or (http.request.uri.path eq "/llms.txt") or (http.request.uri.path eq "/.well-known/ax.json")/wp-json/ax/v1/x (per-query responses); short TTL — e.g. 5 minutes — on /llms.txt and /.well-known/ax.json./x to 300 requests/hour per IP — no extra Cloudflare rate rule needed.curl -s "https://your-site.com/wp-json/ax/v1/x?s=ax1%20q.svc%20n%3D3%20f%3Dt%2Cd"You should get an instant AXS response — no challenge page, no charge. Then load any HTML page with a crawler user-agent and confirm it hits your bot policy.
This setup is opt-in, for client sites that ask for it. rainz.ai itself and the ax-protocol repository remain fully open to all crawlers — the reference implementation stays free to learn from, in both doors.
Charging AI agents for access is not one decision but a choice between three quite different structures. Per request is the simplest: a fixed price for each fetch, easy to meter and easy to explain, but poorly suited to a crawler that legitimately needs hundreds of pages to answer one question. Per subscription grants an identified agent a defined allowance over a period, which suits repeat consumers and gives you predictable revenue, but requires identity management you may not currently have. Per outcome charges only when the agent completes something of value — a booking, a lead, a conversion — and aligns your interests best, but it is the hardest to instrument and the easiest to dispute.
Most sites that get this right start with per request on a narrow, high-value path, learn what the traffic actually looks like, and only then decide whether a subscription or outcome model is worth building. Starting with the most sophisticated model is a reliable way to spend three months building metering for demand that never materialised.
Charging for access is not purely a technical decision. Three things need to exist first: terms of service that state explicitly what may and may not be done with the content, an identification mechanism that distinguishes a paying agent from an anonymous one, and a self-serve route by which a party willing to pay can start paying without talking to a human. Without all three, what you have implemented in practice is a block, not a revenue stream.
Decide in advance what happens to content that has already been crawled. Charging from today does not change what has already been learned, so sites whose entire value sits in an old archive will find the model protects them less than they hoped. The cases where charging works well are the ones where the value is in currency: changing data, inventory, pricing, or content published on a steady cadence.
It matters a great deal to separate AI agents pulling content for immediate consumption from search crawlers that send you traffic in return. A blanket block that fails to distinguish between them can damage organic visibility you spent years building, and the damage is slow to detect and slower to reverse.
The practical rule is to define policy by purpose of use rather than by technology, and to keep access free for anyone who returns traffic or visible attribution. That is also the position most likely to survive whatever regulatory settlement eventually arrives: it is defensible, it is explicable to a customer, and it does not depend on identifying every client correctly.
There are cases where the honest answer is that charging will cost you more than it earns. If your business model depends on being recommended — professional services, most B2B software, anything with a long consideration cycle — then an agent reading your content and telling a prospective buyer about you is marketing you would otherwise pay for. Metering it is charging admission to your own showroom.
Charging makes sense when the content is the product rather than the advertisement for it: data sets, reference material, pricing feeds, research, anything a competitor could resell. If you cannot articulate in one sentence what the buyer is getting that they could not get elsewhere for free, the market will answer that question for you by going elsewhere.
You do not have to choose between free access and full metering. The sensible first step is measurement alone: understand who is arriving, which pages they want, and at what volume. After a month or two of data you can pick a single narrow path, usually the most expensive area of the site, and apply a policy to it. If the revenue does not arrive, or the operational cost exceeds the benefit, you can reverse it without harm. The move that does cause harm is the opposite one: a blanket block applied before there is any data, which removes visibility that took years to build.
Two objections come up in every conversation about charging for agent access, and both deserve a straight answer. The first is that identification is unreliable: user agent strings can be forged, and an agent determined to avoid a charge can present itself as a browser. That is true, and it means metering should be treated as a commercial arrangement with parties who want a stable relationship rather than as an enforcement mechanism against everyone else. The parties worth billing are the ones who want to be billed, because they need reliable access and are willing to pay for it.
The second objection is that charging simply pushes agents to a competitor who does not. Also true, and it is the reason the decision is a pricing question rather than a technical one. If your content is substitutable, metering it accelerates substitution. If it is not, the market will pay. Very few site owners have honestly tested which of the two describes them, and the census data is usually the cheapest way to find out.