cloudflare · CameronWhiteside · Nov 19, 2025 · Nov 20, 2025
@@ -0,0 +1,51 @@
+---
+title: Pay Per Crawl (Private beta) - Discovery API, custom pricing, and advanced configuration
+description: Pay Per Crawl adds Discovery API for crawler content discovery, custom in-band pricing for dynamic content pricing, granular URI-based configuration, and enhanced error handling with standardized crawler-error codes.
+date: 2025-11-18
+---
+
+Pay Per Crawl is introducing enhancements for both AI crawler operators and site owners, focusing on programmatic discovery, flexible pricing models, and granular configuration control.
+
+## For AI crawler operators
+
+### Discovery API
+
+A new authenticated API endpoint allows verified crawlers to programmatically discover domains participating in Pay Per Crawl. Crawlers can use this to build optimized crawl queues, cache domain lists, and identify new participating sites. This eliminates the need to discover payable content through trial requests.
+
+The API endpoint is `GET https://crawlers-api.ai-audit.cfdata.org/charged_zones` and requires Web Bot Auth authentication. Refer to [Discover payable content](/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/discover-payable-content/) for authentication steps, request parameters, and response schema.
+
+### Payment header signature requirement
+
+Payment headers (`crawler-exact-price` or `crawler-max-price`) must now be included in the Web Bot Auth `signature-input` header components. This security enhancement prevents payment header tampering, ensures authenticated payment intent, validates crawler identity with payment commitment, and protects against replay attacks with modified pricing. Crawlers must add their payment header to the list of signed components when [constructing the signature-input header](/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages/#22-sign-your-request-with-web-bot-auth).
+
+### New `crawler-error` header
+
+Pay Per Crawl error responses now include a new `crawler-error` header with 11 specific [error codes](/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages/#error-code-reference) for programmatic handling. Error response bodies remain unchanged for compatibility. These codes enable robust error handling, automated retry logic, and accurate spending tracking.
+
+## For site owners
+
+### Custom pricing support
+
+Site owners can now set dynamic, content-specific prices by returning a `crawler-price` header in-band from their origin or a Cloudflare Worker:
+
+1. Enable custom pricing in **AI Crawl Control** > **Settings**
+2. Check for `cf-pay-per-crawl: protocol=cloudflare, pricing=in-band` header on origin requests
+3. Return `crawler-price` headers based on URL paths, bot characteristics, or request properties
+
+[Custom pricing](/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/advanced-configuration/#custom-pricing) enables sophisticated pricing strategies like premium content tiers, time-based pricing, or bot-specific rates without changing your default zone price and works with Cloudflare Cache to reduce origin load while maintaining dynamic pricing.
+
+### Configure free pages
+
+Site owners can now offer free access to specific pages like homepages, navigation, or discovery pages while charging for other content.
+
+1. Create a [Configuration Rule](/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/advanced-configuration/#disable-pay-per-crawl-by-uri-pattern) in **Rules** > **Configuration Rules**.
+2. Set your URI pattern using wildcard, exact, or prefix matching on the **URI Full** field.
+3. Activate the **Disable Pay Per Crawl** setting. When disabled for a URI pattern, crawler requests pass through without blocking or charging.
+
+Note that `/robots.txt`, `/sitemap.xml`, `/security.txt`, `/.well-known/security.txt`, and `/crawlers.json` will always be accessible to crawlers for free.
+
+## Get started
+
+**AI crawler operators**: [Discover payable content](/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/discover-payable-content/) | [Crawl pages](/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages/)
+
+**Site owners**: [Advanced configuration](/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/advanced-configuration/)
@@ -19,6 +19,8 @@ No. Pay per crawl allows you to configure different actions (Block, Charge, or A
 
 Yes. Every time your AI crawler accesses content on a website protected with pay per crawl, it will incur the cost set by the site owner. You should implement mechanisms within your crawler to track expenditure and enforce any spending limits you want to set.
 
+Note that certain paths are always excluded from charging: `/robots.txt`, `/sitemap.xml`, `/security.txt`, `/.well-known/security.txt`, and `/crawlers.json`.
+
 ### Am I charged for error responses?
 
 No. Charging events are only triggered for successful HTTP response codes. Error responses are not billed, even if you have sent the `crawler-exact-price` or `crawler-max-price` headers.

@@ -4,16 +4,17 @@ pcx_content_type: concept
 tags:
   - Stripe
 sidebar:
-  order: 4
+  order: 5
 ---
 
 import { Steps, DashButton } from "~/components";
 
 ```mermaid
 graph LR
-A[Set up your<br>Cloudflare Account] --> B[Connect to<br>Stripe]:::highlight
-B --> C[Verify your<br>AI crawler]
-C --> D[Crawl pages]
+A[Set up your<br>Cloudflare Account] --> B[Verify your<br>AI crawler]
+B --> C[Discover<br>payable content]
+C --> D[Connect to<br>Stripe]:::highlight
+D --> E[Crawl pages]
 classDef highlight fill:#F6821F,color:white
 ```
 

@@ -9,9 +9,10 @@ import { Steps } from "~/components";
 
 ```mermaid
 graph LR
-A[Set up your<br>Cloudflare Account] --> B[Connect to<br>Stripe]
-B --> C[Verify your<br>AI crawler]
-C --> D[Crawl pages]:::highlight
+A[Set up your<br>Cloudflare Account] --> B[Verify your<br>AI crawler]
+B --> C[Discover<br>payable content]
+C --> D[Connect to<br>Stripe]
+D --> E[Crawl pages]:::highlight
 classDef highlight fill:#F6821F,color:white
 ```
 
@@ -31,46 +32,49 @@ To access this content, the AI crawler must provide headers for paid access.
 
 ## 2. Access paid content
 
-### 2.1. Include WBA headers
-
-Include Web Bot Auth headers by following the steps [Sign your requests](/bots/reference/bot-verification/web-bot-auth/#4-after-verification-sign-your-requests)
-
-### 2.2. Include payment headers
+### 2.1. Include payment headers
 
 Your AI crawler can specify the price it is willing to pay by providing one of two headers:
 
 - `crawler-exact-price`: This is the exact price the AI crawler is configured to pay for access. This value should exactly match the price set in the response header `crawler-price`. Include this header in your second request if your AI crawler first received a HTTP status code 402, and you wish to access the content by paying the `crawler-price`.
 - `crawler-max-price`: This is the maximum price the AI crawler is configured to pay for access on any content. Use this option if you wish to access any pay per crawl content with a crawl price equal or lower than the maximum price. If a page's `crawler-price` is higher than your `crawler-max-price`, your AI crawler will receive a HTTP 402 response.
 
-:::note
-Note that if a content owner has explicitly blocked your AI crawler (instead of charging your crawler), you cannot access their content, even if you provide `crawler-exact-price` or `crawler-max-price` headers.
+### 2.2. Sign your request with Web Bot Auth
+
+Include Web Bot Auth headers by following the steps in [Sign your requests](/bots/reference/bot-verification/web-bot-auth/#4-after-verification-sign-your-requests).
+
+:::caution[Important]
+Payment headers (`crawler-exact-price` or `crawler-max-price`) **must be included in the `signature-input` header components**. When [choosing components to sign](/bots/reference/bot-verification/web-bot-auth/#41-choose-a-set-of-components-to-sign), add your payment header to the list of signed components for successful payment processing.
 :::
 
 ### 2.3. Review response headers
 
 When you specify a header to indicate payment, you may receive one of two responses:
 
-#### Successful HTTP/2 200 response
+#### Successful HTTP 200 response
 
 The value of the `crawler-charged` header indicates the exact amount that will be billed to your Cloudflare account for the request.
 
 ```txt
-HTTP/2 200
+HTTP 200
 date: Fri, 06 Jun 2025 08:42:38 GMT
 crawler-charged: USD 0.01
 ```
 
-#### Unsuccessful HTTP/2 402 response
+#### Unsuccessful response
 
-If the request is unsuccessful due to an incorrect price header, you will receive a response with the `crawler-price` header. Your request header values must be adjusted to match the `crawler-price`.
+If the request is unsuccessful, you will receive an error response with a `crawler-error` header indicating the specific issue.
 
 ```txt
 HTTP/2 402
 date: Fri, 06 Jun 2025 08:42:38 GMT
 content-type: text/plain; charset=utf-8
 crawler-price: USD 0.01
+crawler-error: InvalidCrawlerExactPrice
 ```
 
+Refer to the [Error code reference](#error-code-reference) section below for a complete list of error codes and troubleshooting guidance.
+
 ## 3. Track your spending
 
 When you successfully access pay per crawl content (response with HTTP status code 200), the response will include `crawler-charged`. For example:
@@ -81,8 +85,27 @@ crawler-charged: USD 0.01
 
 Cloudflare strongly recommends tracking and saving these values to keep an accurate record of the bill your AI crawler has accrued.
 
+## Error code reference
+
+All Pay Per Crawl error responses include a `crawler-error` header with a specific error code. The following table provides a complete reference of all possible error codes:
+
+| Error Code                 | HTTP Status | What to do                                                                                                                                                     |
+| -------------------------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `CrawlerForbidden`         | 403         | The site owner has blocked your crawler. You cannot access this content.                                                                                       |
+| `StrongAuthRequired`       | 400         | Include valid Web Bot Auth headers with strong authentication in your request.                                                                                 |
+| `InvalidSignature`         | 400         | Include both `signature-input` and `signature` headers in your request. Refer to [Web Bot Auth documentation](/bots/reference/bot-verification/web-bot-auth/). |
+| `InvalidCrawlerPriceValue` | 400         | Check that your `crawler-exact-price` or `crawler-max-price` header value is properly formatted (for example, `USD 0.01`).                                     |
+| `MissingCrawlerPrice`      | 402         | Include either `crawler-exact-price` or `crawler-max-price` header in your request.                                                                            |
+| `PaymentFailed`            | 403         | Verify your crawler is enabled and payment processing is configured correctly. Contact Cloudflare support if the issue persists.                               |
+| `InvalidCrawlerExactPrice` | 402         | Update your `crawler-exact-price` to match the `crawler-price` value from the response header.                                                                 |
+| `InvalidCrawlerMaxPrice`   | 402         | Increase your `crawler-max-price` to meet or exceed the `crawler-price` value from the response header.                                                        |
+| `ConflictingPriceHeaders`  | 400         | Use only one price header per request. Remove either `crawler-max-price` or `crawler-exact-price`.                                                             |
+| `InvalidContentPrice`      | 502         | The origin returned an invalid price. This is a site owner configuration issue. Try again later or contact the site owner.                                     |
+| `InternalError`            | 500         | A server error occurred. Retry your request with exponential backoff.                                                                                          |
+
 ## Additional resources
 
 You may wish to refer to the following resources.
 
-- [Pay Per Crawl FAQs](/ai-crawl-control/features/pay-per-crawl/faq).
+- [Discover payable content](/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/discover-payable-content/)
+- [Pay Per Crawl FAQs](/ai-crawl-control/features/pay-per-crawl/faq/)
@@ -0,0 +1,113 @@
+---
+title: Discover payable content
+pcx_content_type: how-to
+sidebar:
+  order: 4
+---
+
+import { Steps } from "~/components";
+
+```mermaid
+graph LR
+A[Set up your<br>Cloudflare Account] --> B[Verify your<br>AI crawler]
+B --> C[Discover<br>payable content]:::highlight
+C --> D[Connect to<br>Stripe]
+D --> E[Crawl pages]
+classDef highlight fill:#F6821F,color:white
+```
+
+The Pay Per Crawl Discovery API allows verified AI crawlers to discover which domains offer paid content access. This enables your crawler to proactively identify sites participating in Pay Per Crawl before making crawl requests.
+
+## Prerequisites
+
+Before using the Pay Per Crawl Discovery API, you must:
+
+- [Set up your Cloudflare account](/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/set-up-cloudflare-account/)
+- [Verify your AI crawler](/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/verify-ai-crawler/)
+
+## Authenticate with Web Bot Auth
+
+All requests to the Discovery API must be authenticated using HTTP message signatures with Web Bot Auth headers. This ensures that only verified crawlers can access the list of participating domains.
+
+<Steps>
+
+1. Generate your Web Bot Auth signature following the steps in [Sign your requests](/bots/reference/bot-verification/web-bot-auth/#4-after-verification-sign-your-requests).
+
+2. Construct the required headers as described in [Construct the required headers](/bots/reference/bot-verification/web-bot-auth/#43-construct-the-required-headers):
+   - `Signature`: The cryptographic signature of the request
+   - `Signature-Input`: The signature metadata and parameters
+   - `Signature-Agent`: Information about the signing agent
+
+</Steps>
+
+## Discover participating domains
+
+### API endpoint
+
+```txt
+GET https://crawlers-api.ai-audit.cfdata.org/charged_zones
+```
+
+### Request parameters
+
+- `cursor` (optional): Cursor returned from a previous call for pagination
+- `limit` (optional): Number of results to return per request
+
+### Request headers
+
+Include the HTTP message signature headers generated using Web Bot Auth:
+
+```txt
+Signature: <your-signature>
+Signature-Input: <signature-metadata>
+Signature-Agent: <agent-information>
+```
+
+### Example request
+
+```sh
+curl -X GET "https://crawlers-api.ai-audit.cfdata.org/charged_zones?limit=50" \
+  -H "Signature: <your-signature>" \
+  -H "Signature-Input: <signature-metadata>" \
+  -H "Signature-Agent: <agent-information>"
+```
+
+### Response format
+
+The API returns a list of zones (domains) that have Pay Per Crawl enabled and are accepting payments from your crawler.
+
+```json
+{
+	"result": {
+		"zones": [
+			{
+				"domain": "example.com"
+			},
+			{
+				"domain": "news-site.com"
+			}
+		]
+	},
+	"success": true,
+	"errors": [],
+	"messages": []
+}
+```
+
+### Response fields
+
+- `result.zones`: Array of zone objects containing domains with Pay Per Crawl enabled
+- `result.zones[].domain`: The domain name offering Pay Per Crawl content
+- `success`: Boolean indicating whether the request was successful
+- `errors`: Array of error messages (empty if successful)
+- `messages`: Array of informational messages
+
+## Use discovery data
+
+The Discovery API returns domains where site owners have specifically configured your crawler to be charged for content access. If a domain does not appear in the response, the site owner has not enabled Pay Per Crawl charging for your crawler. Site owners may also block or allow your crawler through WAF rules or set directives in their robots.txt file, which you should check and respect.
+
+Cache discovery results locally and refresh periodically to stay up-to-date with domains joining or leaving Pay Per Crawl.
+
+## Additional resources
+
+- [Crawl pages](/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages/)
@@ -9,9 +9,10 @@ import { Steps } from "~/components";
 
 ```mermaid
 graph LR
-A[Set up your<br>Cloudflare Account]:::highlight --> B[Connect to<br>Stripe]
-B --> C[Verify your<br>AI crawler]
-C --> D[Crawl pages]
+A[Set up your<br>Cloudflare Account]:::highlight --> B[Verify your<br>AI crawler]
+B --> C[Discover<br>payable content]
+C --> D[Connect to<br>Stripe]
+D --> E[Crawl pages]
 classDef highlight fill:#F6821F,color:white
 ```
 

@@ -2,16 +2,17 @@
 title: Verify your AI crawler
 pcx_content_type: concept
 sidebar:
-  order: 10
+  order: 3
 ---
 
 import { Steps, DashButton } from "~/components";
 
 ```mermaid
 graph LR
-A[Set up your<br>Cloudflare Account] --> B[Connect to<br>Stripe]
-B --> C[Verify your<br>AI crawler]:::highlight
-C --> D[Crawl pages]
+A[Set up your<br>Cloudflare Account] --> B[Verify your<br>AI crawler]:::highlight
+B --> C[Discover<br>payable content]
+C --> D[Connect to<br>Stripe]
+D --> E[Crawl pages]
 classDef highlight fill:#F6821F,color:white
 ```
 
@@ -54,6 +55,12 @@ Submit a form to add your AI crawler to Cloudflare's list of verified bots.
    <DashButton url="/?to=/:account/configurations" />
 
 2. Go to the **Bot Submission Form** tab.
-3. Fill out the form.
+
+3. Fill out the form with the following required information:
+   - **Select bot type**: Choose either **Verified Bot** or **Signed Agent**.
+   - **Verification Method**: Select **Request Signature**.
+   - **User-Agents header values**: Provide the User-Agent string(s) your bot uses.
+   - **User-Agents Match Pattern**: Provide substring patterns that match your User-Agent (for example, `GoogleBot | GoogleScraper`).
+
 4. Select **Submit**.
 </Steps>