AI Crawl Control: Pay Per Crawl customization features

[CameronWhiteside]

Summary

This PR adds documentation for new Pay Per Crawl features that enable both AI crawler operators and site owners to have more control and flexibility:

For AI crawler operators:

• Discovery API: New authenticated endpoint to programmatically discover domains participating in Pay Per Crawl, eliminating trial-and-error discovery
• Payment header signature requirement: Enhanced security requiring payment headers to be included in Web Bot Auth signatures
• crawler-error header: new standardized error codes for robust programmatic error handling for crawlers attempting to access paid content

For site owners:

• Custom pricing (in-band pricing): Set dynamic, content-specific prices via origin headers or Cloudflare Workers based on URL paths, bot characteristics, or request properties
• Configure free pages: Use Configuration Rules to selectively disable Pay Per Crawl for specific URI patterns (homepages, navigation, discovery pages, etc.)

Documentation updates:

• New page: discover-payable-content.mdx - Complete Discovery API documentation with authentication, request/response schemas, and examples
• New page: advanced-configuration.mdx - Consolidated guide for custom pricing and URI pattern configuration
• New changelog: 2025-11-18-pay-per-crawl-enhancements.mdx
• Updated existing pages to reflect new workflows
• Added glossary entry for "in-band pricing"

The documentation emphasizes practical implementation with Worker examples, caching strategies, and clear setup steps for dashboard configuration. More to come very soon on more dash-UI native configuration flexibility

Documentation checklist

• Is there a changelog entry (guidelines)? Added 2025-11-18-pay-per-crawl-enhancements.mdx
• The change adheres to the documentation style guide.
• If a larger change - such as adding a new page- an issue has been opened in relation to any incorrect or out of date information that this PR fixes.
• Files which have changed name or location have been allocated redirects. (N/A - no files renamed or moved)

[Oxyjun]

Just two quick suggestions:

• Should we put the error codes in a separate page?
• The free paths - we mention it's free "by default". Aren't these always free?