feat(agent): add curl_cffi TLS impersonation transport for Cloudflare-protected endpoints

[Midnight-Kyo]

Problem

Cloudflare's WAF in front of chatgpt.com/backend-api/codex (and other endpoints) blocks Python httpx clients via TLS fingerprinting (JA3/JA4), not just HTTP headers. From datacenter/VPS IPs, even with correct originator and User-Agent headers, requests get HTTP 403 + cf-mitigated: challenge.

This is the deeper root cause beyond the User-Agent header fixes in #24295/#24559 — identified in #30480.

Fix

This PR adds CurlCffiClient — an httpx.Client subclass that routes send() through curl_cffi with browser TLS fingerprint impersonation. It is config-gated via model.tls_impersonate in config.yaml:

model:
  tls_impersonate: "chrome124"   # or "none" (default — stock httpx)

What happens when enabled:

• _build_keepalive_http_client() returns a CurlCffiClient instead of stock httpx.Client
• The TLS handshake uses Chrome 124's JA3/JA4 fingerprint → Cloudflare allows it
• isinstance(client, httpx.Client) → True (OpenAI SDK gate passes)
• httpx auto-headers (python-httpx/X.Y.Z, accept-encoding, connection) are stripped so curl_cffi sends its own browser-impersonating headers

What is NOT affected:

• Stock behavior is completely unchanged when tls_impersonate is absent or "none"
• Only activates for chatgpt.com and api.openai.com (detected by _is_cloudflare_protected())
• Graceful fallback: if curl_cffi is not installed, logs a warning and falls back to stock httpx

Files changed

• agent/curl_cffi_transport.py (new) — CurlCffiClient class + build_curl_cffi_http_client() factory
• run_agent.py — _build_keepalive_http_client() extended with curl_cffi path, + _tls_impersonation_profile(), + _is_cloudflare_protected()

Tested on

• Ubuntu 24.04 droplet (Hetzner)
• curl_cffi 0.15.0, httpx 0.28.1
• Direct API call to chatgpt.com/backend-api/codex/responses returns HTTP 400 with zero cf-mitigated: challenge (Cloudflare bypass confirmed)
• Stock httpx from same droplet: HTTP 403 + cf-mitigated: challenge

Notes

• chrome131 is blocked — Cloudflare's JA4 database has been updated. Chrome 124, 110, Firefox 133, Safari 17, and Edge 101 all bypass successfully.
• This is scoped to the main-agent path. Auxiliary client paths (auxiliary_client.py) are not touched — those would be a follow-up.

Closes #30480

[tonydwb]

Code Review Summary

Verdict: Approved

Analysis

Correctness ✅

• Subclassing httpx.Client and overriding only send() is the correct pattern to satisfy the OpenAI SDK's isinstance check while using curl_cffi for TLS.
• The CurlCffiClient properly strips httpx auto-headers (python-httpx User-Agent, Accept-Encoding, etc.) so curl_cffi sends browser-impersonating headers instead.
• Stream mode correctly returns b'' for streamed content — the caller handles streaming separately.
• Proxy support is wired through to curl_cffi session correctly.

Security ✅

• curl_cffi is a well-known library for browser TLS impersonation — this is not a security vulnerability, it's the intended use case.
• Import errors and setup failures gracefully fall back to stock httpx.

Code Quality ✅

• Clean separation: transport class is separate from the integration logic in run_agent.py.
• The _is_cloudflare_protected() check limits this to known endpoints only.
• Config-driven via model.tls_impersonate with sensible defaults.

Recommendation
Approve — well-engineered feature with proper fallback handling.

[Midnight-Kyo]

Update on this PR:

I refreshed the branch onto the latest NousResearch/hermes-agent:main and pushed the rebased fork branch. The PR is now only the PR commits on top of current main.

While refreshing it, I also tightened the implementation to make review/CI easier:

• Added a streaming response wrapper so CurlCffiClient.send(..., stream=True) returns an httpx.Response with an httpx.SyncByteStream instead of an empty body.
• Declared curl_cffi==0.15.0 as an optional extra and lazy-installable provider dependency via tools.lazy_deps.
• Added focused tests for:
  • CurlCffiClient remaining an httpx.Client subclass.
  • stripping httpx bot-identifying auto headers before forwarding through curl_cffi.
  • removing content-encoding after curl decompression.
  • preserving streaming chunks and closing the curl response.

Verification run locally after the rebase:

• python3 -m compileall -q agent/curl_cffi_transport.py run_agent.py tools/lazy_deps.py tests/run_agent/test_curl_cffi_transport.py → OK
• python3 -m pytest tests/run_agent/test_curl_cffi_transport.py tests/tools/test_zombie_process_cleanup.py -q → 12 passed
• uv run --extra dev python -m pytest tests/run_agent/test_curl_cffi_transport.py -q → 3 passed

Motivating endpoint behavior still reproduces:

• Stock httpx against https://chatgpt.com/backend-api/codex → 403, cf-mitigated=challenge
• curl_cffi with chrome124 → 404, cf-mitigated=None
• curl_cffi with chrome131 → 403, cf-mitigated=challenge
• This PR's CurlCffiClient with chrome124 → 404, cf-mitigated=None

Default behavior remains unchanged unless model.tls_impersonate is set. If the optional dependency cannot be installed/imported, the path falls back to stock httpx.

Happy to adjust the scope further if maintainers would prefer this split differently.

[teknium1]

Thanks for the focused transport implementation and streaming coverage.

Problems

• run_agent.py's proposed _is_cloudflare_protected uses substring matching. Current endpoint routing intentionally uses base_url_host_matches (run_agent.py:4431-4445); its regression tests reject provider-looking path segments and suffix hosts (tests/test_base_url_hostname.py:41-48, :76-92). This transport must not activate for https://proxy.example.test/api.openai.com/v1 or https://api.openai.com.example/v1.
• Current main changed the keepalive-client contract to carry TLS verification (run_agent.py:3930; agent/agent_runtime_helpers.py:1689-1692, :1745-1750). The proposed curl factory does not receive that verify value, so it would bypass the existing ssl_ca_cert, ssl_verify, and HERMES_CA_BUNDLE handling covered by tests/run_agent/test_create_openai_client_ssl_verify.py:21-46.

Suggested changes

• Use base_url_host_matches and add collision tests.
• Salvage onto the current verify contract and test custom-CA and verification-disabled impersonated requests.
• Document the new model.tls_impersonate setting in the config template/docs.

This is an automated hermes-sweeper review.