Serialize upstream eval JSON directly to bytes

[xeophon]

Overview

Serialize V1 eval upstream requests and responses directly as JSON bytes, avoiding the full intermediate strings created by HTTPX's stdlib JSON helpers.

Why

HTTPX's json=body path serializes the complete body to a Python str and then encodes it to UTF-8 bytes. Large prompts, tool output, and base64 content therefore create another payload-sized object and synchronously block the event loop before the request is sent.

Likewise, Response.json() materializes a decoded JSON source before constructing the returned object, adding another full-payload allocation and synchronous stall after the provider response arrives.

Implementation

• Encode request bodies with pydantic_core.to_json(..., inf_nan_mode="null") and pass those bytes through HTTPX's content= path.
• Serialize non-finite floats as standard JSON null, avoiding invalid NaN/Infinity constants without a separate traversal or fallback serialization.
• Parse non-streaming model and auxiliary responses directly from resp.content with pydantic_core.from_json.
• Keep provider status handling, header/auth routing, dialect validation, wire size, task/connection count, and SSE response streaming unchanged.

Performance

Measured on CPython 3.13.12 with a 32 MiB JSON payload, two warmups, seven timed loops, time.perf_counter() for synchronous/event-loop stall, and tracemalloc for peak traced allocation:

Operation	Before	After	Absolute saved	Reduction
Encode stall	51.824 ms	9.240 ms	42.584 ms	82.2%
Encode peak allocation	72.00 MiB	32.01 MiB	39.99 MiB	55.5%
Decode stall	17.423 ms	1.588 ms	15.835 ms	90.9%
Decode peak allocation	64.00 MiB	32.00 MiB	32.00 MiB	50.0%

The request and response wire sizes are unchanged; the savings come from removing one full-payload transient materialization in each direction.

---

Note

Low Risk
Localized transport change in the eval relay client with equivalent wire semantics aside from NaN/Infinity serialized as null; streaming and auth paths are untouched.

Overview
EvalClient now encodes upstream POST bodies with pydantic_core.to_json and sends them through HTTPX’s content= path instead of json=, and parses non-streaming provider JSON from resp.content via from_json (model completions and relay_aux).

Request encoding sets content-type: application/json explicitly and uses inf_nan_mode="null" so non-finite floats become standard JSON null rather than invalid literals. SSE streaming, header/auth routing, dialect validation, and error handling are unchanged; only the encode/decode path is swapped for lower allocation and event-loop blocking on large payloads.

^{Reviewed by Cursor Bugbot for commit e131149. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Serialize upstream eval JSON directly to bytes using pydantic_core

• Replaces httpx Response.json() with pydantic_core.from_json on raw response bytes in get_response and relay_aux in eval.py.
• Replaces httpx's json= parameter with pydantic_core.to_json (via content=) for request serialization, ensuring NaN/Infinity float values are serialized as null.
• Sets content-type: application/json automatically on outgoing requests if not already present.
• Behavioral Change: request bodies containing NaN or Infinity now serialize as null instead of producing invalid JSON.

^{Macroscope summarized e131149.}

[macroscopeapp]

Approvability

Verdict: Approved

Small, self-contained change that switches to pydantic_core's JSON serialization for better performance and adds graceful handling of inf/nan values (converting to null). The scope is limited to the eval client's HTTP layer with clear defensive intent.

^{You can customize Macroscope's approvability policy. Learn more.}