Release intercepted request bodies before inference

[xeophon]

Overview

Release aiohttp's cached intercepted request bytes immediately after JSON parsing, instead of retaining the serialized body for the duration of model inference.

Why

request.read() is the correct ingestion path because it enforces aiohttp's configured request-size limit, but it also stores the complete body in request._read_bytes. The handler's raw local aliases that allocation, while parsing creates a separate Python object graph. Because the async handler remains suspended during inference, the serialized bytes otherwise stay live alongside the parsed request for the longest part of the request lifecycle.

For large prompts or tool results, this avoidably doubles the live payload representation and compounds across concurrent rollouts.

Implementation

After either the primary pydantic_core.from_json parse or the stdlib fallback succeeds, the handler:

• clears aiohttp's cached body with request._read_bytes = None;
• deletes the local raw alias before entering dialect handling or inference.

The code deliberately keeps request.read(), preserving the existing size guard. Downstream code operates on the parsed body and does not reread the request stream.

Performance

A PEP 723 microbenchmark ran five interleaved old/new subprocesses with a 64 MiB JSON content value (67,108,878 wire bytes), using a real aiohttp Request.read() and pydantic_core.from_json.

Measurement	Previous	New	Change
Retained Python allocations	128.000694 MiB	64.000649 MiB	-64.000045 MiB (-50.0%)
Retained cached wire bytes	67,108,878 B	0 B	-67,108,878 B (-100%)
Live logical payload representations	134,217,742 B	67,108,864 B	-67,108,878 B (-50.0%)
Peak Python allocations	128.001060 MiB	128.000834 MiB	effectively unchanged
Median wall time	5.478 ms	5.489 ms	-0.011 ms saved (-0.20%; noise-scale)
Median event-loop stall	5.537 ms	5.550 ms	-0.013 ms saved (-0.23%; noise-scale)
Bytes copied by guarded request.read()	67,108,878 B	67,108,878 B	unchanged

This is a retained-memory optimization rather than a throughput optimization. Peak parsing memory remains essentially unchanged because the raw and parsed forms must briefly coexist; the benefit is that the raw form no longer survives the potentially long inference wait.

---

Note

Low Risk
Small lifecycle change in one handler after successful parse; relies on aiohttp’s _read_bytes cache but does not alter auth, inference, or response paths.

Overview
Frees intercepted chat-completion request memory sooner by dropping aiohttp’s cached wire body and the local raw buffer immediately after JSON parsing in handle_request, instead of keeping both alongside the parsed body through model inference and user-simulator loops.

The handler still uses request.read() so the existing 1 GiB client_max_size guard is unchanged; only the parsed dict is used downstream.

^{Reviewed by Cursor Bugbot for commit bc81f93. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Release intercepted request body from memory before inference

In InterceptionServer.handle_request in server.py, after parsing the raw request body, the handler now sets request._read_bytes to None and deletes the local raw reference. This reduces memory retention of the wire body during inference processing.

^{Macroscope summarized bc81f93.}

[macroscopeapp]

Approvability

Verdict: Approved

Simple memory optimization that releases raw request bytes after JSON parsing completes. The parsed body is retained; only the redundant raw bytes are freed earlier to reduce memory pressure during model inference.

^{You can customize Macroscope's approvability policy. Learn more.}