[Achuthantm]

Fixes #678.

Streaming requests kept already-delivered chunks until completion, so API-server
memory grew with the total output. Drain the pending list under request_lock
by transferring ownership to the consumer, allowing consumed payloads to be
released before the request ends.

Track per-modality counts and byte totals separately to preserve profiling.
Add regression coverage for reclamation, final flushing, non-streaming
collection, profiling, and errors.

How was it tested?

• ruff check ., git diff --check, and local Mac lint and cpu-smoke:
  passed.
• API regression suites: 192 passed on each of torch 2.12.1/CUDA 13.2
  and torch 2.9.1/CUDA 13.0. The new tests confirm consumed chunks are released
  before request completion and catch the original bug.
• Linux CPU Core coverage: 6,164 passed, 646 skipped. A pre-existing xdist
  collection-order issue required running the affected module serially;
  it passed. The TP simulation also passed separately.
• Real-weights LingBot-World streaming: one A40, torch 2.9.1+cu130,
  1,021 frames / 64 chunks per request, two fresh-process runs per arm.
  Median sampled peak API RSS fell from 1,358.6 to 1,139.8 MiB.
  Between chunks 4 and 63, baseline RSS grew 221.8–224.7 MiB; the fix
  showed 0.0 MiB growth. Queue samples confirmed consumed history was
  released, and full-stream hashes and profiling totals matched.
• Fresh-server ABBA e2e regression matrix: Ran base–head–head–base on one A40 with torch 2.9.1+cu128 for Orpheus streaming TTS, Pi0.5 (multi-stage graph), and LingBot-World video streaming. All 60 requests completed successfully (48 measured + 12 warmups). Profiling output counts and byte totals matched across base/head.

Model	Measure	Base vs base	Base vs head
Orpheus	Median waveform correlation (higher = more similar)	0.99761	0.99763
Pi0.5	Maximum action difference (lower = closer)	0.004804	0.004804
LingBot-World	Full-stream output comparison	Byte-identical	Byte-identical

The API, Linux CPU Core, and A40 results were recorded on 2026-10-22 for the
changes in this PR. Slow consumers can still accumulate unread output;
these measurements do not establish a hard memory cap or a speedup.

Checklist

• ruff check . passes
• Added regression tests

AI assistance was used in developing this change.

[arun-elr]

Thanks @Achuthantm for picking up #678! Nice to see the LingBot RSS growth between chunks 4 and 63 go from about 222 MiB to 0.

I ran it on the PR merged into current main and found nothing blocking:

• Drain vs the producer, plus abort, timeout, an error after some chunks, an engine failure and conductor death mid-stream, all behave as on main (CPU probes driving the real message loop).
• Non-streaming collection still returns every chunk, and the --log-stats output totals match main, including chunks that land after an error.
• The Rust frontend's bridge also reads through iter_result_chunks, so it gets the fix too. With a real APIServer behind both the Python app and mstar-server, a 120-chunk stream held at most 2 unsent chunks (main held all 120) for text, audio, image and video_frame.
• CPU Core: 5,995 passed. 3 signal-timing tests flaked under xdist and pass on their own on both main and this branch.

One thing to line up:

• #593 (full duplex 3/6) rewrites the same drain lines and counts profile outputs at delivery instead, so whichever lands second will conflict. Counting on arrival as done here keeps main's numbers, so this version is probably the one to keep in the rebase.

Since this changes how every streamed request hands chunks to the frontend, could you also run the e2e regression matrix base vs head with alternating fresh servers (ABBA): Orpheus streaming TTS, Pi0.5 (multi-stage graph) and LingBot-World video streaming, and paste the summary into the PR? I swapped Z-Image for LingBot because /v1/images/generations submits with streaming=False and never reaches the new drain, and LingBot is the #678 case anyway.

[Achuthantm]

@arun-elr Thanks for the thorough review! I ran the requested fresh-server ABBA (base-head-head-base) e2e matrix on one A40 with torch 2.9.1+cu128.

All four fresh-server runs completed successfully for each model: Orpheus, Pi0.5, and LingBot-World.

All 60 requests completed (48 measured + 12 warmups). Profiling output counts and byte totals matched across base/head.

LingBot outputs were byte-identical. Orpheus and Pi0.5 showed output variation comparable to repeated-base runs.

Model	Measure	Base vs base	Base vs head
Orpheus	Median waveform correlation (higher = more similar)	0.99761	0.99763
Pi0.5	Maximum action difference (lower = closer)	0.004804	0.004804

Noted the #593 overlap: I’ll preserve arrival-side accounting when resolving the conflict.

[arun-elr]

LGTM. Thanks for running the ABBA matrix: base vs head sits inside base-vs-base noise on Orpheus and Pi0.5, and LingBot is byte-identical. Could you paste that table into the PR description so it's there for the maintainer review?

[Achuthantm]

Added it to the PR description, thanks

[dghosef]

lgtm

[Achuthantm]

@dghosef Thanks for the review! The CI workflow is still awaiting maintainer approval. Could you approve it when you get a chance?

[vasilevklart]

Thanks @Achuthantm