[hnyls2002]

• Per-request output values that change only at a few points in a request's life (prompt_tokens, cached_tokens, cached_tokens_details, the retraction count, the image/audio/video token counts, and the scheduler time stats) now travel together as one RequestOutputStats in request_stats. A request sends it on its first output, on its first output after each retraction, and on its finished row; other rows carry None, and the field is None when no row carries one.
• Req.take_request_stats_due() decides: it compares retraction_count with the retraction count of the last stats sent (Req.request_stats_retraction_count) and records the send. No other code marks anything, so a new output path cannot skip a send. A value that changes between outputs other than across a retraction shows only on the finished output; the cases today are listed below.
• The tokenizer manager keeps the latest RequestOutputStats in ReqState.request_stats (ReqState.update_request_stats) and builds meta_info, metrics and span attributes from it. A row that arrives before its request's stats, which only a late row of an aborted request whose rid was reused can do, is dropped with a warning, as a row for a missing state already is.
• retraction_counts, cached_tokens_details, time_stats and the media token counts are removed as per-row lists from BatchTokenIDOutput, BatchStrOutput and BatchEmbeddingOutput. Rust server mode still fills prompt_tokens, cached_tokens and the media counts per step, which push_generation reads.

The finished output's meta_info, all metrics and the finished-request span attributes are the same as before, and so is every output of a request that is never retracted, is not dLLM, and sees no HiCache storage attach or detach. For those three cases, outputs before the finished one keep the stats values sent last:

• dLLM requests: cached_tokens and cached_tokens_details, which change with each block.
• Retracted requests: num_retractions is current from the first output after the retraction, but queue_time, the storage waits, and after a PD-decode re-bootstrap the cached tokens, change again later and show only on the finished output.
• HiCache storage attached or detached mid-request: the storage keys of cached_tokens_details.

Verification

A differential test drove 20000 random request lifecycles (prefill, chunked prefill, decode with stream intervals 1-3, retraction and re-prefill, PD-decode prebuilt, dLLM blocks, embeddings, metrics on and off, Rust mode) through the real scheduler output path, detokenizer and tokenizer manager on main and on this PR. Finished outputs, metrics calls, span attributes and Rust payload columns matched exactly, and so did every earlier output except the stats values listed above for retracted and dLLM requests.

Micro benchmark

A steady decode step on the CPU only: 512 requests that have all emitted before, one new token each, built by the real _GenerationStreamAccumulator, then pickled and unpickled as the default IPC does. Main and this PR ran interleaved, 3 rounds of 200 steps each; each value is the minimum over rounds.

metrics	build (us/step)	pickle (us/step)	unpickle (us/step)	scheduler build + pickle (us/step)	message (KB)
off, main	562	145	138	707	38.3
off, this PR	397	96	91	492 (-30%)	22.3
on, main	912	375	301	1287	73.8
on, this PR	395	95	90	490 (-62%)	22.3

---

CI States

Latest PR Test (Base): 🚫 Run #40953106559
Latest PR Test (Extra): ❌ Run #40953106875
Latest PR Test (AMD ROCm 10): ⏳ Run #40953106536

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

_{Tip: disable this comment in your organization's Code Review settings.}

[hnyls2002]

/tag-and-rerun-ci

[hnyls2002]

/rerun-test test_output_streamer_customized_info.py test_output_streamer_logprobs.py test_tokenizer_manager_rid_cleanup.py test_multi_tokenizer_mixin.py test_setwise_score_mixin.py test_ray_wrappers.py test_deferred_output.py test_req_time_stats.py test_flat_raw_top_logprobs.py test_retraction_order.py test_io_struct.py test_metrics.py test_basic_sanity.py test_srt_endpoint.py test_multi_tokenizer.py test_skip_tokenizer_init.py test_customized_info_streaming.py test_tracing.py test_embeddings_api.py registered/disaggregation/test_disaggregation_basic.py test_retract_decode.py test_retract_decode_logprob.py

[github-actions]

🚀 ubuntu-latest (10 tests): ✅ View workflow run

cd test/ && python3 registered/unit/managers/test_output_streamer_customized_info.py
cd test/ && python3 registered/unit/managers/test_output_streamer_logprobs.py
cd test/ && python3 registered/unit/managers/test_tokenizer_manager_rid_cleanup.py
cd test/ && python3 registered/unit/managers/test_multi_tokenizer_mixin.py
cd test/ && python3 registered/unit/managers/test_setwise_score_mixin.py
cd test/ && python3 registered/unit/observability/test_ray_wrappers.py
cd test/ && python3 registered/unit/managers/test_deferred_output.py
cd test/ && python3 registered/unit/observability/test_req_time_stats.py
cd test/ && python3 registered/unit/managers/test_flat_raw_top_logprobs.py
cd test/ && python3 registered/unit/managers/test_retraction_order.py

🚀 1-gpu-5090 (9 tests): ✅ View workflow run

cd test/ && python3 registered/unit/managers/test_io_struct.py
cd test/ && python3 registered/observability/test_metrics.py
cd test/ && python3 registered/core/test_basic_sanity.py
cd test/ && python3 registered/core/test_srt_endpoint.py
cd test/ && python3 registered/tokenizer/test_skip_tokenizer_init.py
cd test/ && python3 registered/observability/test_customized_info_streaming.py
cd test/ && python3 registered/observability/test_tracing.py
cd test/ && python3 registered/scheduler/test_retract_decode.py
cd test/ && python3 registered/scheduler/test_retract_decode_logprob.py

🚀 1-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/tokenizer/test_multi_tokenizer.py
cd test/ && python3 registered/basic_perf/test_embeddings_api.py

🚀 2-gpu-h100 (1 test): ❌ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_basic.py