Repository navigation
[Bugfix] Wire iteration_stats through StagePool.process_llm_raw_outputs - #4747
amanchugh89 wants to merge 2 commits into
Conversation
`process_llm_raw_outputs` accepts an `iteration_stats` argument but then unconditionally did `iteration_stats = IterationStats()`, so the output processor populated a throwaway local instead of the caller's object. The orchestrator builds its own `IterationStats`, passes it in, and then records metrics off that same object via the stat logger, so per-(stage, replica) iteration metrics were always recorded empty. Allocate a fresh `IterationStats` only when the caller passes None. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: amanchugh89 <amanchugh.89@gmail.com>
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
|
Validated the regression on the actual code (vLLM 0.23.0): with the fix, |
|
Closing for now — hasn't attracted review and I'm focusing elsewhere. The fix + regression test are validated (red→green) if anyone wants to pick it up; happy to reopen. Thanks! |
What
StagePool.process_llm_raw_outputsaccepts aniteration_statsarg but overwrites it with a freshIterationStats()before calling the output processor, so the processor fills a throwaway and thecaller's object stays empty.
The orchestrator passes its own
IterationStatsand then records it viaOmniPrometheusStatLogger(inherits
PrometheusStatLogger.record, which reads it). So per-(stage, replica) prompt/generationtoken counts, preemption / corrupted-request counters, and TTFT/TPOT/ITL histograms are recorded as
zero whenever Prometheus logging is enabled.
Fix
Allocate only when the caller passes
None; otherwise the processor fills the caller's object.Tests
tests/engine/test_stage_pool_iteration_stats.py— asserts the caller'sIterationStatsreachesprocess_outputsby identity (fails onmain, passes with the fix), plus theNonepath.ruffclean; suite runs in CI.
Not a duplicate: no open PR touches this wiring; the arg was added in #3628 but never threaded
through. AI assistance was used; every line reviewed.