[AlessioNetti]

Purpose

We use a custom vllm-omni implementation of the Moshi/Hibiki model architecture; with version 0.28 we noticed a significant performance degradation, approximately halving the maximum number of concurrent real-time streams on one instance. For our testing, we used a host equipped with a single H100 GPU and with an AMD 7313P CPU.

After collecting profiling data, we discovered that ~40% of the samples collected for the vllm-omni orchestrator process are in OrchestratorAggregator._merge_stage_metric_event(). In turn, this turned out to be related to book-keeping of per-request stage metrics, which is performed by default even when log_stats=False.

This overhead is particularly pronounced in the case of Moshi/Hibiki, as it produces output audio frames at a ~12.5Hz cadence. MiniCPM, on the other hand, operates at a ~1Hz interaction cadence, resulting in much less book-keeping work required to merge stage metrics and thus less overhead.

This PR addresses this performance issue by introducing a new collect_stage_metrics orchestrator flag (as well as --collect-stage-metrics / --no-collect-stage-metrics CLI flags) that allows disabling output timestamp tracking and avoid building most per-request stage metrics.

Note that a small subset of stage metrics (in/out tokens, finish reason, stage gen time) is still collected with collect_stage_metrics=False, as these are needed for correct functioning of some TTS models (e.g., Qwen3). However, the expensive book-keeping and merging work is nonetheless avoided, eliminating most metrics overhead and restoring performance to expected levels.

The option requires setting log_stats=False, and conflicting configurations are rejected. log_stats defaults to False in Python and True in API serving, while collect_stage_metrics=True by default, maintaining legacy behavior.

Test Plan

vLLM Version: 0.31.0

vLLM-Omni Commit:5f95115e

Tested branch commit: f95db5f

For the general orchestrator and serving tests:

python -m pytest \
  tests/entrypoints/test_serve.py \
  tests/engine/test_orchestrator.py \
  tests/entrypoints/test_omni_entrypoints.py \
  tests/entrypoints/openai_api/test_duplex_api_server.py \
  -k 'stage_metrics or does_not_track_output_timestamps or effective_log_stats or serving_mode_selects_engine' \
  -m 'core_model and cpu' -q --disable-warnings --tb=short

For the Qwen3 TTS-specific tests:

python -m pytest tests/entrypoints/openai_api/test_serving_speech.py \
  -k 'codec_limit or base_nonstream_completion or base_sse_completion or qwen3_tts_nonstream' \
  -m 'core_model and cpu' -q --disable-warnings --tb=short

Test Result

• Focused CPU regression selection: 23 passed, 224 deselected, 20 warnings in 17.94s
• Qwen3 TTS tests: 13 passed, 291 deselected, 20 warnings in 26.66s
• Coverage includes configuration validation, headless CLI handling, disabled metric construction/routing, disabled timestamp/audio bookkeeping, streaming-segment metrics, and serving-mode selection.
• Additional Qwen3 TTS tests verify that collect_stage_metrics=False still carries required request information (in/out token counts, finish reason).
• All applicable non-mypy pre-commit hooks passed.
• Mypy hooks for Python 3.10–3.13 failed with the same (pre-existing) nine errors. The Python 3.10 baseline comparison reproduced all nine on the unchanged base.
• The branch merged cleanly with upstream main at the time of checking.

AI assistance: Codex (GPT-6.1 Sol) was used to draft the changes needed to selectively disable per-request telemetry.

BEFORE SUBMITTING: read CONTRIBUTING.md and run the precheck-pr skill with the code agent for a self-check against project conventions.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/observability.md, docs/design/module/profiling.md.

Module owners: @gcanlin @lishunyang12 @vraiti

Routing: @gcanlin via module named in the PR description; @lishunyang12 via module of the changed files, semantic router; @vraiti via module of the changed files, semantic router

@AlessioNetti, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot routing record

Assigned Strict on cursor (cursor-grok-4.6-high) under experiment fleet-strict-cursor-grok46-zcode-glm53flash-5050-c5-z10-20261002.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit f95db5ff2028 produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[linyueqian]

The metrics opt-out makes sense for per-frame workloads, but the skipped path also handles request cleanup and speech completion metadata. Both of these still need to function correctly when metric collection is disabled.

I reviewed d3168cb statically without executing fork code or tests, and I have not verified the performance claim.

[linyueqian]

[important] When collect_stage_metrics=False, completed AR requests leak entries in MultimodalOutputProcessor._native_text_metrics_by_request. Since _update_stats_from_output stores num_generation_tokens even when log_stats=False, and the normal completion path only drains this cache via build_stage_metrics, skipping that call permanently leaks one record per completed request because release_binding does not clear it. Please decouple cache cleanup from metrics construction or avoid creating these records entirely when collection is disabled. A regression test should complete several requests with both flags disabled and verify that the cache remains empty.

[AlessioNetti]

This issue has been fixed as part of commit 54710a3, since now the if condition is once again output.finished or segment_finished, and pop_native_text_metrics() is always called. Regardless, I added a regression test for cache draining with commit f19a650.

[linyueqian]

[important] This change also drops the stage 0 finish_reason and token count from the speech API. Because SpeechOutputTokenCounter reads these exclusively from stage_metrics, running with --no-collect-stage-metrics and --disable-log-stats leaves stage0_finish_reason=None and output_tokens=0. This causes Qwen3TTSAdapter.validate_generation to skip its length-limit check, meaning a Qwen3-TTS Base request that exhausts its codec budget will return incomplete audio as a successful response rather than raising Qwen3TTSCodecLimitError. Please ensure completion and usage fields are preserved independently of optional timing metrics, and cover the flag-off path with the existing Base truncation, error, and usage expectations.

[AlessioNetti]

Thanks for flagging this - I addressed the issue with commit 54710a3.

Now StagePool.build_stage_metrics() always builds and returns a minimal StageRequestMetrics object with all its non-optional fields (including in/out token counts plus finish reason) populated. OrchestratorAggregator._merge_stage_metric_event() then processes the minimal metrics set, and skip any expensive book-keeping and merging work.

I also added regression tests to ensure that the minimal set of StageRequestMetrics fields is always preserved, as well as new Qwen3 TTS tests that run with stage metrics enabled and disabled:

• test_qwen3_tts_base_nonstream_completion_and_usage: a "length" finish reason must raise Qwen3TTSCodecLimitError without publishing usage; a normal "stop" must return audio with correct token usage.
• test_qwen3_tts_base_sse_completion_usage: checks that successful streaming emits audio and a completion event with correct usage, without an error event.

[vllm-omni-review-bot]

Omni ReviewBot: superseded

The CI failure noted on f19a6504149b refers to an earlier head; the pull request now points at f95db5ff2028.

[linyueqian]

I can confirm the earlier issues are fixed at f95db5ff, as completed requests and segment boundaries now properly drain the native metrics cache, and disabling diagnostics preserves the token counts and finish reasons used by speech validation and usage reporting. The new regression tests cover the cache cleanup as well as the Qwen3-TTS success and truncation paths with diagnostics both on and off.

Could you also refresh the PR description and validation SHAs? The lightweight stage metrics are now built and routed, and the listed 01445d98 results predate these recent fixes. This is just a non-blocking description update. As a note, my review was strictly static across the code and tests, and I haven't run the fork code or remeasured the performance claim.

[AlessioNetti]

Thank you for the review @linyueqian - I just updated the PR description like you suggested.

[vllm-omni-review-bot]

Omni ReviewBot attempt record

Review attempt ended as failed (failed; retrying strict/cursor/cursor-grok-4.6-high in 120s (try 2 of 3)).

[vllm-omni-review-bot]

Omni ReviewBot attempt record

Review attempt ended as failed (failed; retrying strict/cursor/cursor-grok-4.6-high in 120s (try 2 of 3)).

[vllm-omni-review-bot]

Omni ReviewBot attempt record

Review attempt ended as failed (failed; retrying strict/cursor/cursor-grok-4.6-high in 600s (try 3 of 3)).

[vllm-omni-review-bot]

Omni ReviewBot attempt record

Review attempt ended as failed (failed; falling back to direct/cursor/auto).