Add Prometheus /metrics Support

[vraiti]

Purpose

Addresses #3228

Add Prometheus metrics for Omni multi-stage pipelines. Pipeline-level metrics (vllm:omni_*) track request counts, e2e latency, queue time, and per-stage diffusion timing breakdowns. Upstream vLLM per-engine metrics (vllm:*) are preserved for AR stages via PrometheusStatLogger.

To distinguish metrics generated by distinct engines, each metric has four labels:

• model_name: The name of the model the engine is running
• engine: A unique integer identifying that engine
• stage_id: The index of the pipeline stage this engine is serving (can be non-unique within a replicated stage)
• replica_id: A base-0 index unique to each replica within a stage

Workarounds

stage_id and replica_id actually can't be supported right now for LLM stages since the upstream PrometheusStatLogger does not accept custom metric labels. This is addressed in vllm-project/vllm#42743. Once that PR is merged, only the following will be required to provide LLM metrics with stage_id and replica_id: vraiti@fa7d3a9

Upstream unregister_vllm_metrics() (called inside PrometheusStatLogger.__init__) strips any collector whose name contains "vllm". There is currently a PR in vLLM to make this more precise: vllm-project/vllm#42331, but until then a new patch has been added to make unregister_vllm_metrics a no-op since the function was just there for CI purposes (it does not affect production use).

This PR also includes a make_stats() throttle (implemented in OmniSchedulerMixin) that limits SchedulerStats collection to once per second, eliminating a performance regression in LLM stages due to serialization overhead

Metrics added

Metric	Type
vllm_omni:num_requests_running	Gauge
vllm_omni:num_requests_waiting	Gauge
vllm_omni:num_requests_success	Counter
vllm_omni:num_requests_fail	Counter
vllm_omni:e2e_request_latency_seconds	Histogram
vllm_omni:request_queue_time_seconds	Histogram
vllm_omni:diffusion_preprocess_time_ms	Histogram
vllm_omni:diffusion_exec_time_ms	Histogram
vllm_omni:diffusion_postprocess_time_ms	Histogram
vllm_omni:diffusion_step_time_ms	Histogram

Test Plan

1. vLLM regression (Qwen3-Omni, 4x A100-40GB)

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \
  --omni --deploy-config vllm_omni/deploy/qwen3_omni_tp2.yaml --port 8000

Run text-only chat completions: 1 warmup request, 3 benchmark requests. Verify vllm:* metrics are present on /metrics.

2. Diffusion regression (Z-Image-Turbo, 1x A100-40GB)

vllm serve Tongyi-MAI/Z-Image-Turbo --omni --port 8000

Run 512x512 image generation at 8 diffusion steps: 1 warmup request, 5 benchmark requests. Verify v_llm_omni:* metrics are present and diffusion timing histograms are populated.

3. vLLM metrics (Qwen3-Omni)

Scrape /metrics from the Qwen3-Omni deployment and verify all expected vllm:* metric families are present (scheduler state, cache usage, token throughput, latencies).

4. Diffusion metrics (Z-Image-Turbo)

Scrape /metrics from the Z-Image-Turbo deployment and verify v_llm_omni:* pipeline metrics are present, diffusion timing histograms have observations, and no vllm:* metrics appear.

5. engine_id and stage_id metadata

Deploy diffusion and LLM pipelines with multiple replicas and observe correct

Test Result

1. vLLM regression (Qwen3-Omni)

	Baseline	feat_prometheus
Mean latency	3.08s	3.07s
Text tok/s	83.4	83.1

No regression observed

2. Diffusion regression (Z-Image-Turbo)

	Baseline	feat_prometheus
Mean latency	1.31s	1.31s

No regression observed.

3. vLLM metrics (Qwen3-Omni)

37 vllm:* metric families present. Full output: metrics_prometheus_qwen3.txt

4. Diffusion metrics (Z-Image-Turbo)

10 v_llm_omni:* metric families present. Diffusion timing histograms populated. No vllm:* metrics (expected — no AR engine). Full output: metrics_prometheus_z-image.txt

5. engine_id and stage_id metadata

Diffusion (Z-Image-Turbo):
metrics_with_replicas_diffusion.txt

LLM (

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
• The test results. Please paste the results comparison before and after, or the e2e results.
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
• (Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[hsliuustc0106]

Good addition of Prometheus /metrics support for observability. The new metrics for diffusion (preprocess, exec, postprocess, total time) will help with performance monitoring. Suggestions: 1) Consider adding metrics for request queue length/wait time. 2) Document which endpoints expose /metrics and how to configure Prometheus scraping. 3) Add tests to verify metrics are recorded correctly.

[hsliuustc0106]

cc @bjf-frz @wuhang2014 PTAL

[vraiti]

Good addition of Prometheus /metrics support for observability. The new metrics for diffusion (preprocess, exec, postprocess, total time) will help with performance monitoring. Suggestions: 1) Consider adding metrics for request queue length/wait time. 2) Document which endpoints expose /metrics and how to configure Prometheus scraping.

1. v_llm_omni:num_requests_waiting is a queue depth gauge and v_llm_omni:request_queue_time_seconds is a queue time histogram.

2. /metrics is mostly exposed unconditionally when serving online, but docs/usage/metrics.md documents when each metric is exposed (just dependent on whether the pipeline includes the relevant stage engine type).

Is there something not covered here you'd like to see?

3. Add tests to verify metrics are recorded correctly.

I will add unit tests

[wuhang2014]

Thanks for the metrics work here — the overall direction looks good, but I found two blocking issues and one doc regression:

1. OmniPrometheusMetrics registers collectors on every instance creation, which is likely to break when multiple Omni/OmniBase instances are created in the same Python process due to duplicate Prometheus timeseries registration.
2. request_queue_time_seconds is defined, documented, and unit-tested synthetically, but the production path never passes queue_seconds into request_succeeded(), so the metric will never accumulate real observations.
3. docs/design/index.md appears to have dropped links to two existing feature design docs.

I left inline comments with details and suggested fixes.

[wuhang2014]

I don’t think v_llm_omni: is the right long-term design here. The orchestrator already uses the correct pattern for upstream metrics: one PrometheusStatLogger with per-engine labels, not one logger per stage. The problem is the separate OmniPrometheusMetrics path in OmniBase, which registers fresh collectors per instance.

That separate registration path is what forces the custom prefix, because upstream unregister_vllm_metrics() removes any collector whose name contains "vllm". So a simple rename to vllm:* would still be fragile.

I’d suggest folding these omni metrics into the single orchestrator-owned/global Prometheus logger instead of registering a second collector set in OmniBase. Then they can live in the same namespace with names like vllm:omni_*, while stage/model separation stays in labels. That would avoid both duplicate registration and the prefix workaround.

[vraiti]

It's not exactly one logger per stage. It's one logger for vLLM stages (PrometheusStatLogger) and one logger for diffuser stages/the pipeline (OmniPrometheusMetrics)

The separate OmniPrometheusMetrics class inside OmniBase is necessary because some vLLM metrics require per-iteration updates, while e2e pipeline statistics require the OmniBase/entrypoint context. For example:

• num_requests_waiting: Requires reading the length of the OmniBase.request_states
• num_requests_fail: Requires tracking in OmniBase since malformed HTTP requests can fail before they reach the orchestrator
  Meanwhile, any metrics that depend upon vLLM's IterationStats logically should be collected at the orchestrator level, because that's where the per-iteration control occurs.

Additionally, I chose to keep the vLLM-Omni metrics separate from the vLLM metrics because PrometheusStatLogger is imported directly from vLLM. The reason vLLM metrics work seamlessly in this PR is that vLLM metrics infrastructure is (mostly) preserved, and trying to merge it with the vLLM-Omni-specific logic is likely to break that support or at least make it more brittle.

Folding OmniPrometheusMetrics into this class (i.e. subclassing PrometheusStatLogger in OmniPrometheusMetrics) would still require at least the following:

• Collecting all generated vLLM IterationStats into a list within the orchestrator and passing this up to OmniBase
• Moving PrometheusStatLogger.record() to OmniBase, where you would have to call it once per collected IterationStats collected
• Creating a new method to record pipeline/diffusion statistics since PrometheusStatLogger.record() takes the vLLM-specific dataclasses IterationStats and SchedulerStats

[wuhang2014]

Thanks, this makes sense.

I still don’t think v_llm_omni: is the right design here. It feels like a workaround for unregister_vllm_metrics() being too broad, rather than the right metric namespace.

I’d rather keep the metric names aligned with the vLLM namespace and add a small hack/fix in unregister_vllm_metrics() so it avoids unnecessary unregister, instead of encoding that workaround into a new long-term prefix.

[vraiti]

Ok,I've submitted a vLLM PR to fix: vllm-project/vllm#42331 . This won't get merged until at least the next minor version, so I've added a patch that changes unregister_vllm_metrics to a no-op temporarily. Either way, vLLM-Omni metrics now use the prefix vllm:omni_

[amy-why-3459]

Thank you very much for your contribution. Would you be interested in supporting performance data statistics for different stages in the benchmark? #1361

[vraiti]

@wuhang2014 changed metrics prefix to vllm_omni: and vllm#42331 should be merged soon enough. Anything else?

[vraiti]

Thank you very much for your contribution. Would you be interested in supporting performance data statistics for different stages in the benchmark? #1361

Will work on #1361 once this PR is merged since this PR already implements mechanisms for per-stage statistics

[amy-why-3459]

Will metrics be printed in the logs?

[wuhang2014]

@wuhang2014 changed metrics prefix to vllm_omni: and vllm#42331 should be merged soon enough. Anything else?

Looks good overall. One remaining issue: since PR #2396, a stage can have multiple replicas. Without a replica_id label, metrics from replicas of the same stage will be merged. Please add replica_id so they can be split per replica.

[vraiti]

Will metrics be printed in the logs?

No, that's the whole point of Prometheus metrics: instead of writing performance statistics in the logs (which are generally reserved for system diagnostics), you make them accessible via a separate channel (the /metrics endpoint).

You then spin up a Prometheus server that runs alongside the main server and queries metrics for aggregation. Then, when you want to observe these metrics, you connect Grafana to the Prometheus server. Grafana allows you to use the Prometheus Query Language to automatically construct graphs using the aggregated metrics.

You can imagine how this provides a much cleaner developer experience than trying to parse server logs and perform your own aggregations and visualizations whenever you want to evaluate server performance.

[vraiti]

Looks good overall. One remaining issue: since PR #2396, a stage can have multiple replicas. Without a replica_id label, metrics from replicas of the same stage will be merged. Please add replica_id so they can be split per replica.

@wuhang2014 done: 6aab25e