[BugenZhao]

Overview

The five request token-count histograms (vllm:request_prompt_tokens, vllm:request_generation_tokens, vllm:request_max_num_generation_tokens, vllm:request_params_max_tokens, vllm:request_prefill_kv_computed_tokens) use build_1_2_5_buckets(max_model_len) in Python (PrometheusStatLogger.__init__, vllm/v1/metrics/loggers.py). The Rust frontend uses a fixed cap of 131072 (with a TODO to use max_model_len) plus an extra bound at the cap. This PR builds the same bounds as Python from the engine-reported max_model_len.

Claims

• On the Rust frontend, these histograms export Python's bucket bounds for the served model: for max_model_len=4096, 1, 2, 5, …, 1000, 2000.
• Before, every Rust deployment exported …, 50000, 100000, 131072: unreachable bounds for short-context models, everything above 131072 in +Inf for 1M-context models, and le series that do not aggregate with Python's in histogram_quantile over a mixed fleet.

Validation

Commands from .buildkite/scripts/run-rust-frontend-cargo-ci.sh (style-clippy and test), run from the repo root:

cargo fmt --manifest-path rust/Cargo.toml --all -- --check   # ok
cargo sort --workspace --check rust                          # ok
cargo deny --manifest-path rust/Cargo.toml --config rust/deny.toml check bans   # ok
cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --all-features --locked -- -D warnings   # ok
cargo nextest run --manifest-path rust/Cargo.toml --workspace --all-features --locked --no-fail-fast
# 2183 tests run: 2183 passed, 13 skipped

• request_token_buckets_are_capped_at_max_model_len (vllm-metrics) renders vllm:request_prompt_tokens for max_model_len=4096 and pins 1.0 … 2000.0 +Inf. With the old builder's extra bound restored, it fails with an extra 4096.0.
• The llm integration tests (rust/src/llm/tests/generate.rs) record these histograms through Llm; without the Llm::new call they panic on the unset cap, so they cover the wiring.

Details

The bucket rule stays in vllm-metrics, next to the other bucket constants. The global registry is built lazily, before any engine is connected, and a Family fixes its metric constructor at registration, so the five families share a small RequestTokensHistogram constructor whose buckets are set once through RequestMetrics::set_max_model_len. Histograms are created per label set on the first recorded request, which is always after that call. Llm::new owns the request metrics and the connected EngineCoreClient, so it sets the cap from client.max_model_len(): the auto-fitted engine value, the minimum across DP engines, which is also the limit the text layer enforces. The registry is process-global and serves one model, so only the first call takes effect.

Out of scope: --custom-histogram-buckets (#48867) can override this family in Python; the Rust frontend rejects the flag at startup (#62431). Supporting it later means filling the same constructor from the override.

Not a duplicate: no open upstream PR changes these Rust bucket bounds; #58563 adds connector metrics descriptors, not request token buckets.

AI assistance (Claude Code) was used to prepare this change; the human submitter reviewed every changed line.

🤖 Generated with Claude Code

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[BugenZhao]

/ci run

[github-actions]

✅ Triggered Buildkite CI #96161 for commit a41133bfa192.

ℹ️ This PR is 16 commits behind upstream main. Merged upstream main into this branch as a41133bfa192 before starting CI. Pull the branch before pushing more changes.