Add vllm:kv_offload_cpu_total_blocks capacity metric

[yanburman]

The native CPU OffloadingConnector reported usage fractions but exposed no metric for the offload tier's static configuration or capacity. Add vllm:kv_offload_cpu_config_info, a cache_config_info-style Info gauge (value always 1; config carried in the labels) reporting num_blocks, blocks_per_chunk (num_gpu_blocks-equivalent capacity = num_blocks * blocks_per_chunk; multiply by the cache block_size for a token count), kv_bytes_per_chunk, cpu_page_size_per_worker and eviction_policy. Emit it once at engine startup via a new KVConnectorBase_V1.get_config_info() hook and the EngineCoreReadyResponse handshake, covering standalone, MultiConnector-nested (PD) and tiering-primary connectors, and cover it with manager and startup-emission unit tests.

Purpose

The native CPU KV OffloadingConnector reports usage-fraction gauges
(vllm:kv_offload_cpu_cache_usage_perc, vllm:kv_offload_cpu_cache_write_usage_perc,
vllm:kv_offload_cpu_cache_read_usage_perc) but exposes no metric for the offload
tier's static configuration or capacity, so none of it is observable on /metrics.

This PR adds an Info gauge, vllm:kv_offload_cpu_config_info, whose value is always 1
and whose labels carry the tier's static config: num_blocks (offload capacity in
offload blocks), blocks_per_chunk (GPU blocks per offload block — num_gpu_blocks
equivalents = num_blocks * blocks_per_chunk; multiply by the cache block_size for
a token count), kv_bytes_per_chunk, cpu_page_size_per_worker and eviction_policy.
This mirrors how the GPU tier's config, including num_gpu_blocks, is exposed via
vllm:cache_config_info.

The metric is emitted once at engine startup (before any request) and deliberately
not on the per-interval connector-stats path — re-serializing a constant gauge over
IPC every interval is wasted work, and the Prometheus gauge persists once set. A new
generic KVConnectorBase_V1.get_config_info() hook exposes a connector's static
config, the values ride the EngineCoreReadyResponse handshake into the frontend
(stored on KVTransferConfig), and PrometheusStatLogger emits them in
log_engine_initialized. MultiConnector aggregates its nested children's config
(and its prom fans the emission to each child), and TieringOffloadingManager
surfaces its CPU primary tier's config — so an OffloadingConnector standalone,
nested in a MultiConnector (PD), or as a tiering primary tier all emit at startup.
Offloading gauges default to multiprocess_mode="mostrecent", so the static gauge
(and the existing per-engine usage gauges) collapse to one series per engine under
data-parallel + api_server_count > 1 instead of one duplicate row per API-server
process; per-process transfer counters and histograms stay summed.

Scope: the native OffloadingConnector only (standalone, MultiConnector-nested,
tiering primary tier). No behavior change beyond the added metric and the
multiprocess_mode correction on the existing offloading gauges.

Test Plan

• Unit tests: pytest tests/v1/kv_offload/cpu/test_manager.py tests/v1/kv_offload/test_factory.py
  • New test_cpu_manager_reports_config_info asserts get_config_info() returns
    {CPU_CONFIG_INFO: {...}} and that config_info is not placed on the per-interval
    stats path.
  • New test_cpu_config_info_wiring_matches_declared_labels, test_cpu_config_info_reflects_tensor_parallel_sizing,
    test_kv_offload_cpu_config_info_startup_emission and ..._via_multi_connector
    cover label/dict anti-drift, TP/PP sizing, and end-to-end startup emission
    (standalone and MultiConnector-nested) with the gauge built as mostrecent.
• Lint / type (pre-commit): pre-commit run --files $(git diff --name-only origin/main...HEAD)
• (Optional, manual e2e) Serve with --kv-offloading-size <GiB> and confirm
  vllm:kv_offload_cpu_config_info appears on /metrics at startup with value 1 and
  labels whose num_blocks * blocks_per_chunk scales with the configured size.

Test Result

Unit tests:

$ pytest tests/v1/kv_offload/cpu/test_manager.py
......................... [100%]
25 passed, 1 warning in 2.05s

$ pytest tests/v1/kv_offload/test_factory.py
.................................. [100%]
34 passed, 15 warnings in 12.38s

pre-commit (ruff check, ruff format, typos, mypy, SPDX headers, forbidden-imports,
config-docstring validation): all hooks passed.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update — n/a: the sibling CPU-offload connector gauges are not listed in docs/design/metrics.md, so this metric follows the existing (undocumented) pattern.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[orozery]

Why do we need to expose the number of CPU blocks?
For the GPU tier I don't see we do that as well.

[yanburman]

Why do we need to expose the number of CPU blocks?
For the GPU tier I don't see we do that as well.

GPU is exposed - num_gpu_blocks is a label on vllm:cache_config_info. For CPU there is only usage %, so this fills that gap. I kept it as a gauge in kv_offload_cpu_* next to usage, rather than a cache_config_info label like GPU, since the offload buffer belongs to the connector rather than the engine's cache config IMO.

[orozery]

GPU is exposed - num_gpu_blocks is a label on vllm:cache_config_info. For CPU there is only usage %, so this fills that gap. I kept it as a gauge in kv_offload_cpu_* next to usage, rather than a cache_config_info label like GPU, since the offload buffer belongs to the connector rather than the engine's cache config IMO.

I see. Then let's follow a similar pattern:
Introduce vllm:kv_offload_cpu_config_info that will hold all static runtime information for the CPU tier.
cc @varun-sundar-rabindranath @Srinivasoo7 WDYT?

[varun-sundar-rabindranath]

Agree that adding vllm:kv_offload_cpu_config_info for static information will be useful and extensible.

[Srinivasoo7]

GPU is exposed - num_gpu_blocks is a label on vllm:cache_config_info. For CPU there is only usage %, so this fills that gap. I kept it as a gauge in kv_offload_cpu_* next to usage, rather than a cache_config_info label like GPU, since the offload buffer belongs to the connector rather than the engine's cache config IMO.

I see. Then let's follow a similar pattern: Introduce vllm:kv_offload_cpu_config_info that will hold all static runtime information for the CPU tier. cc @varun-sundar-rabindranath @Srinivasoo7 WDYT?

Yup, agree on this. Besides I understand that this has the similar fuctionality as trasnfer_type metric and is consistent with existing cache_config_info from GPU

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @yanburman.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[orozery]

I suggest we avoid the changes outside the offloading connector / kv_offload.
I guess this means we observe on the first scheduler step instead of on startup.

[yanburman]

I suggest we avoid the changes outside the offloading connector / kv_offload. I guess this means we observe on the first scheduler step instead of on startup.

The outside change is not introducing a new mechanism. It rides on existing pattern that is in use by num_gpu_blocks and is quite small if you look only at production code. Making it only populate on first request makes monitoring a bit awkward and diverges from current gpu blocks implementation

[orozery]

The outside change is not introducing a new mechanism. It rides on existing pattern that is in use by num_gpu_blocks and is quite small if you look only at production code. Making it only populate on first request makes monitoring a bit awkward and diverges from current gpu blocks implementation

@NickLucche @ivanium your thoughts on this?

TLDR adding a dedicated connector API for exposing runtime level config:

  def get_config_info(self) -> dict[str, dict[str, str]] | None:
        """
        Static, per-connector configuration to emit once at engine startup as
        Info-style gauges, mapping ``metric_name -> {label: value}``. Mirrors
        ``vllm:cache_config_info`` so a connector's static config is observable
        on ``/metrics`` before any request. Returns None when there is nothing
        to expose.
        """
        return None

[nilig]

Wonder if we could expose the CPU offload capacity directly in tokens, not only as num_blocks.

CPUOffloadingSpec.num_blocks counts offload chunks rather than GPU blocks. The PR says token capacity can be calculated as num_blocks * blocks_per_chunk * block_size, but that is exact only for a single KV-cache group. The CPU pool uses a group-specific OffloadKey, so with multiple groups the same logical token range consumes several slots. The formula therefore overstates capacity, even when the groups have equal block spans.

llm-d router #2292 is a concrete consumer. It needs token capacity to size its approximate per-endpoint cache, and already reads SGLang's sglang:hicache_host_total_tokens. If vLLM publishes only blocks and chunk configuration, every consumer must reproduce this conversion and handle grouped models correctly.

Could we add a safely computed vllm:kv_offload_cpu_capacity_tokens alongside the config info? Initially, we could emit it only when there is one KV-cache group and omit it otherwise. The calculation and group handling can remain within offloading/ and kv_offload/, while the value can use whichever emission path this PR settles on.

I have a small prototype with tests for one group, multiple equal-span groups, and multiple differing-span groups. Would you prefer folding it into this PR or handling it as a follow-up after this lands?

[nilig]

I think this can be split into two independent pieces:

1. The router-facing token-capacity metric.
2. The broader config_info and startup-emission API.

I opened #51615 as a concrete example of the first piece. It adds vllm:kv_offload_cpu_capacity_tokens using the existing connector-stats path. The production change is 28 lines, entirely under vllm/v1/kv_offload/, with no new connector or engine API. The metric appears after the first scheduler step rather than at startup.

It emits only for a single KV-cache group. With multiple groups, the CPU pool stores group-specific OffloadKeys, so the documented num_blocks * blocks_per_chunk * block_size conversion overstates the available token capacity.

This draft is not intended to replace the full static configuration proposed here. It demonstrates how the token-capacity piece required by llm-d router #2292 can be unblocked while the generic startup/configuration plumbing is reviewed separately.

The implementation can be folded into this PR if preferred; otherwise the narrow metric can land first and the broader config_info work can follow independently.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @yanburman.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork