Add tiering offloading metrics by Srinivasoo7 · Pull Request #48798 · vllm-project/vllm

Srinivasoo7 · 2026-07-16T02:31:18Z

Summary

Covers the scope of TieringOffloadingSpec-Level Metrics in [RFC]: Offloading Metrics Redesign #44008
add TieringOffloadingSpec-level metric definitions with a per-tier label
extend secondary-tier JobResult payloads with optional transfer size/time
emit tiering read/write, failure, and block query/hit counters from TieringOffloadingManager
cover metric registration and manager aggregation paths in focused tests

mergify · 2026-07-16T02:34:31Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Srinivasoo7.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

orozery

Thanks @Srinivasoo7 !
Great work!!

orozery · 2026-07-22T03:34:51Z

+        for metric_name, documentation in (
+            (
+                TieringOffloadingMetrics.READ_BYTES,
+                "Total bytes read from secondary tiers into the primary tier.",


Maybe it's a good idea to add a short note in each description about the metrics supporting per-tier labels?

Addressed in 2addc9cc8: added notes to the tier-labeled metric descriptions that they are labeled by tier.

orozery · 2026-07-22T03:38:56Z

+                ),
+            )
+        )
+        for metric_name, documentation in (


Just wondering, why are we splitting this loop from the previous one?
Another option is just avoid loops at all (and pay the price for the labelnames=("tier",), line per each metric.

Addressed in 2addc9cc8: removed the grouped loops and declared these metric definitions explicitly.

orozery · 2026-07-22T04:00:07Z

        self._transfer_jobs: dict[JobId, JobMetadata] = {}
+        self._transfer_job_tiers: dict[JobId, SecondaryTierManager] = {}


Let's consolidate _transfer_job_tiers with _transfer_jobs as follows:

Rename JobMetadata to TransferJob.

Introduce _JobMetadata as a named tuple wrapping a TransferJob and tier_idx: int.

Replace self._transfer_jobs and self._transfer_job_tiers with self._jobs: dict[JobId, _JobMetadata]

Addressed in 2addc9cc8: renamed JobMetadata to TransferJob, added _JobMetadata(transfer_job, tier_idx), and consolidated tracking into self._jobs.

orozery · 2026-07-22T04:02:32Z

+    @staticmethod
+    def _tier_label(tier_idx: int, tier: SecondaryTierManager) -> tuple[str]:
+        return (f"{tier_idx}:{tier.tier_type}",)


Suggested change

@staticmethod

def _tier_label(tier_idx: int, tier: SecondaryTierManager) -> tuple[str]:

return (f"{tier_idx}:{tier.tier_type}",)

@functools.lru_cache(maxsize=None)

def _tier_label(self, tier_idx: int) -> tuple[str]:

return (f"{tier_idx}:{self.secondary_tiers[tier_idx].tier_type}",)

Addressed in 2addc9cc8: _tier_label() now uses the manager secondary tier list and caches labels. I used functools.cache, which is equivalent to lru_cache(maxsize=None) here.

orozery · 2026-07-22T04:29:01Z

            if tier is exclude_tier:
                continue
+            labelvalues = self._tier_label(i, tier)
+            self._stats.increase_counter(


We should increase queries/hit counter only once per [req][block][tier].
Let's do this:

Suggested refactor: per-request lookup metrics dedup in the tiering manager

Tier indexing

Secondary tiers use 0-based tier_idx from enumerate(self.secondary_tiers).
Labels follow "{idx}:{type}" format with +1 offset: "0:primary", "1:fs", "2:p2p".

TierLabel = tuple[str] # e.g. ("0:primary",) or ("1:fs",) _PRIMARY_TIER_LABEL: TierLabel = ("0:primary",) @functools.lru_cache(maxsize=None) def _tier_label(self, tier_idx: int) -> TierLabel: return (f"{tier_idx + 1}:{self.secondary_tiers[tier_idx].tier_type}",)

Define _RequestState in the tiering manager

@dataclass class _RequestState: # Keys whose resolution (HIT/MISS) has already been counted, grouped by tier label. # When set to None, the request has been allocated — no further metrics are emitted. observed_lookups: dict[TierLabel, set[OffloadKey]] | None = field(default_factory=dict)

Stored on the manager as self._request_states: dict[str, _RequestState].

Usage in lookup()

state = self._request_states[req_context.req_id] # created in on_new_request # --- Primary tier --- primary_result = self.primary_tier.lookup(key, req_context) if primary_result is LookupResult.HIT or primary_result is LookupResult.MISS: self._observe_lookup(state, self._PRIMARY_TIER_LABEL, key, primary_result) if primary_result is LookupResult.HIT: return LookupResult.HIT if primary_result is LookupResult.HIT_PENDING: return LookupResult.HIT_PENDING # --- Secondary tiers (tier_idx 0-based) --- any_retry = False for tier_idx, tier in enumerate(self.secondary_tiers): if tier is exclude_tier: continue result = tier.lookup(key, req_context) if result is LookupResult.HIT or result is LookupResult.MISS: self._observe_lookup(state, self._tier_label(tier_idx), key, result) if result is LookupResult.HIT: # initiate promotion... return LookupResult.RETRY elif result is LookupResult.RETRY: any_retry = True if any_retry: return LookupResult.RETRY return LookupResult.MISS

Helper:

def _observe_lookup( self, state: _RequestState, label: TierLabel, key: OffloadKey, result: LookupResult ) -> None: if state.observed_lookups is None: return tier_set = state.observed_lookups.get(label) if tier_set is not None and key in tier_set: return state.observed_lookups.setdefault(label, set()).add(key) self._stats.increase_counter(TieringOffloadingMetrics.BLOCK_QUERIES, labelvalues=label) if result is LookupResult.HIT: self._stats.increase_counter(TieringOffloadingMetrics.BLOCK_HITS, labelvalues=label)

Lifecycle

on_new_request: create _RequestState() in self._request_states[req_id].

on_schedule_end: for each req in new_req_ids → set observed_lookups = None.
(Preempted requests: no action — stays None. Metrics are only observed on the first allocation attempt.)

on_request_finished: delete from _request_states.

Summary

_RequestState serves one purpose: dedup metrics. BLOCK_QUERIES and BLOCK_HITS are
counted once per unique (tier, key) resolution, and only before the request's first
allocation. No lookup caching — tiers are always queried, remaining the source of truth.
Tier labels use "{idx}:{type}" format with primary at 0 and secondaries at 1+.

Addressed in 2addc9cc8: lookup metrics are now deduped per request/block/tier, include the primary tier label, and stop after the first allocation.

orozery · 2026-07-22T04:44:32Z


    @override
    def get_finished_jobs(self) -> Iterable[JobResult]:
        """
        Collect completed jobs from the finished-jobs queue.
        """
        results = []
+        now = time.monotonic()


We want the read/write time to be just the IO time, not including the time the job waits until its collected, and the time until its completion is collected.
This means in the DualQueueThreadPool._worker, time before and after task().
Need to change DualQueueThreadPool.get_finished to return not just the success bool but also the time it took.

Addressed in 2addc9cc8: FS timing now happens around task() inside DualQueueThreadPool._worker, and get_finished() returns the accumulated transfer time.

orozery · 2026-07-22T04:49:42Z

+                    job_id=job_id,
+                    success=success,
+                    transfer_size=entry.transfer_size,
+                    transfer_time=time.monotonic() - entry.submitted_at,


We should use agent.get_xfer_telemetry(handle) to get the time.
No need to extend TransferEntry.

Addressed in 2addc9cc8: object tier now uses agent.get_xfer_telemetry(handle) for transfer time and no longer stores timing fields in TransferEntry.

orozery · 2026-07-22T04:52:06Z

+            job_id=job_id,
+            success=success,
+            transfer_size=self._job_transfer_sizes.pop(job_id, None),
+            transfer_time=transfer_time,


Like obj, we should use agent.get_xfer_telemetry(handle)
This requires some more plumbing to expose via DataTransport.
I suggest we leave this as a follow-up to @liranschour .

Addressed in 2addc9cc8: removed the local P2P submit-time approximation and left P2P telemetry as follow-up, per your suggestion.

orozery · 2026-07-22T04:54:50Z

@@ -55,6 +70,8 @@ class JobResult:

    job_id: JobId
    success: bool
+    transfer_size: int | None = field(default=None, compare=False)


Let's remove this, and instead have the tiering manager compute this size based on the number of keys in the job and the CPU page size.

Addressed in 2addc9cc8: removed JobResult.transfer_size; the tiering manager now computes bytes from the job key count and primary block size.

Srinivasoo7 · 2026-07-24T13:49:43Z

Hi @orozery
Kindly help me with your review on the latest commit

Thanks

orozery · 2026-07-24T15:17:01Z

Hi @orozery Kindly help me with your review on the latest commit

Thanks

@Srinivasoo7 I went over it when you pushed it but I thought it's still WIP since I still see major things not addressed from the previous review.
Can you please take another look at it?
#48798 (review)

Srinivasoo7 · 2026-07-24T15:18:30Z

Hi @orozery Kindly help me with your review on the latest commit
Thanks

@Srinivasoo7 I went over it when you pushed it but I thought it's still WIP since I still see major things not addressed from the previous review. Can you please take another look at it? #48798 (review)

My bad @orozery, it's in my local. Just realized. Will review and it's coming your way soon.

Thanks!

mergify · 2026-07-26T10:06:53Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Srinivasoo7.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

orozery · 2026-07-27T04:41:42Z

@@ -629,27 +800,29 @@ def create_store_job(
        self,
        keys: Collection[OffloadKey],
        req_context: ReqContext,
-    ) -> JobMetadata:
+        tier_idx: int | None = None,


Let's remove the None option.

orozery · 2026-07-27T05:08:28Z

        #   True:  secondary → primary (promotion)
        #   False: primary → secondary (cascade)
-        self._transfer_jobs: dict[JobId, JobMetadata] = {}
+        self._jobs: dict[JobId, _JobMetadata] = {}
+        self._tier_states: list[_TierState] = [


Can we assert _tier_states is clean at the end of reset_cache?

orozery · 2026-07-27T05:09:07Z

+
+    def _observe_lookup(
+        self,
+        req_state: RequestState | None,


Let's remove the None option (and assert not None in the caller)

orozery · 2026-07-27T05:10:15Z

+    primary_write_block_ids: set[int] = field(default_factory=set)
+    primary_read_block_ids: set[int] = field(default_factory=set)


Let's switch those to integers as well (and assert they are non-negative whenever decreasing).

orozery · 2026-07-27T05:11:39Z

        #   True:  secondary → primary (promotion)
        #   False: primary → secondary (cascade)
-        self._transfer_jobs: dict[JobId, JobMetadata] = {}
+        self._jobs: dict[JobId, _JobMetadata] = {}


Can we assert _jobs is empty after reset_cache?

orozery · 2026-07-27T06:07:48Z

@@ -748,6 +921,7 @@ def on_schedule_end(self, context: ScheduleEndContext) -> None:
            state = self._req_state.get(req_id)
            if state is None:
                continue
+            state.observed_lookups = None


Can we add a test that checks no metrics are observed after allocation?

orozery

Thanks @Srinivasoo7 !
I see the metrics code adds quite a bit code to tiering/manager.py.
I asked Claude to plan moving it to tiering/metrics.py:

Suggested refactor: extract `TieringMetricsTracker` into `tiering/metrics.py`

Motivation

The tiering manager's metrics code (~130 lines) has a cohesive responsibility
that can be cleanly separated. Extracting it:

Reduces manager.py to pure orchestration logic
Makes the metrics tracker independently testable
Gives a single class ownership of all per-request metrics state

New file: `vllm/v1/kv_offload/tiering/metrics.py`

Types and constants

TierLabel = tuple[str]
_PRIMARY_TIER_LABEL: TierLabel = ("0:primary",)

Internal state (private to the module)

@dataclass(slots=True)
class _RequestMetricsState:
    observed_lookups: dict[TierLabel, set[OffloadKey]] | None = field(default_factory=dict)
    sync_lookup_delay: float = 0.0
    secondary_lookup_start_time: float | None = None


@dataclass
class _TierState:
    active_promotion_count: int = 0
    active_cascade_count: int = 0
    primary_write_block_count: int = 0
    primary_read_block_count: int = 0

Class

class TieringMetricsTracker:

    def __init__(
        self,
        tier_types: list[str],        # [tier.tier_type for tier in secondary_tiers]
        num_primary_blocks: int,
        primary_block_size: int,
    ): ...

    # ─── Label helpers ───

    @functools.cache
    def tier_label(self, tier_idx: int) -> TierLabel:
        """Returns e.g. ("1:fs",) for tier_idx=0 with tier_type="fs"."""
        ...

    @property
    def primary_tier_label(self) -> TierLabel: ...

    # ─── Request lifecycle ───

    def on_new_request(self, req_context: ReqContext) -> None:
        """Create _RequestMetricsState entry."""

    def on_request_allocated(self, req_context: ReqContext) -> None:
        """Flush accumulated sync delay histogram.
        Flush async delay histogram: time.monotonic() - secondary_lookup_start_time.
        Set observed_lookups = None (no further lookup metrics emitted)."""

    def on_request_finished(self, req_context: ReqContext) -> None:
        """Flush remaining delays, delete entry."""

    # ─── Lookup ───

    def on_lookup(
        self,
        req_context: ReqContext,
        key: OffloadKey,
        tier_label: TierLabel,
        result: LookupResult,
        elapsed: float,
    ) -> None:
        """Called once per tier.lookup() invocation (primary and each secondary).

        elapsed: wall-clock time of this tier's lookup call.

        Internally:
        - Dedup (key, tier_label) via observed_lookups; emit BLOCK_QUERIES /
          BLOCK_HITS counters
        - Accumulate elapsed into sync_lookup_delay
        - On first secondary HIT for this request: record
          secondary_lookup_start_time = time.monotonic() (async delay start)
        """

    # ─── Job lifecycle ───

    def on_job_registered(self, job_metadata: JobMetadata) -> None:
        """Increment _TierState active counts and block counts."""

    def on_job_finished(self, job_metadata: JobMetadata, result: JobResult) -> None:
        """Decrement _TierState counters AND emit completion metrics
        (transfer bytes/time on success, failure counter otherwise)."""

    # ─── Singular events ───

    def on_promotion_allocation_failure(self) -> None:
        """Increment PROMOTION_ALLOCATION_FAILURES."""

    # ─── Stats / assertions ───

    def take_stats(self) -> OffloadingConnectorStats | None:
        """Drain buffered stats + emit active transfer gauges."""

    def assert_idle(self) -> None:
        """Assert all _TierState counters are zero. Called by reset_cache."""

Changes to `manager.py`

`RequestState` loses metrics fields

@dataclass(slots=True)
class RequestState:
    req_context: ReqContext
    pending_primary_stores: int = 0
    is_finished: bool = False
    request_level_tiers: set[SecondaryTierManager] | None = None

`JobMetadata` (renamed from `_JobMetadata`) stays in `manager.py`

class JobMetadata(NamedTuple):
    transfer_job: TransferJob
    tier_idx: int

Manager call sites

# on_new_request
self._metrics.on_new_request(req_context)

# on_schedule_end (for req_id in new_req_ids)
self._metrics.on_request_allocated(state.req_context)

# on_request_finished
self._metrics.on_request_finished(req_context)

# _register_job
self._jobs[transfer_job.job_id] = jm
self._metrics.on_job_registered(jm)

# _process_finished_jobs
jm = self._jobs.pop(job_id, None)
assert jm is not None
self._metrics.on_job_finished(jm, completed_job)

# _initiate_promotion (failure path)
self._metrics.on_promotion_allocation_failure()

# lookup() — one on_lookup call per tier visited
t0 = time.monotonic()
primary_hit = self.primary_tier.lookup(key, req_context)
elapsed = time.monotonic() - t0
self._metrics.on_lookup(req_context, key, self._metrics.primary_tier_label, primary_hit, elapsed)

if primary_hit is LookupResult.HIT:
    return LookupResult.HIT
if primary_hit is LookupResult.HIT_PENDING:
    return LookupResult.HIT_PENDING

any_retry = False
for i, tier in enumerate(self.secondary_tiers):
    t0 = time.monotonic()
    result = tier.lookup(key, req_context)
    elapsed = time.monotonic() - t0
    self._metrics.on_lookup(req_context, key, self._metrics.tier_label(i), result, elapsed)

    if result is LookupResult.HIT:
        self._initiate_promotion(...)
        return LookupResult.RETRY
    elif result is LookupResult.RETRY:
        any_retry = True

return LookupResult.RETRY if any_retry else LookupResult.MISS

# get_stats()
metrics_stats = self._metrics.take_stats()
# ...aggregate with secondary tier stats...

# reset_cache()
assert not self._jobs
self._metrics.assert_idle()

Tests

All metrics-related tests should move to a new
tests/v1/kv_offload/tiering/test_metrics.py that tests the
TieringMetricsTracker directly (construct with tier_types list, call
on_* methods, assert take_stats() output). The existing tests in
test_tiering_offloading.py that exercise metrics through the full manager
(test_tiering_manager_records_secondary_lookup_metrics,
test_tiering_manager_stops_lookup_metrics_after_allocation,
test_tiering_manager_records_finished_job_metrics,
test_tiering_manager_reports_active_job_and_primary_usage_gauges,
test_tiering_manager_records_promotion_allocation_failures) should move
there and be rewritten against the tracker's interface.

orozery · 2026-07-29T04:47:27Z

+            assert state.primary_read_block_count >= 0
+        return job_metadata
+
+    def _record_finished_job_stats(


Suggested change

def _record_finished_job_stats(

def _observe_finished_job_stats(

orozery · 2026-07-29T04:47:43Z

+                time_metric, completed_job.transfer_time, labelvalues
+            )
+
+    def _record_active_transfer_stats(self, stats: OffloadingConnectorStats) -> None:


Suggested change

def _record_active_transfer_stats(self, stats: OffloadingConnectorStats) -> None:

def _observe_active_transfer_stats(self, stats: OffloadingConnectorStats) -> None:

orozery · 2026-07-29T04:53:02Z

        tier: SecondaryTierManager,
+        tier_idx: int,


With the added tier_idx, we can now eliminate the tier: SecondaryTierManager parameter (and self._origin.

orozery · 2026-07-29T13:35:57Z


    def on_new_request(self, req_context: ReqContext) -> RequestOffloadingContext:
-        return self._m.on_new_request(req_context, exclude_tier=self._origin)
+        return self._m.on_new_request(
+            req_context, exclude_tier=self._m.secondary_tiers[self._origin_idx]


Let's switch to exclude_tier_idx=self._origin_idx.

orozery · 2026-07-29T13:36:13Z


    def lookup(self, key: OffloadKey, req_context: ReqContext) -> LookupResult:
-        return self._m.lookup(key, req_context, exclude_tier=self._origin)
+        return self._m.lookup(
+            key, req_context, exclude_tier=self._m.secondary_tiers[self._origin_idx]


Same, exclude_tier_idx.

orozery · 2026-07-29T13:36:22Z


    def on_request_finished(self, req_context: ReqContext) -> None:
-        return self._m.on_request_finished(req_context, exclude_tier=self._origin)
+        return self._m.on_request_finished(
+            req_context, exclude_tier=self._m.secondary_tiers[self._origin_idx]


Same, exclude_tier_idx.

orozery · 2026-07-29T13:53:26Z

+        result: LookupResult,
+        elapsed: float,
+        *,
+        starts_async_delay: bool = False,


Right now it seems we're using this to measure time which includes promotion time.
I think it's not clear how to measure it correctly per-request (we want to measure just the delay caused by lookup, not including promotion).
On the other hand, measuring it per-block seems straight-forward.

So this is my suggestion:
I think we should measure like we do in the offloading connector _maybe_observe_lookup_async_delay:
Per request, Per-tier, per-block:

Take time before the first lookup which returned unresolved (not hit/miss)

Observe when lookup is resolved. Don't observe in any other case.

So we should keep the start time not in _RequestMetricsState directly, but on _RequestMetricsState.observed_lookups, which will hold the start time of the first deferred lookup:

@dataclass(slots=True) class _RequestMetricsState: observed_lookups: dict[TierLabel, dict[OffloadKey, float | None]] | None = field( default_factory=dict )

We should also change the sync lookup time to be per-tier. And for conformity we should also change it to be per-block.
The metrics docstrings should also be updated.

@Srinivasoo7 please feel free to suggest a better alternative if you have one.

orozery · 2026-07-29T14:34:39Z

+        self._tier_indices = {
+            tier: tier_idx for tier_idx, tier in enumerate(self.secondary_tiers)
+        }


Let's get rid of this.
Instead, switch all places that need it to use tier indices instead of secondary tier objects:

complete_store — iterates self.secondary_tiers → just use enumerate:

_flush_pending_promotions — iterates _pending_load_submissions keyed by tier → change the key to int (tier_idx)

_cascade_existing_blocks_to_request_level_tiers — iterates request_level_tiers → change from set[SecondaryTierManager] to set[int]:

orozery · 2026-08-02T04:33:07Z

+                    key,
+                    labelvalues,
+                    result,
+                    time.monotonic() - start_time,


We should take the second time right after the tier.lookup call (and not add _initiate_promotion to the time).

orozery · 2026-08-02T04:35:17Z

+        result: LookupResult,
+        elapsed: float,
+        *,
+        async_delay_start_time: float | None = None,


I think we can drop this param, and instead just take time.monotonic() inside this function.

orozery · 2026-08-02T04:41:31Z

+        key: OffloadKey,
+        tier_label: TierLabel,
+        result: LookupResult,
+        elapsed: float,


Can we rename this to lookup_duration?

orozery · 2026-08-04T12:15:15Z

+                )
+        elif result is LookupResult.RETRY and state.observed_lookups is not None:
+            observed = state.observed_lookups.setdefault(tier_label, {})
+            observed.setdefault(key, time.monotonic())


Suggested change

observed.setdefault(key, time.monotonic())

observed.setdefault(key, time.monotonic() - lookup_duration)

mergify · 2026-08-05T05:00:18Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Srinivasoo7.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>

mergify · 2026-08-09T16:04:04Z

Hi @Srinivasoo7, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>

orozery · 2026-08-09T17:09:16Z

/ci run

github-actions · 2026-08-09T17:09:32Z

✅ Triggered Buildkite CI #83066 for commit f6b59f8e2545.

orozery

Great work @Srinivasoo7 !

Srinivasoo7 · 2026-08-10T05:04:23Z

Great work @Srinivasoo7 !

Thanks @orozery, let me know next chapter we can conquer!

…ng connectors Port connector-side capability from vllm-project#50897 (commits 2b7eaf1, b7af87c): - MooncakeStoreConnector/MooncakeConnector/OffloadingConnector declare supports_eagle_prefix_cache_hashing=True and propagate the toggle to their scheduler/worker. - Store worker skips the legacy EAGLE drop (trailing volatile draft chunk) when successor hashing is active; offloading scheduler stops excluding the trailing chunk and gates offloadable tokens on the publishable-hash boundary. - Store ReqMeta gains max_save_tokens; apply_eagle(a)drop renamed for clarity in coordinator/protocol paths. - Port PR's connector unit tests; drop test_prom_metrics...tiering which depends on vllm-project#48798 (not in this branch). Test: 418 passed / 3 failed (= baseline) across Mooncake store + offloading connector suites; ruff check & format clean.

Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com> Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>

mergify Bot added v1 kv-connector labels Jul 16, 2026

mergify Bot added the needs-rebase label Jul 16, 2026

Srinivasoo7 marked this pull request as ready for review July 21, 2026 21:58

Srinivasoo7 requested review from ApostaC, ivanium and orozery as code owners July 21, 2026 21:58

claude Bot reviewed Jul 21, 2026

View reviewed changes

Srinivasoo7 force-pushed the srisa/tiering-offloading-metrics branch from fbf37fe to 87660eb Compare July 22, 2026 01:42

mergify Bot removed the needs-rebase label Jul 22, 2026

orozery previously requested changes Jul 22, 2026

View reviewed changes

mergify Bot added the needs-rebase label Jul 26, 2026

orozery reviewed Jul 27, 2026

View reviewed changes

Change72 mentioned this pull request Jul 27, 2026

[KV Offload] Carry chunk index in OffloadKey #49506

Closed

Srinivasoo7 force-pushed the srisa/tiering-offloading-metrics branch from 91e3358 to 56acf8f Compare July 29, 2026 03:26

mergify Bot removed the needs-rebase label Jul 29, 2026

orozery reviewed Jul 29, 2026

View reviewed changes

orozery reviewed Aug 3, 2026

View reviewed changes

ronensc mentioned this pull request Aug 3, 2026

[KV Offloading] Back-pressure detection and remediation #50045

Open

orozery reviewed Aug 4, 2026

View reviewed changes

mergify Bot added the needs-rebase label Aug 5, 2026

Srinivasoo7 force-pushed the srisa/tiering-offloading-metrics branch from 83bbf4f to d728a7c Compare August 6, 2026 02:16

mergify Bot removed the needs-rebase label Aug 6, 2026

srinivas_oo7 added 9 commits August 9, 2026 10:44

updated code

19b2e7c

Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>

Extract tiering metrics tracker

90a7010

Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>

Address tiering metrics review comments

9d77c66

Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>

Refine tiering lookup timing

4b33637

Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>

Adjust tiering async lookup timer

3b7edf9

Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>

Fix tiering rebase fallout

d1506cf

Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>

Fix tiering metrics CI failure

54f6db2

Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>

Fix tiering lookup test regressions

2d42c1f

Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>

Fix rebased tiering test setup

a6bdf9c

Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>

Srinivasoo7 force-pushed the srisa/tiering-offloading-metrics branch from 1e7ef43 to a6bdf9c Compare August 9, 2026 15:57

mergify Bot removed the needs-rebase label Aug 9, 2026

Fix FS tier JobResult typing

f6b59f8

Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>

orozery approved these changes Aug 10, 2026

View reviewed changes

orozery merged commit a123159 into vllm-project:main Aug 10, 2026
84 checks passed

Etelis mentioned this pull request Aug 17, 2026

[Refactor][kv_offload]: rename block→chunk #52615

Merged

4 tasks

zyp2014 pushed a commit to zyp2014/vllm that referenced this pull request Aug 21, 2026

Add tiering offloading metrics (vllm-project#48798)

35d73ab

Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com> Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>

This was referenced Aug 23, 2026

[RFC]: Generalize Connector Metrics #53484

Open

[RFC]: Reconcile Backpressure Admission #53485

Open

[RFC]: Secondary Tier Usage Metrics #53486

Open

RichardoMrMu mentioned this pull request Aug 24, 2026

[Feature]: Expose CPU vs P2P attribution for multi-tier KV restores #53368

Open

1 task

cquil11 mentioned this pull request Aug 27, 2026

[Metrics] Expose cached prompt tokens by cache tier cquil11/vllm#2

Draft

zupengwang mentioned this pull request Sep 6, 2026

[Bugfix][KV Offload] Enable NIXL telemetry for the OBJ tier #55576

Open

		self._transfer_jobs: dict[JobId, JobMetadata] = {}
		self._transfer_job_tiers: dict[JobId, SecondaryTierManager] = {}

		primary_write_block_ids: set[int] = field(default_factory=set)
		primary_read_block_ids: set[int] = field(default_factory=set)

	def _record_finished_job_stats(
	def _observe_finished_job_stats(

	def _record_active_transfer_stats(self, stats: OffloadingConnectorStats) -> None:
	def _observe_active_transfer_stats(self, stats: OffloadingConnectorStats) -> None:

	observed.setdefault(key, time.monotonic())
	observed.setdefault(key, time.monotonic() - lookup_duration)

Uh oh!

Uh oh!

Conversation

Srinivasoo7 commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

mergify Bot commented Jul 16, 2026

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

orozery left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

orozery Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Suggested refactor: per-request lookup metrics dedup in the tiering manager

Tier indexing

Define _RequestState in the tiering manager

Usage in lookup()

Lifecycle

Summary

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Srinivasoo7 commented Jul 24, 2026

Uh oh!

orozery commented Jul 24, 2026

Uh oh!

Srinivasoo7 commented Jul 24, 2026

Uh oh!

mergify Bot commented Jul 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Srinivasoo7 commented Jul 16, 2026 •

edited

Loading

orozery Jul 22, 2026 •

edited

Loading

Define `_RequestState` in the tiering manager

Usage in `lookup()`

Suggested refactor: extract `TieringMetricsTracker` into `tiering/metrics.py`

New file: `vllm/v1/kv_offload/tiering/metrics.py`

Changes to `manager.py`

`RequestState` loses metrics fields

`JobMetadata` (renamed from `_JobMetadata`) stays in `manager.py`