[kv_offload] Establish tier-owned KV event handling by Change72 · Pull Request #46544 · vllm-project/vllm

Change72 · 2026-06-23T23:04:00Z

Motivation

KV event responsibility should follow storage ownership. A tier is the only component that can distinguish an actual store from a no-op and report its own placement changes. Having a parent construct child-tier events would assume a single destination medium and make the take_events() contract harder to reason about.

This PR establishes the ownership and aggregation contract. Actual secondary-tier event implementations remain a follow-up.

It also removes LoadStoreSpec.medium(). Since #45053, each OffloadingWorker serves one offloaded medium and transfer direction is explicit through submit_store() / submit_load(), so the spec-level medium is no longer used for dispatch.

Changes

Keep CPUOffloadingManager responsible for its own BlockStored and BlockRemoved events.
Add a default SecondaryTierManager.take_events() hook.
Make TieringOffloadingManager.take_events() aggregate the primary tier followed by each secondary tier, without constructing events for them.
Define MEDIUM_CPU next to MEDIUM_GPU and use it for CPU KV events.
Remove LoadStoreSpec.medium() and the GPU, CPU, and test overrides.
Add tests for tier-event aggregation and CPU BlockRemoved-before-BlockStored ordering.

Behavior

Existing single-tier CPU wire behavior is unchanged: the same keys and CPU medium are emitted, and eviction events precede the stores that reuse their capacity.

Tiering deployments continue to expose primary CPU events. Secondary managers now have an event hook and aggregation path, but the built-in secondary tiers do not emit events in this PR.

Out of scope

Secondary-tier BlockStored implementations and their opt-in configuration.
Secondary-tier BlockRemoved / invalidation events.
Stable FS/object-store wire media.
Self-describing payloads for tiering.

Testing

tests/v1/kv_offload/cpu/test_manager.py: 18 passed.
tests/v1/kv_offload/tiering/test_tiering_offloading.py: 26 passed.
Offloading connector event and scheduler tests: 91 passed.
Pre-commit hooks passed, including Ruff, mypy, typos, SPDX, and forbidden-import checks.

Related: #38260

AI assistance

This PR was developed with assistance from Claude and Codex. I reviewed the changes and ran the tests listed above.

When a block is cascaded from the primary (CPU) tier to a secondary offloading tier under TieringOffloadingSpec, append a placeholder BlockStored presence event (block hash + tier medium; no self-describing token payload; no secondary BlockRemoved). take_events() now yields primary-tier (CPU) events before secondary-tier presence events. The events are gated on enable_kv_cache_events and are inert otherwise. Because TieringOffloadingSpec rejects self_describing_kv_events, the secondary keys have no metadata snapshot, so the existing OffloadingEventsTracker renders them as hash-only placeholder BlockStored events (token_ids=[], block_size=0, medium=<tier_type>). Step 1 of the multi-tier KV-event contract (vllm-project#38260). Signed-off-by: Change72 <changg@nvidia.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Define medium-name constants (MEDIUM_CPU/FS/OBJ) in kv_events.py next to MEDIUM_GPU. CPULoadStoreSpec.medium() returns MEDIUM_CPU (behavior-identical). Secondary tiers expose a `medium` property (mirroring CPUOffloadingManager.medium): each built-in tier sets the EVENT_MEDIUM class attribute to its stable wire name (FileSystemTierManager -> MEDIUM_FS, ObjectStoreSecondaryTierManager -> MEDIUM_OBJ), and the base falls back to the registered tier_type for custom tiers. Secondary offloading events now carry that wire medium instead of the arbitrary registered tier-type string. Config/registry tier types are unchanged ("fs"/"obj"). Per the multi-tier KV-event discussion in vllm-project#38260. Signed-off-by: Change72 <changg@nvidia.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Follow-up to the stable-wire-medium-names change: GPULoadStoreSpec.medium() returned the literal "GPU" while the sibling CPULoadStoreSpec was migrated to the MEDIUM_CPU constant. Return MEDIUM_GPU instead so all offload-spec mediums reference the kv_events.py constants. Behavior-identical (MEDIUM_GPU == "GPU"). Signed-off-by: Change72 <changg@nvidia.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

orozery · 2026-06-25T08:38:19Z

Thanks @Change72 for proposing this!

I think we need to cleanup first before we add secondary tier events.
Right now, we have medium on LoadStoreSpec, which propagates to OffloadingManager, which uses it in it's take_events.
The OffloadingManager is currently responsible for emitting Stored events coming from the parent (GPU).
We should make an effort that SecondaryTierManager has a similar behavior to OffloadingManager with respect to its parent (CPU).

My proposal:

Introduce medium() -> str | None property for both OffloadingManager / SecondaryTierManager. None would be the default, meaning Stored events should not be automatically emitted for this manager. We don't need to introduce a class var.
Remove LoadStoreSpec.medium() as it becomes redundant by OffloadingManager.medium().
OffloadingConnectorScheduler will automatically emit Stored events on complete_store, using OffloadingManager.medium() assuming it's not None.
Similarly, TieringOffloadingManager will automatically emit Stored events for any SecondaryTierManager with a medium, upon a store is reported finished by SecondaryTierManager.get_finished.
The above will be opt-in. So by default the tiering manager will not emit stored events for a secondary tier, unless its config has some enable_kv_events: True.
CPUOffloadingManager will only report eviction events.
TieringOffloadingManager will be responsible for emitting promotion events (CPU stored from a secondary tier).

WDYT?

Per review feedback (vllm-project#38260 discussion), introduce a medium() -> str | None accessor on both OffloadingManager and SecondaryTierManager as the source of the wire medium for Stored KV events: - OffloadingManager.medium() defaults to None; CPUOffloadingManager returns MEDIUM_CPU. SecondaryTierManager.medium() defaults to None. - Secondary tiers are opt-in: fs/obj accept enable_kv_events (default False) and return MEDIUM_FS/MEDIUM_OBJ only when enabled, else None. A tier whose medium() is None emits no secondary BlockStored; custom tiers are silent unless they override medium(). (Replaces the EVENT_MEDIUM class attribute.) - Managers still emit their own Stored/Removed events, now gated on medium(); the CPU wire output and event ordering (eviction Removed before the new Stored) are unchanged. No secondary BlockRemoved. LoadStoreSpec.medium() is kept: it is the worker transfer-dispatch key ((src.medium(), dst.medium()) in worker.py), independent of KV events, so it cannot be removed. Moving Stored-event emission out of the managers into the OffloadingConnectorScheduler is left as a follow-up, to keep this change behavior-preserving on the merged CPU path. Config/registry tier types are unchanged (fs/obj). Signed-off-by: Change72 <changg@nvidia.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

mergify · 2026-06-25T19:17:37Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Change72.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

- The medium() docstrings/comments said the scheduler emits Stored events; in the current design the managers emit them gated on medium(). Reword to describe the manager's event path using this medium (no scheduler-emit claim). - Add lightweight tests that the real FileSystemTierManager / ObjectStoreSecondaryTierManager report MEDIUM_FS / MEDIUM_OBJ only when enable_kv_events is set (else None), exercising medium() via __new__ to avoid their heavy __init__ (the obj test skips if NIXL is unavailable). Signed-off-by: Change72 <changg@nvidia.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Resolve the conflict in vllm/v1/kv_offload/tiering/obj/manager.py by keeping both this PR's medium() opt-in accessor and main's LookupResult-based lookup(). Update the secondary-event tests to call on_new_request() before prepare_store() (main now tracks per-request state via _req_state, populated in on_new_request). Signed-off-by: Change72 <changg@nvidia.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The medium applies to both Stored and Removed events (the CPU eviction Removed event also uses medium()), so describe it as the managers KV-event medium rather than Stored-only. Signed-off-by: Change72 <changg@nvidia.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The previous commit accidentally reverted mains base.py changes (a stale pre-merge copy was applied, dropping LookupResult and other merged content). Restore the merged base.py and apply the intended docstring clarification: medium() applies to both Stored and Removed events (the CPU eviction Removed event also uses medium()), so it documents the managers KV-event medium rather than Stored-only. Signed-off-by: Change72 <changg@nvidia.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Per the reviewer cleanup (orozery): clean up KV-event responsibility before adding secondary-tier events. - The connector (OffloadingConnectorScheduler) now emits the GPU->primary Stored KV event on complete_store, using OffloadingManager.medium(); complete_store returns the stored keys for this. - CPUOffloadingManager reports only eviction Removed events. - take_events drains the manager events (eviction Removed) before the queued Stored events, preserving the Removed-before-Stored wire order. - Remove LoadStoreSpec.medium() (+ the GPU/CPU overrides). It is no longer the worker transfer-dispatch key: since vllm-project#45053 each OffloadingWorker serves a single medium and uses submit_store/submit_load instead of (src_medium, dst_medium) routing, so it is redundant with OffloadingManager.medium(). - Drop this PR secondary-tier Stored emission (and the per-tier enable_kv_events plumbing); opt-in secondary Stored and promotion events are deferred to a follow-up. SecondaryTierManager.medium() stays (default None). A side effect: tier promotions (secondary->primary) no longer emit a phantom primary Stored event, since the connector only emits for GPU->primary store jobs. Signed-off-by: Change72 <changg@nvidia.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

- Gate record_store (the prepare_store metadata snapshot) on the same condition as the Stored emission: KV events enabled AND manager.medium() is not None. Previously, with self-describing events on but medium=None, metadata was captured but never consumed by a Stored or Removed event, so a custom manager could accumulate it indefinitely. - Drop MEDIUM_FS / MEDIUM_OBJ (no production callers now; they belong with the secondary-tier events follow-up) and restore TieringOffloadingManager.take_events() to its original ordering. - Make LoadStoreSpec a plain marker class; it no longer has abstract methods. - Document the engine-drain invariant in take_events. Signed-off-by: Change72 <changg@nvidia.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ned keys - complete_store path: read manager.medium() once instead of via the _emit_stored_events property and again for the event, so a custom manager whose medium() is not perfectly stable stays consistent. - Strengthen the connector Stored test: complete_store returns a sentinel key disjoint from the input store keys, and the test asserts the published BlockStored is built from that returned key (not the input keys). Signed-off-by: Change72 <changg@nvidia.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Change72 · 2026-06-29T00:35:38Z

Thanks @orozery. Reworked around medium(), cleanup-first as you suggested. Point by point:

medium() -> str | None on OffloadingManager and SecondaryTierManager (default None = don't auto-emit Stored), no class var.
Removed LoadStoreSpec.medium() (+ the GPU/CPU overrides). You're right it's redundant now — I'd initially thought it was still the worker transfer-dispatch key, but [KV Offload] Replace OffloadingHandler with OffloadingWorker #45053 already replaced that routing with the per-medium OffloadingWorker (submit_store/submit_load), so nothing dispatches on it.
OffloadingConnectorScheduler emits the Stored event on complete_store using manager.medium(); complete_store returns the actually-stored keys for it. Metadata capture and emission share one gate, so a None-medium manager does not accumulate tracker metadata.
(6.) CPUOffloadingManager reports only eviction Removed events.

4/5/7 (opt-in secondary-tier Stored + promotion events): deferred to a follow-up PR — this one is the cleanup. I dropped the secondary emission, the per-tier enable_kv_events plumbing, and the FS/OBJ medium constants so they
land together with the feature.

Single-tier CPU wire output is unchanged (primary Stored medium=CPU, eviction Removed before it). Happy to open the follow-up once this lands — does the two-PR split work for you?

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Replace the _emit_stored_events bool property with a _stored_event_medium property that returns the medium (or None). prepare_store metadata capture checks "is not None" and the complete_store emission reuses the returned medium, so both paths go through one helper (evaluated once each) and the docstring matches the implementation. Signed-off-by: Change72 <changg@nvidia.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Signed-off-by: Chang Guo <changg@nvidia.com>

orozery · 2026-07-01T08:50:16Z

@Change72 Thanks for the quick adaptation to my cleanup suggestion.
Looking at it again, I feel like my approach was wrong.
My motivation is always to move common logic upwards, to make the lower tiers more simple.
This is why I suggested that BlockStored be handled by the parent tier (offloading connector for CPU events, tiering manager for secondary tiers stores).
However, I now see a few downsides of this approach:

It assumes that the tier below us only has a single primary medium where all stores are routed to. This is the case for the current implementations, but in theory we can envision other types of tiering strategies.
It does not allow fine-grained stored events, filtered by whether the store was actual, or was a no-op since the block already existed on the medium.
I feel like the contract for take_events becomes more complicated to explain, since parent of the events are now handled by the parent tier, while some are still is expected from the tier itself.

Given all of that, I think it's better if we revert the changes I suggested.
Instead, the contract would be: Each tier is fully responsible for its kv events.
This already holds for today's CPU tier.
The tiering manager should only delegate to the primary tier (as your original commit did) and the secondary tiers, without building any events of its own.

BTW we can still take down LoadStoreSpec.medium(), but this is orthogonal.
We don't need to expose OffloadingManager/SecondaryTierManager.medium() as each has their own independent take_events().

Change72 · 2026-07-01T18:02:57Z

@orozery Thanks for revisiting this. I agree tier-owned events are cleaner, especially for distinguishing actual stores from no-ops and supporting multiple child media.

My understanding is:

Restore CPUOffloadingManager ownership of its BlockStored and BlockRemoved events.
Remove the new manager medium() APIs and scheduler-side Stored emission.
Make TieringOffloadingManager only aggregate take_events() from its primary and secondary tiers.
Let each secondary tier own its events when that support is added.
Keep the LoadStoreSpec.medium() removal as an orthogonal cleanup.

One scope question: should this PR also implement the opt-in secondary-tier BlockStored events, or should it stop at restoring this contract and leave those events to a follow-up?

orozery · 2026-07-06T12:15:21Z

One scope question: should this PR also implement the opt-in secondary-tier BlockStored events, or should it stop at restoring this contract and leave those events to a follow-up?

Actual secondary tier event implementations should be on a follow-up.

Keep Stored and Removed event production with the tier that owns the storage state. Restore CPU manager event ownership, remove connector-side synthesis and manager medium APIs, and make the tiering manager aggregate primary and secondary event streams only. Secondary-tier event implementations remain a follow-up; LoadStoreSpec.medium() stays removed as orthogonal cleanup. Signed-off-by: Change72 <changg@nvidia.com>

Keep the pre-existing enable_events constructor argument, internal event queue, and drain behavior. The ownership cleanup only adds secondary-tier delegation and does not need unrelated constructor API churn. Signed-off-by: Change72 <changg@nvidia.com>

This reverts commit c891bab. Signed-off-by: Change72 <changg@nvidia.com>

Pin the tier-owned CPU event contract: when a store reuses evicted capacity, BlockRemoved events are drained before the resulting BlockStored event. Signed-off-by: Change72 <changg@nvidia.com>

Change72 · 2026-07-06T19:06:36Z

@orozery Updated as discussed: each tier now owns its events, while the tiering manager only aggregates them. Secondary-tier events remain a follow-up. I also kept the orthogonal LoadStoreSpec.medium() cleanup and updated the tests and PR description. Ready for another look, thanks!

orozery

Thanks @Change72 !

orozery · 2026-07-01T05:55:16Z

+        When set, the connector emits a Stored event with this medium on
+        complete_store, and the manager tags its eviction Removed events with
+        it. Whether events are collected at all is controlled by the connector's
+        enable_kv_cache_events.


We should clarify that Stored events are emitted automatically by the offloading connector (the manager should not emit them), while Removed events are expected to be emitted by the manager itself (via take_events).
Also, the manager CAN emit events with a different medium (for example, the tiered manager).

orozery · 2026-07-01T05:56:34Z

+
+        Returns:
+            The keys that transitioned to stored by this call. The connector
+            uses these to emit a Stored KV event (see medium()). The default
+            implementation stores nothing and returns an empty list.
        """
-        return
+        return []


Why do we need this?
The offloading connector already has all of the information it needs in the keys parameter.

orozery · 2026-07-01T06:06:57Z

    policy: OffloadPolicy = OffloadPolicy.BLOCK_LEVEL


+


Signed-off-by: Change72 <changg@nvidia.com> Signed-off-by: Chang Guo <changg@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Or Ozeri <oro@il.ibm.com>

Change72 requested review from ApostaC and orozery as code owners June 23, 2026 23:04

mergify Bot added the v1 label Jun 23, 2026

Change72 mentioned this pull request Jun 23, 2026

[RFC]: Multi-tier KV offloading via the vLLM offloading connector #38260

Open

1 task

Change72 and others added 2 commits June 24, 2026 22:50

mergify Bot added the needs-rebase label Jun 25, 2026

Change72 and others added 2 commits June 25, 2026 19:45

mergify Bot removed the needs-rebase label Jun 25, 2026

Change72 and others added 2 commits June 26, 2026 00:33

Change72 marked this pull request as draft June 28, 2026 04:22

mergify Bot added the kv-connector label Jun 28, 2026

Change72 and others added 2 commits June 28, 2026 20:43

Change72 changed the title ~~[kv_offload] Emit secondary-tier BlockStored presence events~~ [kv_offload] Emit Stored KV events from the connector; drop LoadStoreSpec.medium() Jun 28, 2026

Change72 marked this pull request as ready for review June 29, 2026 00:35

Change72 requested review from NickLucche, ivanium and xuechendi as code owners June 29, 2026 00:35

claude Bot reviewed Jun 29, 2026

View reviewed changes

Change72 mentioned this pull request Jun 29, 2026

[SimpleCPUOffloadConnector]: Add KV Events #46397

Open

Merge branch 'main' into feat/tiering-secondary-presence-events

b323dda

Signed-off-by: Chang Guo <changg@nvidia.com>

Change72 added 4 commits July 6, 2026 17:11

Revert "[kv_offload] Preserve tiering event configuration"

7135538

This reverts commit c891bab. Signed-off-by: Change72 <changg@nvidia.com>

[kv_offload] Test eviction event ordering

c492bf4

Pin the tier-owned CPU event contract: when a store reuses evicted capacity, BlockRemoved events are drained before the resulting BlockStored event. Signed-off-by: Change72 <changg@nvidia.com>

Change72 changed the title ~~[kv_offload] Emit Stored KV events from the connector; drop LoadStoreSpec.medium()~~ [kv_offload] Establish tier-owned KV event handling Jul 6, 2026

orozery added the ready ONLY add when PR is ready to merge/full CI is needed label Jul 7, 2026

Merge branch 'main' into feat/tiering-secondary-presence-events

af1ac90

orozery approved these changes Jul 7, 2026

View reviewed changes

orozery merged commit c46ced1 into vllm-project:main Jul 7, 2026
86 checks passed

Change72 deleted the feat/tiering-secondary-presence-events branch July 7, 2026 16:48

Change72 mentioned this pull request Jul 7, 2026

[kv_offload] Emit tier-owned BlockStored events from FS/OBJ secondary tiers #47923

Merged

Navigation Menu

Uh oh!

[kv_offload] Establish tier-owned KV event handling - #46544

[kv_offload] Establish tier-owned KV event handling#46544
orozery merged 18 commits into
vllm-project:mainfrom
Change72:feat/tiering-secondary-presence-events

Change72 commented Jun 23, 2026 •

edited

Loading

Uh oh!

orozery commented Jun 25, 2026

Uh oh!

mergify Bot commented Jun 25, 2026

Uh oh!

Change72 commented Jun 29, 2026 •

edited

Loading

Uh oh!

claude Bot left a comment

Uh oh!

orozery commented Jul 1, 2026

Uh oh!

Change72 commented Jul 1, 2026

Uh oh!

orozery commented Jul 6, 2026

Uh oh!

Change72 commented Jul 6, 2026

Uh oh!

orozery left a comment

Uh oh!

orozery Jul 1, 2026

Uh oh!

orozery Jul 1, 2026

Uh oh!

orozery Jul 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Change72 commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Behavior

Out of scope

Testing

AI assistance

Uh oh!

orozery commented Jun 25, 2026

Uh oh!

mergify Bot commented Jun 25, 2026

Uh oh!

Change72 commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

orozery commented Jul 1, 2026

Uh oh!

Change72 commented Jul 1, 2026

Uh oh!

orozery commented Jul 6, 2026

Uh oh!

Change72 commented Jul 6, 2026

Uh oh!

orozery left a comment

Choose a reason for hiding this comment

Uh oh!

orozery Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

orozery Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

orozery Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Change72 commented Jun 23, 2026 •

edited

Loading

Change72 commented Jun 29, 2026 •

edited

Loading