Skip to content

[KV Offloading] Add tiering metric plumbing - #45959

Merged
orozery merged 1 commit into
vllm-project:mainfrom
Srinivasoo7:tiering-metric-plumbing
Jun 23, 2026
Merged

[KV Offloading] Add tiering metric plumbing#45959
orozery merged 1 commit into
vllm-project:mainfrom
Srinivasoo7:tiering-metric-plumbing

Conversation

@Srinivasoo7

@Srinivasoo7 Srinivasoo7 commented Jun 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • add default build_metric_definitions() and get_stats() hooks to SecondaryTierManager
  • plumb secondary tier metric definitions through TieringOffloadingSpec
  • aggregate primary and secondary tier stats through TieringOffloadingManager
  • add a test-only secondary tier to validate labeled metric definition and stats plumbing

@Srinivasoo7
Srinivasoo7 marked this pull request as ready for review June 17, 2026 19:50
@mergify

mergify Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Srinivasoo7.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Jun 17, 2026
@Srinivasoo7 Srinivasoo7 mentioned this pull request Jun 17, 2026
Closed
1 task
@Srinivasoo7
Srinivasoo7 force-pushed the tiering-metric-plumbing branch from 619a6d3 to a769ab5 Compare June 22, 2026 02:31
@mergify mergify Bot removed the needs-rebase label Jun 22, 2026
orozery
orozery requested changes Jun 22, 2026
View reviewed changes
Comment thread vllm/v1/kv_offload/tiering/spec.py Outdated
metrics = super().build_metric_definitions(extra_config)
secondary_tier_configs = extra_config.get("secondary_tiers", [])
if not isinstance(secondary_tier_configs, list):
return metrics

@orozery orozery Jun 22, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should raise ValueError instead

Comment thread vllm/v1/kv_offload/tiering/spec.py Outdated
Comment on lines +84 to +89
if not isinstance(tier_config, dict):
continue
tier_type = tier_config.get("type")
if not tier_type:
continue
tier_cls = SecondaryTierFactory.get_tier_class(tier_type)

@orozery orozery Jun 22, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's re-use this code from SecondaryTierFactory.create_secondary_tier
Maybe let's change to def get_tier_class(cls, tier_config: dict)

Comment thread tests/v1/kv_offload/tiering/test_tiering_offloading.py Outdated

_CTX = ReqContext(req_id="test")
_MOCK_OFFLOADING_SPEC = MagicMock()
_TEST_TIER_METRIC = "vllm:kv_offload_test_tier_bytes"

@orozery orozery Jun 22, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can move this inside MetricsSecondaryTierManager.
Also, let's use a more generic name (e.g. my_tier_metric).

Comment thread vllm/v1/kv_offload/tiering/spec.py

BLOCK_SIZE_ALIGNMENT = SharedOffloadRegion.BLOCK_SIZE_ALIGNMENT

@classmethod

@orozery orozery Jun 22, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should add @override I think.

@Srinivasoo7 Srinivasoo7 Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, missed it. Will add it!

@Srinivasoo7
Srinivasoo7 force-pushed the tiering-metric-plumbing branch from a769ab5 to 1b1b4b8 Compare June 22, 2026 11:27
orozery
orozery reviewed Jun 22, 2026
View reviewed changes
Comment thread vllm/v1/kv_offload/tiering/spec.py Outdated
Comment on lines +85 to +86
if not isinstance(tier_config, dict):
continue

@orozery orozery Jun 22, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's assert instead.

@orozery orozery added the ready ONLY add when PR is ready to merge/full CI is needed label Jun 22, 2026
srinivas_oo7
Add tiering metric plumbing
336252d 
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
@Srinivasoo7
Srinivasoo7 force-pushed the tiering-metric-plumbing branch from 1b1b4b8 to 336252d Compare June 22, 2026 18:24
orozery
orozery approved these changes Jun 23, 2026
View reviewed changes

@orozery orozery left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Srinivasoo7 !

@orozery
orozery merged commit 091bc10 into vllm-project:main Jun 23, 2026
70 checks passed
nkzhenhua pushed a commit to nkzhenhua/vllm that referenced this pull request Jun 24, 2026
@Srinivasoo7 srinivas_oo7 @nkzhenhua
[KV Offloading] Add tiering metric plumbing (vllm-project#45959)
2f473e1 
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
qli88 pushed a commit to qli88/vllm that referenced this pull request Jun 26, 2026
@Srinivasoo7 srinivas_oo7 @qli88
[KV Offloading] Add tiering metric plumbing (vllm-project#45959)
66cb088 
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Signed-off-by: Qiang Li <qiang.li2@amd.com>
wincent8 pushed a commit to wincent8/vllm that referenced this pull request Jun 29, 2026
@Srinivasoo7 srinivas_oo7 @wincent8
[KV Offloading] Add tiering metric plumbing (vllm-project#45959)
48e9d00 
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Dao007forever pushed a commit to Dao007forever/vllm that referenced this pull request Jul 18, 2026
@Srinivasoo7 srinivas_oo7 @Dao007forever
[KV Offloading] Add tiering metric plumbing (vllm-project#45959)
1f8f7e5 
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
efschu pushed a commit to efschu/shvllm that referenced this pull request Jul 18, 2026
@Srinivasoo7 srinivas_oo7
[KV Offloading] Add tiering metric plumbing (vllm-project#45959)
95bda5d 
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
plasticchris pushed a commit to plasticchris/vllm that referenced this pull request Jul 20, 2026
@Srinivasoo7 srinivas_oo7
[KV Offloading] Add tiering metric plumbing (vllm-project#45959)
4307586 
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@orozery orozery orozery approved these changes
@ApostaC ApostaC Awaiting requested review from ApostaC ApostaC is a code owner
@NickLucche NickLucche Awaiting requested review from NickLucche
@xuechendi xuechendi Awaiting requested review from xuechendi

Assignees

No one assigned

Labels

kv-connector ready ONLY add when PR is ready to merge/full CI is needed v1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Srinivasoo7 @orozery