[SimpleCPUOffloadConnector]: Add KV Events - #46397
jonathanc-n wants to merge 1 commit into
Conversation
| parent_block_hash=None, | ||
| token_ids=[], |
There was a problem hiding this comment.
QQ: can we prepare them correctly rather than skipping them? Would that be super complex?
There was a problem hiding this comment.
I think we need to populate parent_block_hash and token_ids, such that external routers like dynamo can consume the cpu events correctly
(see #43468)
| @@ -131,6 +132,7 @@ def __init__( | |||
| hash_block_size=self.hash_block_size, | |||
| ) | |||
| self.cpu_block_pool: BlockPool = self.cpu_coordinator.block_pool | |||
| self.cpu_block_pool.medium = "CPU" | |||
There was a problem hiding this comment.
let's add the definition in vllm/distributed/kv_events.py?
There was a problem hiding this comment.
@zhewenl @jonathanc-n #46544 already adds MEDIUM_CPU = "CPU" to
vllm/distributed/kv_events.py (next to MEDIUM_GPU) — the definition you're
asking for here. Centralizing the wire medium names there is the direction
@orozery suggested in #38260, so once #46544 lands this PR can just import
MEDIUM_CPU instead of the "CPU" literal. Disjoint files otherwise, no conflict.
There was a problem hiding this comment.
Makes sense thanks
There was a problem hiding this comment.
@jonathanc-n @zhewenl Hi, #46544 has been merged today. You can safely use MEDIUM_CPU in vllm/distributed/kv_events.py now.
| parent_block_hash=None, | ||
| token_ids=[], |
There was a problem hiding this comment.
I think we need to populate parent_block_hash and token_ids, such that external routers like dynamo can consume the cpu events correctly
(see #43468)
|
This pull request has merge conflicts that must be resolved before it can be |
Signed-off-by: Jonathan <chenleejonathan@gmail.com>
fa7fec0 to
5bdfab6
Compare
|
This pull request has merge conflicts that must be resolved before it can be |
4 participants
Purpose
Add KV events.
Changes
Add
.mediumparameter to specify what medium the KV events are being reported from (defaults togpu)