Repository navigation
[Bugfix] adjust_dcp_kv_cache_interleave_size for NixlConnector only - #54803
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
/ci run |
|
✅ Triggered Buildkite CI #86650 for commit |
|
Is it safe for other variants of nixlconnectors (nixlpush) or mooncake? Wonder if it is okay here just because we only support nixl pdd for dcp yet or the other PD connectors are inherently tokenwise transfer. |
|
Actually I think we should not enforce cp_interleave_size to be block_size for P/D transfer. Attention backend with native cp support always assume cp_interleave_size = 1 (https://github.com/flashinfer-ai/flashinfer/blob/v0.6.18/flashinfer/mla/_core.py#L3724-L3734) So we should fix the connector piece to support random interleave size. |
it's the only pd connector supporting it
block is the minimal unit for pd, if you scatter tokens you need to read from all ranks all the times and reconstruct the block you actually need; so big no-no pd side |
The parametrized MultiConnector case asserts on an info_once message that the lru_cache in vllm.logger dedupes after the identical NixlConnector param emits it first in the same process. Request the disable_log_dedup fixture so both params observe the log record. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai> Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
|
/ci run |
|
✅ Triggered Buildkite CI #86832 for commit |
ivanium
left a comment
ivanium
left a comment
There was a problem hiding this comment.
LGTM. Fixing a real issue and I think we can merge this first.
…vllm-project#54803) Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai> Co-authored-by: cjackal <44624812+cjackal@users.noreply.github.com>
…vllm-project#54803) Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai> Co-authored-by: cjackal <44624812+cjackal@users.noreply.github.com>
…vllm-project#54803) Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai> Co-authored-by: cjackal <44624812+cjackal@users.noreply.github.com>
4 participants
DCP kv block adjustment is not being gated on PD connector being present.
I like the alternative approach in #54457, but I think it might be slightly overkill: I do not expect the number of connectors to grow here.
Basically for PD this is a requirement ow block content gets fractioned and the unit of transfer is the block itself.
This PR simply gates the adjustment behind NixlConnector being present.
I would like to do "is_pd" here, but we have some interface work to do that reliably here.
added @cjackal as co-author as it reported the issue and I think the original fix was still valid