[XPU] Set consistent default KV cache layout #24745
Conversation
Signed-off-by: NickLucche <[email protected]>
NickLucche
requested review from
WoosukKwon,
aarnphm,
alexm-redhat,
chaunceyjiang,
comaniac,
jikunshang,
njhill,
robertgshaw2-redhat and
ywang96
as code owners
Signed-off-by: NickLucche <[email protected]>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request refactors the handling of KV cache layout settings to be more consistent, especially for XPU which requires the 'NHD' layout. It introduces a centralized set_kv_cache_layout function, which is an improvement over direct environment variable manipulation. My review includes a suggestion to make the validation of user-provided layouts more robust. A significant part of this PR is the addition of the vllm/entrypoints/cli/coordinate.py file, which appears to be a refactoring of the server startup logic, though it's not mentioned in the PR description.
Signed-off-by: NickLucche <[email protected]>
vllm/v1/attention/backends/utils.py
Outdated
| KVCacheLayoutType = Literal["NHD", "HND"] | ||
| _KV_CACHE_LAYOUT_OVERRIDE: KVCacheLayoutType | None = None |
There was a problem hiding this comment.
@LucasWilkinson any other meaningful layout we want to allow here?
|
|
||
| # Supported xPUs and types of kv transfer buffer. | ||
| # {xPU: tuple of supported kv buffer types} | ||
| _NIXL_SUPPORTED_XPUS = { |
There was a problem hiding this comment.
I think xPU was a great name for a generic PU, but since Intel already claimed it this is just confusing now >.<
|
LGTM |
Signed-off-by: NickLucche <[email protected]>
jikunshang
left a comment
jikunshang
left a comment
There was a problem hiding this comment.
thanks for fixing. LGTM!
Signed-off-by: NickLucche <[email protected]>
Signed-off-by: NickLucche <[email protected]> Signed-off-by: xuebwang-amd <[email protected]>
Signed-off-by: NickLucche <[email protected]>
Signed-off-by: NickLucche <[email protected]> Signed-off-by: xuebwang-amd <[email protected]>
4 participants
Small quality of life change to keep the way we interact with KV cache layout consistent as I explain a bit here #22735.
cc @zhenwei-intel to keep me true on the XPU-related change. PS that limit will prevent XPU from being compatible with heteroTP as of now.
cc @LucasWilkinson as this is another kv layout constraint which is good to keep in mind.