Repository navigation
Conversation
…sors With SGLANG_DISAGG_STAGING_BUFFER=1 on non-MLA models, prefill.py and decode.py call kv_manager.set_kv_buffer_tensors(..., slot_layer_ids=...) unconditionally. MooncakeKVManager takes that kwarg; NixlKVManager does not, so the kv manager dies with TypeError during init and the server never comes up (sgl-project#42684). Accept and stash slot_layer_ids for signature parity. The NIXL staged gather walks the registered buffers in list order, so nothing reads the mapping back; this matches the scoped fix also carried inside sgl-project#40909 and sgl-project#40861. Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
|
One honest note on verification: I tried to run |
|
Closing as superseded by #43723 ("[PD][NIXL] Fix staging buffer startup by accepting slot_layer_ids"), which landed the same fix on main: |
Motivation
Fixes #42684.
With
SGLANG_DISAGG_STAGING_BUFFER=1on non-MLA models, bothprefill.pyanddecode.pycallunconditionally.
MooncakeKVManageraccepts theslot_layer_idskwarg, butNixlKVManagernever got it, so picking the NIXL backend in staging mode kills the kv manager withduring init and the server never comes up.
Modifications
NixlKVManager.set_kv_buffer_tensorsnow takesslot_layer_idsand stashes it inkv_buffer_tensors, matching the Mooncake signature. This is parity only: the NIXL staged gather walks the registered buffers in list order and nothing reads the mapping back. The same scoped fix is currently carried inside the larger #40909 and #40861; this PR lands it on its own so staging mode works on main without waiting for those.Added two unit tests to
TestNixlStagingcovering the kwarg and its default.Accuracy Tests
No model output changes; disaggregation metadata path only.
Speed Tests and Profiling
No hot path impact.
Checklist
ruff check --select=F401,F821,F823,UP037andruff format --checkboth pass on the touched files)TestNixlStaging(base-a-test-cpu); I do not have NIXL hardware here, so the staging transfer paths themselves are covered by CI, as before.CI States
Latest PR Test (Base): ❌ Run #37628033810
Latest PR Test (Extra): ❌ Run #37628033368
Latest PR Test (AMD ROCm 10): ❌ Run #37628033989