[CPU] Fix NUMA/core binding for DP ranks - #32856
Conversation
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
… if using sglang router
…s always 1 using sglang router
|
/tag-run-ci-label |
| cd python && \ | ||
| cp pyproject_cpu.toml pyproject.toml && \ | ||
| uv pip install . && \ | ||
| uv pip install ".[diffusion]" && \ |
There was a problem hiding this comment.
This change is needed to fix Xeon CI failure. Please check details in PR description (Modifications 2)
There was a problem hiding this comment.
i am wondering whether does test/registered/cpu/test_subblock_sparse_attention.py really work?
this is registered under cpu tests, but uses cuda device.
maybe we should just remove this file.
There was a problem hiding this comment.
I verified locally on a CPU-only machine and the test passes after installing diffusion dependencies. I can wait for the xeon CI to finish to double confirm this.
test_dispatch_is_resolved_once_per_device (__main__.TestSubBlockSparseAttentionDispatch) ... ok
test_dispatches_sm100 (__main__.TestSubBlockSparseAttentionDispatch) ... ok
test_rejects_unsupported_compute_capability (__main__.TestSubBlockSparseAttentionDispatch) ... ok
----------------------------------------------------------------------
Ran 3 tests in 0.002sThe test won't create tensor on CUDA, and it patches the torch.cuda.get_device_capability.
From what I understand, the test covers the dispatch logic of _get_subblock_sparse_attention_runner:
capability 9.0 → _sm90_sparse_attention, 10.0 → _sm100_sparse_attention, anything else → RuntimeError.
Shall we keep it and add the diffusion installation into Dockerfile?
There was a problem hiding this comment.
yes, we are going to add it in futuer after all.
…n-transport1 * 'main' of https://github.com/sgl-project/sglang: (326 commits) [diffusion] feat: cache LoRA-merged weights in files the page cache can hold (sgl-project#36062) [diffusion] Speed up LingBot high-quality VAE decode (sgl-project#36024) [diffusion] Honor XDG cache for model overlays (sgl-project#36019) Support streaming session on NPU (sgl-project#32597) fix(xpu): read enable_deterministic_inference from the config bag (sgl-project#36149) xeon ci fail fast strategy change (sgl-project#36146) [diffusion] Fix Hunyuan QKV pack indexing at production video shapes (sgl-project#36009) [diffusion] Refresh quality and BCG benchmark skills (sgl-project#36016) [MoE] Gather the cutlass MoE activation and its scales in one launch (sgl-project#34915) [diffusion] feat: add plain component weight overrides (sgl-project#36086) [diffusion] feat: support loading mixed w4a8 text encoders (sgl-project#36037) [diffusion] Default Hunyuan VAE to tiled decode (sgl-project#36012) fix(xpu): enable compressed-tensors FP8 W8A8 on XPU (RedHatAI FP8-dynamic models) (sgl-project#33057) chore: move cuda_vmm_utils.py under srt/utils/ (sgl-project#36053) [Intel XPU] Add xpu pass for biased_topk and hash_topk (sgl-project#33323) [CPU] Fix NUMA/core binding for DP ranks (sgl-project#32856) [Fix] Harden FlashAttention CUDA graph metadata bounds (sgl-project#35454) [XPU] Use a fused GDN kernel from sgl-kernel for Qwen3.5 (sgl-project#33354) [diffusion] Fuse LongCat-Image QKNorm and interleaved RoPE (sgl-project#35995) [diffusion] Keep LongLive2 components resident on large GPUs (sgl-project#35993) ... # Conflicts: # python/sglang/srt/multimodal/processors/base_processor.py # python/sglang/srt/server_args.py
Motivation
Before this PR,
ModelRunner.init_threads_bindingpicked the NUMA node / core group for a worker usingtp_rank. Under DP, DP rank 0 and DP rank 1 both resolve to tp_rank=0 and get bound to the same NUMA node/cores instead of being spread across the machine.This PR fixes this incorrect binding.
Modifications
init_threads_bindingnow takesnuma_index(the worker's global device id) andworld_size = dp_size * tp_size * pp_size, instead oftp_rankandtp_size, so every rank across every DP replica maps to a distinct NUMA node / core group.Removed the
world_size == len(threads_bind_list)equality assert for the explicitSGLANG_CPU_OMP_THREADS_BINDpath, replacing it with only the per-rank bounds check. This permits the router-mode case (localworld_size=1, multi-group bind string) while still catching a bind list that's too short.test/registered/cpu/test_subblock_sparse_attention.py, added in [diffusion][Minimax H3]support subblock sparse attention on SM90 #34680, fails in Xeon CPU CI withModuleNotFoundError: No module named 'imageio': importing it pulls insglang.multimodal_gen, which needsdiffusers/imageio/addictfrom the diffusion extra.Install
.[diffusion]instead of.indocker/xeon.Dockerfile— the extra is already declared inpython/pyproject_cpu.toml. Note this grows the CI image.Accuracy Tests
Speed Tests and Profiling
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ❌ Run #32679423999
Latest PR Test (Extra): ❌ Run #32679423888
Latest PR Test (AMD ROCm 7.2): ⏳ Run #32679424006