Skip to content

[CPU] Fix NUMA/core binding for DP ranks - #32856

Merged
mingfeima merged 20 commits into
sgl-project:mainfrom
chunyuan-w:chunyuan/pr_dp_fix
Aug 24, 2026
Merged

mingfeima merged 20 commits into
sgl-project:mainfrom
chunyuan-w:chunyuan/pr_dp_fix

Conversation

@chunyuan-w

@chunyuan-w chunyuan-w commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Motivation

Before this PR, ModelRunner.init_threads_binding picked the NUMA node / core group for a worker using tp_rank. Under DP, DP rank 0 and DP rank 1 both resolve to tp_rank=0 and get bound to the same NUMA node/cores instead of being spread across the machine.
This PR fixes this incorrect binding.

Modifications

  1. init_threads_binding now takes numa_index (the worker's global device id) and world_size = dp_size * tp_size * pp_size, instead of tp_rank and tp_size, so every rank across every DP replica maps to a distinct NUMA node / core group.

    Removed the world_size == len(threads_bind_list) equality assert for the explicit SGLANG_CPU_OMP_THREADS_BIND path, replacing it with only the per-rank bounds check. This permits the router-mode case (local world_size=1, multi-group bind string) while still catching a bind list that's too short.

  2. test/registered/cpu/test_subblock_sparse_attention.py, added in [diffusion][Minimax H3]support subblock sparse attention on SM90 #34680, fails in Xeon CPU CI with ModuleNotFoundError: No module named 'imageio': importing it pulls in sglang.multimodal_gen, which needs diffusers/imageio/addict from the diffusion extra.

    Install .[diffusion] instead of . in docker/xeon.Dockerfile — the extra is already declared in python/pyproject_cpu.toml. Note this grows the CI image.

Accuracy Tests

Speed Tests and Profiling

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Run #32679423999
Latest PR Test (Extra): ❌ Run #32679423888
Latest PR Test (AMD ROCm 7.2): ⏳ Run #32679424006

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@chunyuan-w
chunyuan-w marked this pull request as ready for review August 5, 2026 01:53
@chunyuan-w

Copy link
Copy Markdown
Contributor Author

/tag-run-ci-label

@github-actions github-actions Bot added the run-ci label Aug 5, 2026
@mingfeima mingfeima added intel cpu cpu backend performance optimization labels Aug 18, 2026
Comment thread test/registered/cpu/test_binding.py
Comment thread python/sglang/srt/model_executor/model_runner.py
Comment thread docker/xeon.Dockerfile
cd python && \
cp pyproject_cpu.toml pyproject.toml && \
uv pip install . && \
uv pip install ".[diffusion]" && \

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change is needed to fix Xeon CI failure. Please check details in PR description (Modifications 2)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i am wondering whether does test/registered/cpu/test_subblock_sparse_attention.py really work?
this is registered under cpu tests, but uses cuda device.

maybe we should just remove this file.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I verified locally on a CPU-only machine and the test passes after installing diffusion dependencies. I can wait for the xeon CI to finish to double confirm this.

test_dispatch_is_resolved_once_per_device (__main__.TestSubBlockSparseAttentionDispatch) ... ok
test_dispatches_sm100 (__main__.TestSubBlockSparseAttentionDispatch) ... ok
test_rejects_unsupported_compute_capability (__main__.TestSubBlockSparseAttentionDispatch) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.002s

The test won't create tensor on CUDA, and it patches the torch.cuda.get_device_capability.
From what I understand, the test covers the dispatch logic of _get_subblock_sparse_attention_runner:
capability 9.0 → _sm90_sparse_attention, 10.0 → _sm100_sparse_attention, anything else → RuntimeError.

Shall we keep it and add the diffusion installation into Dockerfile?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, we are going to add it in futuer after all.

@mingfeima
mingfeima merged commit 1daa94a into sgl-project:main Aug 24, 2026
111 of 138 checks passed
longxin9715 added a commit to longxin9715/sglang that referenced this pull request Aug 24, 2026
…n-transport1

* 'main' of https://github.com/sgl-project/sglang: (326 commits)
  [diffusion] feat: cache LoRA-merged weights in files the page cache can hold (sgl-project#36062)
  [diffusion] Speed up LingBot high-quality VAE decode (sgl-project#36024)
  [diffusion] Honor XDG cache for model overlays (sgl-project#36019)
  Support streaming session on NPU (sgl-project#32597)
  fix(xpu): read enable_deterministic_inference from the config bag (sgl-project#36149)
  xeon ci fail fast strategy change (sgl-project#36146)
  [diffusion] Fix Hunyuan QKV pack indexing at production video shapes (sgl-project#36009)
  [diffusion] Refresh quality and BCG benchmark skills (sgl-project#36016)
  [MoE] Gather the cutlass MoE activation and its scales in one launch (sgl-project#34915)
  [diffusion] feat: add plain component weight overrides (sgl-project#36086)
  [diffusion] feat: support loading mixed w4a8 text encoders (sgl-project#36037)
  [diffusion] Default Hunyuan VAE to tiled decode (sgl-project#36012)
  fix(xpu): enable compressed-tensors FP8 W8A8 on XPU (RedHatAI FP8-dynamic models) (sgl-project#33057)
  chore: move cuda_vmm_utils.py under srt/utils/ (sgl-project#36053)
  [Intel XPU] Add xpu pass for biased_topk and hash_topk (sgl-project#33323)
  [CPU] Fix NUMA/core binding for DP ranks (sgl-project#32856)
  [Fix] Harden FlashAttention CUDA graph metadata bounds (sgl-project#35454)
  [XPU] Use a fused GDN kernel from sgl-kernel for Qwen3.5 (sgl-project#33354)
  [diffusion] Fuse LongCat-Image QKNorm and interleaved RoPE (sgl-project#35995)
  [diffusion] Keep LongLive2 components resident on large GPUs (sgl-project#35993)
  ...

# Conflicts:
#	python/sglang/srt/multimodal/processors/base_processor.py
#	python/sglang/srt/server_args.py
saturn-acc pushed a commit to saturn-acc/sglang that referenced this pull request Aug 31, 2026
jakki-amd pushed a commit to jakki-amd/sglang that referenced this pull request Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cpu cpu backend performance optimization intel run-ci

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants