Skip to content

[cherrypick from #36389] [LoRA] Let DP-attention LoRA run idle ranks and eager prefill (partial) - #41507

Open
yushengsu-thu wants to merge 69 commits into
sglang-milesfrom
yusheng/cherrypick-36389-partial-sglang-miles
Open

yushengsu-thu wants to merge 69 commits into
sglang-milesfrom
yusheng/cherrypick-36389-partial-sglang-miles

Conversation

@yushengsu-thu

@yushengsu-thu yushengsu-thu commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Port two hunks of #36389 to the sglang-miles branch: its managers/scheduler_components/dp_attn.py and lora/utils.py changes. Without them, a LoRA server with --enable-dp-attention dies on its first batch:

  • AttributeError: 'EagerRunner' object has no attribute 'can_replay_locally' in _local_prefill_cuda_graph_vote. With LoRA the prefill runner is the EagerRunner fallback, and the vote now returns False for it.
  • ValueError: Unsupported forward mode: 4 in get_batch_token_counts, from MoE-LoRA batch preparation. DP ranks without requests run IDLE batches, which get_batch_token_counts and generate_sequence_lengths now accept.

Port notes

Testing


CI States

Latest PR Test (Base): ❌ Run #36483117678
Latest PR Test (Extra): ❌ Run #36483116907
Latest PR Test (AMD ROCm 10): ❌ Run #36483117405

ishandhanani and others added 30 commits September 16, 2026 15:31
Co-authored-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
…s in exact-token preprocessing (#30368) (#40009)

Signed-off-by: Zhuangcheng(Jesse) Gu <zcgu@connect.hku.hk>
Co-authored-by: Zhuangcheng(Jesse) Gu <40918450+Chokoyo@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
…-gather case (#39899) (#40012)

Co-authored-by: Byron Hsu <byronhsu1230@gmail.com>
Co-authored-by: Byron Hsu <byron+per@periodiclabs.ai>
… dropping swiglu_limit clamped SwiGLU activation (#39920) (#40035)

Co-authored-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>
…39915) (#40137)

Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
Co-authored-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
…ense) (#18639)

(cherry picked from commit a6cc76d)
(cherry picked from commit 29d5285)
…ort (#18642)

(cherry picked from commit f78421f)
(cherry picked from commit 72e9b93)
(cherry picked from commit cd31ae6)
(cherry picked from commit 77b31fd)
(cherry picked from commit f9747a5)
(cherry picked from commit 6e7483c)
(cherry picked from commit c4b6f74)
(cherry picked from commit 049efd7)
(cherry picked from commit d669fb3)
(cherry picked from commit e18e48c)
(cherry picked from commit ab3c05f)
(cherry picked from commit 17c3be8)
(cherry picked from commit 16b8d2e)
(cherry picked from commit 4661ad8)
(cherry picked from commit 3a4659e)
(cherry picked from commit 547283d)
…h/RL fixes (#25141, #29874, #31251)

(cherry picked from commit 7fea23f)
(cherry picked from commit 00639c4)
…27728)

(cherry picked from commit 3e32648)
(cherry picked from commit b63c007)
…psert (#27268, #31759, #30913)

(cherry picked from commit f8a11ee)
(cherry picked from commit 8528623)
…capture + meta_info threading

(cherry picked from commit 2aee68b)
(cherry picked from commit 9e6e62c)
… for spec draft worker(s) (#27749, #28575, #18565, #22663, #28001, #29675, #27750)

(cherry picked from commit 4290cf4)
(cherry picked from commit 3c0c432)
… from RL weight check (#29339)

(cherry picked from commit 5d36347)
(cherry picked from commit 3e92eeb)
…ht updates (#30421)

(cherry picked from commit b09c3c7)
(cherry picked from commit 02715e0)
(cherry picked from commit 5944cf6)
(cherry picked from commit f2ba83f)
… into a host-local checkpoint (#30366, #28524)

(cherry picked from commit f09356f)
(cherry picked from commit 00a2ca2)
… multi-lora needs (#30912)

(cherry picked from commit 80aa4ec)
(cherry picked from commit f0cbd48)
… in retract (#31962)

(cherry picked from commit e9ed3cb)
(cherry picked from commit 9fa67e5)
…h the merged per-role payload

(cherry picked from commit 8fea606)
(cherry picked from commit a264f30)
…t to v0.5.16's parallel state

(cherry picked from commit 19207ba)
(cherry picked from commit 174ddab)
(cherry picked from commit 5296432)
(cherry picked from commit 4c974df)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.