Skip to content

[Diffusion] Recover IPC JIT initialization after interrupted builds - #39034

Merged
mickqian merged 1 commit into
sgl-project:mainfrom
mickqian:codex/fix-diffusion-ipc-jit-recovery
Sep 11, 2026
Merged

mickqian merged 1 commit into
sgl-project:mainfrom
mickqian:codex/fix-diffusion-ipc-jit-recovery

Conversation

@mickqian

@mickqian mickqian commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Motivation

The two-rank Ulysses IPC synchronization loader uses torch.utils.cpp_extension.load_inline with persistent per-device build directories. If a build is interrupted, its FileBaton lock can remain after the owner exits. Later workers wait indefinitely in FileBaton.wait() during the first IPC operation, preventing server warmup from completing.

This addresses a reproducible interrupted-build failure mode investigated after a diffusion CI warmup timeout. The historical job did not capture a blocked-worker stack, so this PR does not claim conclusive attribution of that particular timeout.

Modifications

  • Load the existing IPC sequence-counter kernels through SGLang's shared JIT loader, which uses process-lifetime flock locking and atomic cache publication.
  • Preserve the IPC signaling protocol, peer timeout propagation, current-stream execution, and CUDA Graph support.
  • Add a bounded two-GPU regression test with orphaned legacy lock files and an isolated cold JIT cache. Register it in the diffusion two-GPU CUDA suite, once on partition 0.
  • Do not delete existing caches or increase CI timeouts.

Accuracy Tests

The added regression checks exact IPC input/output round trips on a nondefault stream and three CUDA Graph replays. NVIDIA CI results are pending on this PR.

Speed Tests and Profiling

This is an initialization reliability fix, not a throughput optimization. No speedup claim.

Checklist

  • Follow the SGLang code style and pre-commit requirements.
  • Add a registered regression test.
  • No user-facing API or documentation change required.
  • Complete NVIDIA CI validation on this PR head.

CI States

Latest PR Test (Base): ⏳ Run #34569843379
Latest PR Test (Extra): ❌ Run #34569843057
Latest PR Test (AMD ROCm 10): ❌ Run #34569843327

@mickqian

Copy link
Copy Markdown
Collaborator Author

/tag-and-rerun-ci

@mickqian
mickqian merged commit 358c163 into sgl-project:main Sep 11, 2026
278 of 331 checks passed
mqhc2020 pushed a commit to mqhc2020/sglang that referenced this pull request Sep 15, 2026
…lds (sgl-project#39034)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant