[CI] Fix test_resolution_is_reproducible after cuda_ipc became opt-in - #34746
Merged
hnyls2002 merged 1 commit intoAug 13, 2026
Merged
Conversation
#34662 made CUDA IPC opt-in: the `nnodes == 1` arm of `_handle_multimodal_feature_transport` now resolves to `cpu` instead of `cuda_ipc`. `test_a_resolution_does_not_leak_into_the_next` still expected `cuda_ipc` and failed on the GPU registrations with `'cpu' != 'cuda_ipc'`; the CPU and ROCm suites passed because they already expect `cpu`. Flipping the expectation to `cpu` alone would have hidden a second consequence of that change. With auto-resolution picking `cpu` everywhere, the multimodal intermediate now writes exactly what the next resolution would have picked on its own, so the `_STICKY_ACROSS_RESOLUTIONS` exception stopped pinning anything observable. Add a CUDA-gated `multimodal_cuda_ipc` shape that opts in explicitly: post-#34662 it is the only construction under which the SGLANG_USE_CUDA_IPC_TRANSPORT carry is visible, and the only coverage left of the handler's `cuda_ipc` arm. It is wired into `_SHAPES` (dual-resolve and sibling matrices) as well as the leak test's intermediates. Also generalize the sticky assertion to run for every intermediate instead of one label, and derive the expected value from what the intermediate itself resolved to rather than from its kwargs. What carries is the legacy boolean, which the handler writes as 1 only for `cuda_ipc`, so any other selection -- including a `cuda_vmm` shape if one is ever added -- carries `cpu`. Verified on a CUDA host: 4 tests / 16 subtests pass, and the leak case still fails if the handler's env write is stubbed out. The gates drop out to 10 subtests without a visible NVIDIA device.
Collaborator
Author
|
/rerun-test test/registered/unit/server_args/test_resolution_is_reproducible.py |
Contributor
|
Results for 🚀 |
Collaborator
|
/tag-and-rerun-ci |
hnyls2002
deleted the
mmangkad/fix-mm-feature-transport-resolution-test
branch
August 13, 2026 19:47
saturn-acc
pushed a commit
to saturn-acc/sglang
that referenced
this pull request
Aug 16, 2026
Atituiset
pushed a commit
to Atituiset/sglang
that referenced
this pull request
Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
test/registered/unit/server_args/test_resolution_is_reproducible.pyis failing on the GPUregistrations (seen on
base-b-test-1-gpu-small,1-gpu-5090):#34662 (
69bf601e3c, "fix: restore VLM nightly regression coverage") made CUDA IPC opt-in: thennodes == 1arm of_handle_multimodal_feature_transportnow resolves tocpuinstead ofcuda_ipc, because even an idle IPC pool consumes HBM that would otherwise back the KV cache. ThatPR updated the CLI help text and the
TestMultimodalFeatureTransportcases intest_server_args.py,but this test still asserted
expected = "cuda_ipc" if is_cuda() else "cpu". The CPU and ROCmregistrations kept passing because they already expect
cpu, so only the CUDA runners went red.The policy change itself is intentional, so the fix belongs in the test.
There is a second, quieter consequence worth calling out. With auto-resolution now picking
cpueverywhere, the multimodal intermediate writes exactly what the next resolution would have picked on
its own, so the documented
_STICKY_ACROSS_RESOLUTIONSexception no longer pins anything observable.Flipping the expectation to
cpuand stopping there would have left a green but vacuous assertion.Modifications
Test-only; no runtime behaviour is changed.
Add a CUDA-gated
multimodal_cuda_ipcshape (_CUDA_IPC_SHAPES) that requestsmm_feature_transport="cuda_ipc"explicitly. Post-fix: restore VLM nightly regression coverage #34662 this is the only construction underwhich the
SGLANG_USE_CUDA_IPC_TRANSPORTcarry is observable — the handler writes1only forcuda_ipc, and the following text-only resolution adopts it via thelegacy_ipc_is_setbranch —and it is also the only remaining coverage of the handler's
cuda_ipcarm, which noauto-resolution reaches any more. It is wired into
_SHAPES(dual-resolve and sibling matrices)as well as the leak test's
intermediates. Gated onis_cuda()rather thantorch.cuda.is_available()because the handler raises forcuda_ipcoff NVIDIA CUDA, ROCmincluded.
Generalize the sticky assertion to run for every intermediate instead of only the
label == "multimodal"one (every resolution runs the transport handler, soafter.mm_feature_transportis pinned for all of them), and derive the expected value from whatthe intermediate itself resolved to rather than from its kwargs. What carries is the legacy
boolean, not the tri-state field, so any non-
cuda_ipcselection — including acuda_vmmshapeif one is ever added — carries
cpu.Verification
intermediate='multimodal'— reproduces the CI failureset("0")intermediate='multimodal_cuda_ipc', so the new shape genuinely guards the carry rather than passing triviallycuda_vmmintermediatecuda_vmmunder a kwargs-derived expectationtest_server_args.py(142),test_mm_process_config.py(42)pre-commit run --files <file>Accuracy Tests
Not applicable — this changes a
ServerArgsresolution unit test only; no kernel, model forward, oroutput path is touched.
Speed Tests and Profiling
Not applicable — no inference path is touched.
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): 🚫 Run #31710106430
Latest PR Test (Extra): ❌ Run #31710105883