Repository navigation
[Bugfix][ROCm] Fix stale model_config test stub breaking MoE multi-GPU combos; add mi300_2 CI job - #61260
Closed
divakar-amd wants to merge 2 commits into
Closed
[Bugfix][ROCm] Fix stale model_config test stub breaking MoE multi-GPU combos; add mi300_2 CI job#61260divakar-amd wants to merge 2 commits into
divakar-amd wants to merge 2 commits into
Conversation
…ts; add mi300_2 CI coverage test_modular_kernel_combinations.py fakes vllm_config.model_config as a bare SimpleNamespace missing the sleep-mode fields added by vllm-project#59160 (sleep_mode_offload_cudagraph, enable_sleep_mode, sleep_mode_backend). VllmConfig.use_cumem_cudagraph_pool reads those fields, and CudaCommunicator.__init__ evaluates that property whenever a GroupCoordinator with world_size > 1 is built, so any DP/TP group construction in these tests raised AttributeError. This went uncaught because the only AMD CI job covering this file (":amd: (MI300) MoE Kernels Shard %N") runs on the single-GPU mi300_1 pool, so test_modular_kernel_combinations_multigpu (world_size=2) self-skips via its own GPU-count check before ever reaching the affected code path. Add a dedicated mi300_2 job so the multi-GPU combinations actually run in CI. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Divakar Verma <divakar.verma@amd.com>
They self-skip there anyway (insufficient GPUs) and now run for real in the new mi300_2 job; excluding them avoids the two jobs both claiming the same test IDs, matching the "carved out, nothing double-runs" pattern used in vllm-project#53925. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Divakar Verma <divakar.verma@amd.com>
Contributor
Author
|
Moved these fixes to #60322. Hence, Closing this draft PR |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR does 2 things:
Why
Running
VLLM_ROCM_USE_AITER=1 pytest tests/kernels/moe/test_modular_kernel_combinations.py -k "aiter"on an MI300 (2-GPU) box,test_modular_kernel_combinations_multigpu[...]died with:tests/kernels/moe/modular_kernel_tools/common.pyfakesvllm_config.model_configas a bareSimpleNamespace(enforce_eager=True, is_moe=True). #59160 later addedVllmConfig.use_cumem_cudagraph_pool, which readsmodel_config.sleep_mode_offload_cudagraph/.enable_sleep_mode/.sleep_mode_backend— fields the stub predates. That property is evaluated inCudaCommunicator.__init__whenever aGroupCoordinatorwithworld_size > 1is built underset_current_vllm_config, which the multi-GPU DP test path does.This fixes the stub by adding the three fields with the same defaults as real
ModelConfig(False,False,"cumem").Why this wasn't caught by existing CI
The only AMD CI job covering this file,
":amd: (MI300) MoE Kernels Shard %N", runs on the single-GPUmi300_1agent pool.test_modular_kernel_combinations_multigpualways needsworld_size=2, so it self-skips via its owncurrent_platform.device_count() < world_sizecheck before ever reaching the broken code path — the job shows green while never actually exercising the multi-GPU combinations.This is the same bug class as #53840 / #53925 (ROCm CI jobs under-provisioned relative to a test's declared GPU requirement), but this specific instance wasn't in that audit's list — their scan matched the standard
"Need at least N GPUs to run the test."skip message, while this test uses a differently-worded custom message ("Not enough GPUs available to run, got ... expected ..."), so it didn't show up in their log grep.What's added
tests/kernels/moe/modular_kernel_tools/common.py: add the three missingsleep_mode_*fields to themodel_configstub..buildkite/test-amd.yaml: new":amd: (MI300) MoE Kernels Multi-GPU"job onmi300_2(num_gpus: 2) running just the-k "multigpu"cases, so they actually execute instead of perpetually skipping. The existingmi300_1job now deselectsmultigpucases (-k "not multigpu") so the two jobs don't both claim the same test IDs — mirrors the "carved out, nothing double-runs" pattern from [ROCm][CI] Add multi-GPU test groups for nightly-log-confirmed skips (#53840) #53925.Test commands run and results
Full suite, MI300 (2 GPUs), before and after the fix:
test_modular_kernel_combinations_multigpu[...]processes died with theAttributeErrorabove (see failure traceback in PR context).54 passed, 6 skipped, 21 deselected(skips are pre-existing/expected — mxfp4 unsupported here, and known AITER×DeepEP-HT illegal-memory-access cases tracked in [Bug][ROCm] AiterExperts + DeepEPHT/Mori PrepareAndFinalize: illegal memory access at world_size=2 (gfx942) #57029).Also spot-checked individually:
test_aiter_moe_token_padding_garbage_rows[single_nan-unquantized]— PASSEDtest_modular_kernel_combinations_multigpu[2048-1024-32-dtype1-None-MoEActivation.SILU-DeepEPHTPrepareAndFinalize-AiterExperts-2]— PASSED (previously crashed)This change doesn't affect model output/accuracy, so no model eval was run.
🤖 Generated with Claude Code