Skip to content

[Bugfix][ROCm] Fix stale model_config test stub breaking MoE multi-GPU combos; add mi300_2 CI job - #61260

Closed
divakar-amd wants to merge 2 commits into
vllm-project:mainfrom
divakar-amd:fix/moe-multigpu-test-stub-and-ci
Closed

divakar-amd wants to merge 2 commits into
vllm-project:mainfrom
divakar-amd:fix/moe-multigpu-test-stub-and-ci

Conversation

@divakar-amd

@divakar-amd divakar-amd commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

This PR does 2 things:

  1. Fixes a multi-gpu error in mk
  2. Adds mi300_2 mirror to run multi-gpu mk tests

Why

Running VLLM_ROCM_USE_AITER=1 pytest tests/kernels/moe/test_modular_kernel_combinations.py -k "aiter" on an MI300 (2-GPU) box, test_modular_kernel_combinations_multigpu[...] died with:

AttributeError: 'types.SimpleNamespace' object has no attribute 'sleep_mode_offload_cudagraph'

tests/kernels/moe/modular_kernel_tools/common.py fakes vllm_config.model_config as a bare SimpleNamespace(enforce_eager=True, is_moe=True). #59160 later added VllmConfig.use_cumem_cudagraph_pool, which reads model_config.sleep_mode_offload_cudagraph / .enable_sleep_mode / .sleep_mode_backend — fields the stub predates. That property is evaluated in CudaCommunicator.__init__ whenever a GroupCoordinator with world_size > 1 is built under set_current_vllm_config, which the multi-GPU DP test path does.

This fixes the stub by adding the three fields with the same defaults as real ModelConfig (False, False, "cumem").

Why this wasn't caught by existing CI

The only AMD CI job covering this file, ":amd: (MI300) MoE Kernels Shard %N", runs on the single-GPU mi300_1 agent pool. test_modular_kernel_combinations_multigpu always needs world_size=2, so it self-skips via its own current_platform.device_count() < world_size check before ever reaching the broken code path — the job shows green while never actually exercising the multi-GPU combinations.

This is the same bug class as #53840 / #53925 (ROCm CI jobs under-provisioned relative to a test's declared GPU requirement), but this specific instance wasn't in that audit's list — their scan matched the standard "Need at least N GPUs to run the test." skip message, while this test uses a differently-worded custom message ("Not enough GPUs available to run, got ... expected ..."), so it didn't show up in their log grep.

What's added

  • tests/kernels/moe/modular_kernel_tools/common.py: add the three missing sleep_mode_* fields to the model_config stub.
  • .buildkite/test-amd.yaml: new ":amd: (MI300) MoE Kernels Multi-GPU" job on mi300_2 (num_gpus: 2) running just the -k "multigpu" cases, so they actually execute instead of perpetually skipping. The existing mi300_1 job now deselects multigpu cases (-k "not multigpu") so the two jobs don't both claim the same test IDs — mirrors the "carved out, nothing double-runs" pattern from [ROCm][CI] Add multi-GPU test groups for nightly-log-confirmed skips (#53840) #53925.

Test commands run and results

Full suite, MI300 (2 GPUs), before and after the fix:

VLLM_ROCM_USE_AITER=1 pytest -s -v tests/kernels/moe/test_modular_kernel_combinations.py -k "aiter"

Also spot-checked individually:

  • test_aiter_moe_token_padding_garbage_rows[single_nan-unquantized] — PASSED
  • test_modular_kernel_combinations_multigpu[2048-1024-32-dtype1-None-MoEActivation.SILU-DeepEPHTPrepareAndFinalize-AiterExperts-2] — PASSED (previously crashed)

This change doesn't affect model output/accuracy, so no model eval was run.

🤖 Generated with Claude Code

divakar-amd and others added 2 commits October 12, 2026 18:22
…ts; add mi300_2 CI coverage

test_modular_kernel_combinations.py fakes vllm_config.model_config as a
bare SimpleNamespace missing the sleep-mode fields added by vllm-project#59160
(sleep_mode_offload_cudagraph, enable_sleep_mode, sleep_mode_backend).
VllmConfig.use_cumem_cudagraph_pool reads those fields, and
CudaCommunicator.__init__ evaluates that property whenever a
GroupCoordinator with world_size > 1 is built, so any DP/TP group
construction in these tests raised AttributeError.

This went uncaught because the only AMD CI job covering this file
(":amd: (MI300) MoE Kernels Shard %N") runs on the single-GPU mi300_1
pool, so test_modular_kernel_combinations_multigpu (world_size=2)
self-skips via its own GPU-count check before ever reaching the
affected code path. Add a dedicated mi300_2 job so the multi-GPU
combinations actually run in CI.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Divakar Verma <divakar.verma@amd.com>
They self-skip there anyway (insufficient GPUs) and now run for real
in the new mi300_2 job; excluding them avoids the two jobs both
claiming the same test IDs, matching the "carved out, nothing
double-runs" pattern used in vllm-project#53925.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Divakar Verma <divakar.verma@amd.com>
@mergify mergify Bot added ci/build rocm Related to AMD ROCm bug Something isn't working labels Oct 12, 2026
@divakar-amd

Copy link
Copy Markdown
Contributor Author

Moved these fixes to #60322. Hence, Closing this draft PR

@github-project-automation github-project-automation Bot moved this from Todo to Done in AMD Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ci/build rocm Related to AMD ROCm

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant