[CI] Refactor CI workflow into stage classified by hardware and gpu num, fix buggy tests

[guapisolo]

(written by human)

Motivation

Existing CI workflow is classified by label. But when scaling up to different type of hardware, (h100, h200 or even b/gb), the current design sucks, hard to delegate tasks to different runners.

Design

Worker split

Split to 3 types of workers, each include:

• 8-gpu-h100:
  • Tests that must need a 8 gpu env, like test different paralism (tp pp cp >=2 at the same time).
  • Tests that need large host memory (> 1TB), degrading optimizer precision to fp16 still insufficient. (especially tests with deepep, but not sure about the reason.)
• 4-gpu-h200:
  • The most cost-efficient setting, covering most of test cases.
• 2-gpu-h200:
  • Low gpu utlis tests. Add more gpu never speed up these tests.

These three worker matching to the name of stages. And remove old gpu_exec_lock.py design.

Test split

• Split common launch logic in _common.py. Reused by other CI. e.g. tests/e2e/megatron/test_qwen3_30B_A3B/_common.py
• The test discovery logic skips all tests starting with an underscore, allowing these tests to avoid registering any CI.
• This can reduce granularity of test cases and improve perf a lot .

Hardcode removal

• Change most test cases to determine the GPU count from the NUM_GPUS label or CaseConfig, and infer parallelism settings and train/rollout GPU placement from formulas instead of hardcoding them. This will make future CI migration easier.

Keep PR label-trigger

Contributor still need to add labels manually to trigger CI. No auto-detect logic.

Skills

Improved runner setup skills. one-click to setup different type of runner.

Existing CI Bug fixes

Notice: This PR is mostly about CI workflow refactor, only including naive CI bug fix.

• Occasionally gets stuck at wandb.init(), with no response for 90 seconds. Fix by extending default time.
• Existing r3 check never run with --use-kl-loss, so the r3 check for actor & ref at step 0 is always skipped. Add back the check and loose threshold to 5e-3.
• Reduce --num-rollout for megatron test from 3 to 2. 3 is not effective but consume more gpus.
• Fix incorrect mount of /tmp folder into container, which cause race condition of writing ray logs.

Other buggy CI are disabled and easy repro on local machine, and FIXME will cover them.

[gemini-code-assist]

Code Review

This pull request refactors the CI test registration and filtering system to implement a label-based execution model. Key changes include the introduction of a canonical label registry in tests/ci/labels.py, updating the CIRegistry dataclass to support labels and always_on flags, and refactoring the AST-based test collection logic to validate these labels. The execution logic in run_suite.py now supports filtering tests by intersecting PR labels with test metadata, and the suite taxonomy has been updated to a new naming convention. Additionally, numerous test files were migrated to the new registration signature. Feedback was provided regarding the readability of the control flow in the collect_tests function, suggesting the restoration of an else block to make the logic more explicit.

[gemini-code-assist]

For improved readability and to make the control flow more explicit, it would be better to restore the else block here. While the current implementation is functionally correct because raise ValueError(msg) will exit the function, an else block clearly separates the code paths for sanity_check=True and sanity_check=False.

Suggested change

	if sanity_check:
	raise ValueError(msg)
	else:
	warnings.warn(msg, stacklevel=2)
	continue
	warnings.warn(msg, stacklevel=2)
	continue
	if sanity_check:
	raise ValueError(msg)
	else:
	warnings.warn(msg, stacklevel=2)
	continue

[guapisolo]

synced offline with Yusheng