Run GPT-OSS-120B on 2 nodes, and let the deployment set the perf knobs by mkuznet1 · Pull Request #271 · ROCm/MAD

mkuznet1 · 2026-10-05T15:14:27Z

What this is for

Widening the RCCL CI to a second training workload. GPT-OSS-120B is the
candidate, and two things in primus_megatron-lm_benchmark_report.sh stood
between it and a usable perf gate.

1. GPT-OSS-120B could not run at 16 GPUs

The micro_batch_size / recompute_num_layers overrides were keyed on DEVICE
alone, so MI355X always ran the shipped config (mbs 8, recompute_num_layers 4).
That config does not fit at the 16-GPU topology minimum. TP1 x PP2 fixes model
parallelism at 2, so DP is 8 at 16 GPUs against 16 at 32, and the distributed
optimizer's 12/DP bytes/param cost about 6 GB/GPU more. Measured, not inferred:
Megatron's own theoretical report gives 377448 MB of weight+optimizer state at
DP16 against 419386 MB at DP8, with the activation term byte-identical, and the
ratio is exactly (6 + 12/8) / (6 + 12/16).

Two jobs confirm the failure mode. One OOM'd in the MoE grouped GEMM
(grouped_linear.py:206 via experts.py:386, inside _checkpointed_forward)
with 428 MiB free on a 288GB card; a retry with
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True bought exactly one iteration
and then died in the same place.

The fix triggers the existing overrides at NUM_GPUS <= 16 regardless of HBM
size. That restores the configuration that AICOMNET-141 measured on 2 x 8 GPUs
on both gfx942 and gfx955, where it came from a baked-in MI300X config directory
selected by PRIMUS_CONFIG_DEVICE; this branch expresses the same end state as
CLI overrides on top of the MI355X config.

Verified end to end on 2 x 8 MI355X with stock rocm/primus:v26.4, both
precisions, 10/10 iterations, loss decreasing:

precision	mbs	tok/s/GPU	avg	TFLOPS/GPU	peak HBM
BF16	2	8281.8	7331.0	318.8	182.60 GB
FP8	1	4529.5	3978.5	174.4	191.60 GB

BF16 lands within 3.3% of the independent AICOMNET-141 measurement on another
cluster of the same architecture.

The 32-GPU MI355X path is unchanged, so existing 4-node baselines stand.
global_batch_size 2048 still divides evenly at mbs 2 and 1, so
scaleout_gbs_override remains a no-op.

2. `train_iters` and the profiler were unreachable from outside

run_primus() is the launch point for six models: GPT-OSS-120B,
Llama-3.1-8B/70B/405B, Llama-4-Scout-17B-16E and Kimi-K2-Thinking, which
covers every Primus workload the RCCL CI measures. The other branches call
runner/primus-cli direct themselves, the knobs do not reach them, and the
script prints a [WARN] when they are set for such a model. But nothing
outside the script could reach either setting: they come from the shipped
config, no env var was consulted, and run.sh parses only --model_repo.

This matters for a perf gate. The GPT-OSS-120B config ships profile: true with
the window on iterations 6-7. At 2 nodes that costs 3.2x the baseline iteration
on BF16 and 3.7x on FP8, and 8-9 stay elevated while the trace is written — with
train_iters 10 that is 3 of the 8 measured iterations:

BF16  121691 93266 64215 65772 63295 70850 [207608] 65783 65116 63306
FP8   148723 132941 116363 115813 119327 144448 [433676] 116609 115557 115749

PRIMUS_TRAIN_ITERS and PRIMUS_DISABLE_PROFILE are now read from the
environment, which madengine injects from context.docker_env_vars. Both are
optional, and unset leaves the command byte-identical, so no other model changes
behaviour.

Checked inside rocm/primus:v26.4 before relying on it: Primus coerces these
overrides rather than assigning strings, so --profile false yields
profile=False of type bool and not the truthy string "false";
--train_iters 15 yields int; and an unrecognised key is reported as an
unknown override instead of being silently dropped.

3. Tests

scripts/primus_megatron-lm/tests/test_primus_overrides.sh, following the
convention of scripts/primus_train/tests: the two blocks are extracted from
the script and evaluated in isolation, so the test needs neither a GPU nor
Primus. run_primus() is exercised by shadowing the bash command with a shell
function, which captures the assembled argv without reaching the launcher.

It pins the properties that a later edit would silently break — unset knobs
leave the command untouched, the knobs come last so the final occurrence of a
key wins, MI355X is untouched at 32 GPUs and picks the overrides up at 16, and
192GB parts keep them at every scale. Confirmed to actually fail on regression
by mutating the script three ways: reverting the world-size clause, flipping the
knob order, and disabling the profiler unconditionally.

bash scripts/primus_megatron-lm/tests/test_primus_overrides.sh

Not in this PR

The numbers above were measured with the profiler still on, so they are proof
that the run works, not a CI baseline. A baseline wants a run with
PRIMUS_DISABLE_PROFILE=1 and PRIMUS_TRAIN_ITERS around 15.

The micro_batch_size / recompute_num_layers overrides were keyed on DEVICE alone, so MI355X always ran the shipped config (mbs 8, recompute 4). That config does not fit at 16 GPUs: TP1 x PP2 fixes model parallelism at 2, so DP is 8 there against 16 at 32 GPUs, and the distributed optimizer's 12/DP bytes/param grow by ~6 GB/GPU. Job 56918 OOM'd in the MoE grouped GEMM with 428 MiB free on a 288GB card, and expandable_segments only moved the failure one iteration later. Trigger the same overrides at NUM_GPUS <= 16 regardless of HBM size. This restores the configuration AICOMNET-141 measured on 2 x 8 GPUs on both gfx942 and gfx955; there it came from a baked-in MI300X config directory selected by PRIMUS_CONFIG_DEVICE, which this branch instead expresses as CLI overrides on top of the MI355X config. The 32-GPU MI355X path is unchanged, so the existing 4-node baseline stands. GBS 2048 still divides evenly at mbs 2 and 1, so scaleout_gbs_override remains a no-op. Co-authored-by: Cursor <cursoragent@cursor.com>

…er off run_primus() is the single launch point for every Primus model here, but nothing outside the script could reach train_iters or the profiler: they come from the shipped config and no env var was consulted. For a perf gate both matter. The GPT-OSS-120B config ships profile: true with the window on iterations 6-7, which costs 3.2x the baseline iteration on BF16 and 3.7x on FP8 at 2 nodes, and leaves 8-9 elevated while the trace is written -- with train_iters 10 that pollutes 3 of the 8 measured iterations. PRIMUS_TRAIN_ITERS and PRIMUS_DISABLE_PROFILE are read from the environment, which madengine injects from context.docker_env_vars. Both are optional and unset leaves the command byte-identical, so no other model changes. Verified inside rocm/primus:v26.4 that Primus coerces these overrides rather than assigning strings: --profile false yields profile=False of type bool, --train_iters 15 yields int 15, and an unrecognised key is reported as an unknown override instead of being silently dropped. Co-authored-by: Cursor <cursoragent@cursor.com>

Both blocks changed in this branch decide what does *not* get added to the command, which is the part a later edit silently breaks. Follows the convention of scripts/primus_train/tests: the blocks are extracted from the script and evaluated in isolation, so no GPU and no Primus are needed. run_primus() is exercised by shadowing the `bash` command with a shell function, which captures the assembled argv without reaching the launcher. The checks pin the two properties worth keeping: unset knobs leave the command byte-identical (an explicit PRIMUS_DISABLE_PROFILE=0 included), and the knobs come after the caller's arguments so the last occurrence of a key wins. The MEM_OVERRIDE checks pin that MI355X is untouched at 32 GPUs, picks the overrides up at 16, and that 192GB parts keep them at every scale. Verified by mutation: reverting the world-size clause, flipping the knob order, or disabling the profiler unconditionally each fails a check. Co-authored-by: Cursor <cursoragent@cursor.com>

Copilot

Copilot review overview

🟡 Changes recommended

The new environment knobs bypass numerous direct Primus launch paths despite being described as applying to every model.

Review effort: Balanced
Findings: 1

Open (1)

Route all Primus launches through helper to honor environment knobs · New

What changed in this PR

Enables GPT-OSS-120B training on 16 GPUs and exposes deployment-controlled Primus performance settings.

Changes:

Adds 16-GPU memory overrides.
Adds optional iteration and profiler controls.
Adds shell regression tests.

File	Description
`primus_megatron-lm_benchmark_report.sh`	Adds runtime knobs and topology-based overrides.
`tests/test_primus_overrides.sh`	Tests argument assembly and memory overrides.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

PRIMUS_TRAIN_ITERS and PRIMUS_DISABLE_PROFILE only take effect through run_primus(). Most models in this script still invoke `primus-cli direct` themselves, and there the two variables are ignored without a word: the run is full-length where a short one was requested, and profiled where the profiler was meant to be off. Either way it looks like an ordinary successful run, and the perf number it produces is not the one the caller thinks they asked for. Routing all ~40 remaining launch sites through run_primus() is the real fix, but it is a large change to a script many models share and does not belong in a PR that adds one model. Scope the knobs explicitly instead and warn on exit when they were set but never applied. Reported by Copilot on ROCm#271. Co-authored-by: Cursor <cursoragent@cursor.com>

Copilot

Copilot review overview

🟢 Approval recommended

The changes are scoped, syntactically valid, and covered by focused regression tests.

Review effort: Balanced
Findings: None

Resolved since last review (1)

Route all Primus launches through helper to honor environment knobs

i-kosarev

LGTM, just maybe replace "run_primus() is the single launch point for every Primus model here" in description with "run_primus() is the launch point for the six models this CI measures, GPT-OSS-120B among them; the other branches call runner/primus-cli direct themselves and the knobs do not reach them. But nothing outside the script could reach either setting: …"

## What this changes Adds `gpt-oss-120b-training` at 2 nodes (16 GPUs) to the MADEngine gate as a second training workload. It runs through the same same-node A/B as llama-3.1-70b. Stacked on #12899 and based on its branch, so the diff shows only the commits below. Merge after #12899. - **Workload matrix.** The scheduled caller hardcoded `llama-3.1-70b-training`, so it becomes a matrix with `fail-fast: false`. `WORK_DIR` now includes the workload name, because matrix jobs share `run_id` and `run_attempt`. The legs run one at a time (`max-parallel: 1`): each wants an exclusive 2-node `salloc` on `meta64`, and queue time counts against its `--timeout-minutes`. The `workflow_dispatch` input becomes a `choice`. - **GPT-OSS-120B entry.** The `mad-rccl` pin from #12899 (`7a9a25e`) contains ROCm/MAD#271 (`07ecef61`), which makes GPT-OSS-120B fit at 16 GPUs and adds `PRIMUS_TRAIN_ITERS` and `PRIMUS_DISABLE_PROFILE`. The workload sets them through a new per-workload `env_vars` key, merged into the container env: 15 iterations with the profiler off. It expects BF16 and FP8, so a leg that stops running fails the gate. A workload's `env_vars` now also wins for `NCCL_SOCKET_IFNAME` and `GLOO_SOCKET_IFNAME`; neither workload sets them. - **Weekly, not nightly.** The nightly gate keeps only `llama-3.1-70b-training`. GPT-OSS-120B moves to a new `therock-test-madengine-weekly` job that runs only when `test_scope` is `all`, which the Saturday cron sets and a dispatch can request. It is a matrix of `include` entries with `max-parallel: 1`, so later weekly workloads queue behind each other rather than compete for exclusive nodes. It `needs` the nightly job for ordering only, and `!cancelled()` keeps a failed nightly from skipping it. Nodes are pinned per entry, because each workload was validated at one scale. - **Log size.** RCCL prints `ncclAllToAllv is deprecated` at WARN on every call. GPT-OSS's MoE all-to-all produced 777600 such lines per node per phase, 97% of a 180 MB node log; run 38238634668 alone left 2.1 GB under the runner account's disk quota. After the allocation, before anything reads or uploads the logs, `squash_log_spam()` keeps the first occurrence of each such line and replaces the repeats with a count: 1.1 GB to 24 MB on that run's baseline phase, with identical RCCL banners and parsed results. Real warnings are untouched. Tested in `tests/test_log_squash.py`. - **llama-4-scout dropped.** The entry predates this work and has never run on ruby, in CI or by hand. It pins `rocm/primus:v26.4`, while the MAD template keeps Scout on v26.3. It reached the dispatch choices only because that list mirrors `WORKLOAD_CONFIGS`. The gate now has exactly the two workloads that have run end to end. ## Why the profiler has to be off The shipped config profiles iterations 6-7. At 2 nodes those cost 3.2x (BF16) and 3.7x (FP8) a steady iteration, and iterations 8-9 stay elevated while the trace is written. Primus averages from iteration 3, so the profiler window would sit inside the measured window. With 15 iterations, 13 are measured, at about 65 s (BF16) and 116 s (FP8) each. ## Risk to the existing workloads - The two knobs are unset for llama-3.1-70b. I rendered its manifest to confirm: no `PRIMUS_*` variables appear, so its command is byte-identical. ## Testing and cost [Run 38238634668](https://github.com/ROCm/rocm-systems/actions/runs/38238634668) on ruby, 2x8 MI355X (job 70409): PASSED, BF16 ratio 0.9993, FP8 0.9924, with results read from `perf_primus-megatron-GPT-OSS-120B.csv` rather than the live-log fallback. Phases took 53 and 61 min, so about 2 h per A/B. `time_limit` is 4 h, which fits inside the existing `--timeout-minutes` 330. On the weekly schedule it adds about 2 h plus runner wait to the Saturday run. Peak HBM at 2 nodes was 191.6 GB, about 60 GB under the smaller nodes in `meta64`, so no exclude list is needed. That margin is arithmetic: the workload has not yet run on a small node. ## Not in this PR - A recorded baseline. The A/B gate is a ratio and needs no absolute reference. - Changing the llama iteration count. JIRA ID AICOMNET-427

(#13180) ## What this changes Adds `gpt-oss-120b-training` at 2 nodes (16 GPUs) to the MADEngine gate as a second training workload. It runs through the same same-node A/B as llama-3.1-70b. Stacked on #12899 and based on its branch, so the diff shows only the commits below. Merge after #12899. - **Workload matrix.** The scheduled caller hardcoded `llama-3.1-70b-training`, so it becomes a matrix with `fail-fast: false`. `WORK_DIR` now includes the workload name, because matrix jobs share `run_id` and `run_attempt`. The legs run one at a time (`max-parallel: 1`): each wants an exclusive 2-node `salloc` on `meta64`, and queue time counts against its `--timeout-minutes`. The `workflow_dispatch` input becomes a `choice`. - **GPT-OSS-120B entry.** The `mad-rccl` pin from #12899 (`7a9a25e`) contains ROCm/MAD#271 (`07ecef61`), which makes GPT-OSS-120B fit at 16 GPUs and adds `PRIMUS_TRAIN_ITERS` and `PRIMUS_DISABLE_PROFILE`. The workload sets them through a new per-workload `env_vars` key, merged into the container env: 15 iterations with the profiler off. It expects BF16 and FP8, so a leg that stops running fails the gate. A workload's `env_vars` now also wins for `NCCL_SOCKET_IFNAME` and `GLOO_SOCKET_IFNAME`; neither workload sets them. - **Weekly, not nightly.** The nightly gate keeps only `llama-3.1-70b-training`. GPT-OSS-120B moves to a new `therock-test-madengine-weekly` job that runs only when `test_scope` is `all`, which the Saturday cron sets and a dispatch can request. It is a matrix of `include` entries with `max-parallel: 1`, so later weekly workloads queue behind each other rather than compete for exclusive nodes. It `needs` the nightly job for ordering only, and `!cancelled()` keeps a failed nightly from skipping it. Nodes are pinned per entry, because each workload was validated at one scale. - **Log size.** RCCL prints `ncclAllToAllv is deprecated` at WARN on every call. GPT-OSS's MoE all-to-all produced 777600 such lines per node per phase, 97% of a 180 MB node log; run 38238634668 alone left 2.1 GB under the runner account's disk quota. After the allocation, before anything reads or uploads the logs, `squash_log_spam()` keeps the first occurrence of each such line and replaces the repeats with a count: 1.1 GB to 24 MB on that run's baseline phase, with identical RCCL banners and parsed results. Real warnings are untouched. Tested in `tests/test_log_squash.py`. - **llama-4-scout dropped.** The entry predates this work and has never run on ruby, in CI or by hand. It pins `rocm/primus:v26.4`, while the MAD template keeps Scout on v26.3. It reached the dispatch choices only because that list mirrors `WORKLOAD_CONFIGS`. The gate now has exactly the two workloads that have run end to end. ## Why the profiler has to be off The shipped config profiles iterations 6-7. At 2 nodes those cost 3.2x (BF16) and 3.7x (FP8) a steady iteration, and iterations 8-9 stay elevated while the trace is written. Primus averages from iteration 3, so the profiler window would sit inside the measured window. With 15 iterations, 13 are measured, at about 65 s (BF16) and 116 s (FP8) each. ## Risk to the existing workloads - The two knobs are unset for llama-3.1-70b. I rendered its manifest to confirm: no `PRIMUS_*` variables appear, so its command is byte-identical. ## Testing and cost [Run 38238634668](https://github.com/ROCm/rocm-systems/actions/runs/38238634668) on ruby, 2x8 MI355X (job 70409): PASSED, BF16 ratio 0.9993, FP8 0.9924, with results read from `perf_primus-megatron-GPT-OSS-120B.csv` rather than the live-log fallback. Phases took 53 and 61 min, so about 2 h per A/B. `time_limit` is 4 h, which fits inside the existing `--timeout-minutes` 330. On the weekly schedule it adds about 2 h plus runner wait to the Saturday run. Peak HBM at 2 nodes was 191.6 GB, about 60 GB under the smaller nodes in `meta64`, so no exclude list is needed. That margin is arithmetic: the workload has not yet run on a small node. ## Not in this PR - A recorded baseline. The A/B gate is a ratio and needs no absolute reference. - Changing the llama iteration count. JIRA ID AICOMNET-427 [rocm-systems] ROCm/rocm-systems#13180 (commit f5c8b3a)

mkuznet1 and others added 3 commits October 5, 2026 15:08

Copilot AI balanced review requested due to automatic review settings October 5, 2026 15:14

Copilot started reviewing on behalf of mkuznet1 October 5, 2026 15:15

View session

Copilot AI reviewed Oct 5, 2026

View reviewed changes

Comment thread scripts/primus_megatron-lm/primus_megatron-lm_benchmark_report.sh Resolved

mkuznet1 marked this pull request as ready for review October 5, 2026 15:18

mkuznet1 requested review from Cemberk, amathews-amd, coketaste, gargrahul, itej89, lcskrishna and leconcio as code owners October 5, 2026 15:18

Copilot AI balanced review requested due to automatic review settings October 6, 2026 11:37

Copilot started reviewing on behalf of mkuznet1 October 6, 2026 11:38

View session

Copilot AI reviewed Oct 6, 2026

View reviewed changes

i-kosarev approved these changes Oct 6, 2026

View reviewed changes

i-kosarev removed request for Cemberk, amathews-amd, coketaste, gargrahul, itej89, lcskrishna and leconcio October 6, 2026 13:35

mkuznet1 merged commit 07ecef6 into ROCm:mad-rccl Oct 8, 2026

mkuznet1 mentioned this pull request Oct 9, 2026

feat(rccl): add GPT-OSS-120B on 2 nodes to the madengine gate ROCm/rocm-systems#13180

Merged

Navigation Menu

Repository navigation

Run GPT-OSS-120B on 2 nodes, and let the deployment set the perf knobs - #271

Run GPT-OSS-120B on 2 nodes, and let the deployment set the perf knobs#271
mkuznet1 merged 4 commits into
ROCm:mad-rcclfrom
mkuznet1:mkuznet1/primus-gpt-oss-120b-2node

mkuznet1 commented Oct 5, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

i-kosarev left a comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

mkuznet1 commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this is for

1. GPT-OSS-120B could not run at 16 GPUs

2. train_iters and the profiler were unreachable from outside

3. Tests

Not in this PR

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟢 Approval recommended

Uh oh!

i-kosarev left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mkuznet1 commented Oct 5, 2026 •

edited

Loading

2. `train_iters` and the profiler were unreachable from outside