Repository navigation
feat(rccl): add GPT-OSS-120B on 2 nodes to the madengine gate - #13180
Merged
Merged
Conversation
github-actions
Bot
added
github actions
mkuznet1
marked this pull request as ready for review
October 9, 2026 11:46
1 task done
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
![amd-argus[bot]](https://avatars.githubusercontent.com/in/4740020?s=60&v=4){kind=small}
coketaste
reviewed
Oct 9, 2026
coketaste
left a comment
coketaste
left a comment
There was a problem hiding this comment.
The workload entry matches MAD at 07ecef61: multiple_results is perf_primus-megatron-GPT-OSS-120B.csv, and GPT-OSS-120B goes through run_primus(), so PRIMUS_TRAIN_ITERS=15 and PRIMUS_DISABLE_PROFILE=1 reach Primus. Llama-3.1-70B does not set those variables, so its command stays unchanged. Dropping Llama-4-Scout is justified because that entry has never run on ruby and pins rocm/primus:v26.4 while MAD keeps Scout on v26.3.
The other reason in the description does not hold. An FP8 leg disabled in MAD is absent from both phases. pair_by_precision only builds a row for a precision that came back, so that case stays green on BF16. It is not scored as a missing comparison.
One fix before this lands, inline: the per-run archive path.
mkuznet1
force-pushed
the
ci/madengine-gpt-oss-120b
branch
from
October 12, 2026 10:48
009e1cb to
165b446
Compare
Contributor
There was a problem hiding this comment.
ARGUS
Bottom line: Worth settling before approval: the per-run archive path at test_madengine.py:1505 still has no workload component, so the two matrix legs overwrite each other's copied perf files.
Comment @argus review to re-run, or @argus re-review after pushing.
i-kosarev
added a commit
that referenced
this pull request
Oct 13, 2026
…e gate Adds `llama-3.1-405b-maxtext` at 4 nodes (32 GPUs) as the first JAX workload in the gate, and the first whose base image is not rocm/primus. Stacked on Draft until #12899 and #13180 land. The enabling work is ROCm/MAD#228, which validated this workload at 4N. Model family ------------ Until now every workload was PyTorch/Megatron-LM out of MAD's scripts/primus_megatron-lm. A "family" key now dispatches the three places that differ: install_madengine() copies scripts/jax-maxtext and checks Primus out, generate_manifest() emits `--config_path <yaml>` plus a `distributed.primus` block instead of `--model_repo`, and the overlay builder uses a different recipe. Settings lifted from MAD#228's template are each a failure mode, not taste: XLA_GPU_AUTOTUNE_LEVEL=0 Primus defaults to level 4, which at 32 ranks presents as a dead hang -- no steps, no NCCL output -- rather than a slow start. The MAD-native XLA_AUTOTUNE_LEVEL is a silent no-op. per_device_batch_size=1 the shipped config OOMs at 241.25 GiB for jit_train_step at 4 nodes. --ulimit nofile=1048576 a 32-device clique exhausts the default and stalls silently at "Initialize clique". They ride the per-workload env_vars key #13180 introduced. Why this family needs its own A/B baseline ------------------------------------------ The Megatron baseline is the stock registry image. That cannot work here: rocm/jax-training:maxtext-* ships MaxText but no Primus, so scripts/jax-maxtext/run.sh would not find primus-cli and the baseline phase would die before step 0. So the baseline is built: base image plus the Primus payload, stock RCCL. The candidate is that same Dockerfile plus the librccl swap -- literally a prefix, so the payload layers come out of the cache and only the swap builds twice. The pair therefore differs by the RCCL swap and nothing else, which is tighter than the Megatron A/B rather than looser. The baseline manifest advertises its own Dockerfile. Left at the default a compute node would rebuild the swapped image under the baseline tag, giving an A/B that compares the CI RCCL against itself. Overlay ------- The swap for this family is discovery-based -- find, per-target RUNPATH preservation, and an unresolved-soname set gate -- ported from MAD#228, rather than aimed at the Primus SDK venv paths, which this base lacks. The libamd_smi soname bridge is kept and now runs over every directory the sweep finds, ordered after the copy and before the post-swap measurement so a .27-against-.26 skew is reconciled instead of failing the gate. Unlike MAD#228's dockerfile the Primus payload is gated on primus-cli, not examples/run_pretrain.sh: AMD-AGI/Primus#999 retired that file and run.sh drives primus-cli. v26.7.0 still carries both so MAD's gate passes today, but it would hard-fail on jax-maxtext-v26.8. PRIMUS_REF is pinned here. --skip-overlay-build is refused for this family: its bind-mount targets are Primus-image paths absent from this base, and `docker run -v` creates a missing source as an empty directory rather than refusing, so the run would look healthy while measuring the image's own RCCL. Scheduling ---------- Its own job rather than a third matrix entry: the weekly gate has to be a job-level `if:`, where the matrix context is unavailable, and it is fixed at 4 nodes while the matrix entries take madengine_nodes. WORK_DIR already carries inputs.workload, so they do not collide. Not gated on the matrix job -- a 70B failure should not hide a 405B result. time_limit is 05:00:00 for the A/B pair, against MAD#228's 12:00:00 for a single run: a 4-node exclusive request with a 12 h limit backfills badly, the measured run is ~17 min of training (MAD#228 reports 17.40 s/step) plus prep, and 300 min stays inside the step's existing 330 so the script still reports. The existing 360/330 budget needed no change. Risk to the existing workloads ------------------------------ MAD_REF moves 07ecef61 -> 7acfac7c, which is exactly one commit, ROCm/MAD#279, repointing tools/fetch_primus.sh at a ref that still exists. It touches benchmark/jax_maxtext/README.md, docker/primus_maxdiffusion.ubuntu.amd.Dockerfile and tools/fetch_primus.sh -- nothing under scripts/primus_megatron-lm. I rendered both A/B phase manifests for llama-3.1-70b, llama-4-scout and gpt-oss-120b against this branch and its base: byte-identical, all six. Also verified: the 405B manifest matches MAD#228's validated template field for field; the baseline and candidate manifests differ only in image and dockerfile; both generated Dockerfiles parse under `bash -n`, with the bridge ordered after the copy and before the gate. Not yet run on hardware. The first run has to establish whether the TheRock kpack librccl loads on this base at all, and whether its libamd_smi soname needs the bridge here. JIRA ID AICOMNET-420
mkuznet1
force-pushed
the
ci/madengine-gpt-oss-120b
branch
from
October 14, 2026 10:20
165b446 to
ebe6e1f
Compare
Contributor
Author
i-kosarev
added a commit
that referenced
this pull request
Oct 14, 2026
…e gate Adds `llama-3.1-405b-maxtext` at 4 nodes (32 GPUs) as the first JAX workload in the gate, and the first whose base image is not rocm/primus. Stacked on Draft until #12899 and #13180 land. The enabling work is ROCm/MAD#228, which validated this workload at 4N. Model family ------------ Until now every workload was PyTorch/Megatron-LM out of MAD's scripts/primus_megatron-lm. A "family" key now dispatches the three places that differ: install_madengine() copies scripts/jax-maxtext and checks Primus out, generate_manifest() emits `--config_path <yaml>` plus a `distributed.primus` block instead of `--model_repo`, and the overlay builder uses a different recipe. Settings lifted from MAD#228's template are each a failure mode, not taste: XLA_GPU_AUTOTUNE_LEVEL=0 Primus defaults to level 4, which at 32 ranks presents as a dead hang -- no steps, no NCCL output -- rather than a slow start. The MAD-native XLA_AUTOTUNE_LEVEL is a silent no-op. per_device_batch_size=1 the shipped config OOMs at 241.25 GiB for jit_train_step at 4 nodes. --ulimit nofile=1048576 a 32-device clique exhausts the default and stalls silently at "Initialize clique". They ride the per-workload env_vars key #13180 introduced. Why this family needs its own A/B baseline ------------------------------------------ The Megatron baseline is the stock registry image. That cannot work here: rocm/jax-training:maxtext-* ships MaxText but no Primus, so scripts/jax-maxtext/run.sh would not find primus-cli and the baseline phase would die before step 0. So the baseline is built: base image plus the Primus payload, stock RCCL. The candidate is that same Dockerfile plus the librccl swap -- literally a prefix, so the payload layers come out of the cache and only the swap builds twice. The pair therefore differs by the RCCL swap and nothing else, which is tighter than the Megatron A/B rather than looser. The baseline manifest advertises its own Dockerfile. Left at the default a compute node would rebuild the swapped image under the baseline tag, giving an A/B that compares the CI RCCL against itself. Overlay ------- The swap for this family is discovery-based -- find, per-target RUNPATH preservation, and an unresolved-soname set gate -- ported from MAD#228, rather than aimed at the Primus SDK venv paths, which this base lacks. The libamd_smi soname bridge is kept and now runs over every directory the sweep finds, ordered after the copy and before the post-swap measurement so a .27-against-.26 skew is reconciled instead of failing the gate. Unlike MAD#228's dockerfile the Primus payload is gated on primus-cli, not examples/run_pretrain.sh: AMD-AGI/Primus#999 retired that file and run.sh drives primus-cli. v26.7.0 still carries both so MAD's gate passes today, but it would hard-fail on jax-maxtext-v26.8. PRIMUS_REF is pinned here. --skip-overlay-build is refused for this family: its bind-mount targets are Primus-image paths absent from this base, and `docker run -v` creates a missing source as an empty directory rather than refusing, so the run would look healthy while measuring the image's own RCCL. Scheduling ---------- Its own job rather than a third matrix entry: the weekly gate has to be a job-level `if:`, where the matrix context is unavailable, and it is fixed at 4 nodes while the matrix entries take madengine_nodes. WORK_DIR already carries inputs.workload, so they do not collide. Not gated on the matrix job -- a 70B failure should not hide a 405B result. time_limit is 05:00:00 for the A/B pair, against MAD#228's 12:00:00 for a single run: a 4-node exclusive request with a 12 h limit backfills badly, the measured run is ~17 min of training (MAD#228 reports 17.40 s/step) plus prep, and 300 min stays inside the step's existing 330 so the script still reports. The existing 360/330 budget needed no change. Risk to the existing workloads ------------------------------ MAD_REF moves 07ecef61 -> 7acfac7c, which is exactly one commit, ROCm/MAD#279, repointing tools/fetch_primus.sh at a ref that still exists. It touches benchmark/jax_maxtext/README.md, docker/primus_maxdiffusion.ubuntu.amd.Dockerfile and tools/fetch_primus.sh -- nothing under scripts/primus_megatron-lm. I rendered both A/B phase manifests for llama-3.1-70b, llama-4-scout and gpt-oss-120b against this branch and its base: byte-identical, all six. Also verified: the 405B manifest matches MAD#228's validated template field for field; the baseline and candidate manifests differ only in image and dockerfile; both generated Dockerfiles parse under `bash -n`, with the bridge ordered after the copy and before the gate. Not yet run on hardware. The first run has to establish whether the TheRock kpack librccl loads on this base at all, and whether its libamd_smi soname needs the bridge here. JIRA ID AICOMNET-420
i-kosarev
added a commit
that referenced
this pull request
Oct 14, 2026
…e gate Adds `llama-3.1-405b-maxtext` at 4 nodes (32 GPUs) as the first JAX workload in the gate, and the first whose base image is not rocm/primus. Stacked on Draft until #12899 and #13180 land. The enabling work is ROCm/MAD#228, which validated this workload at 4N. Model family ------------ Until now every workload was PyTorch/Megatron-LM out of MAD's scripts/primus_megatron-lm. A "family" key now dispatches the three places that differ: install_madengine() copies scripts/jax-maxtext and checks Primus out, generate_manifest() emits `--config_path <yaml>` plus a `distributed.primus` block instead of `--model_repo`, and the overlay builder uses a different recipe. Settings lifted from MAD#228's template are each a failure mode, not taste: XLA_GPU_AUTOTUNE_LEVEL=0 Primus defaults to level 4, which at 32 ranks presents as a dead hang -- no steps, no NCCL output -- rather than a slow start. The MAD-native XLA_AUTOTUNE_LEVEL is a silent no-op. per_device_batch_size=1 the shipped config OOMs at 241.25 GiB for jit_train_step at 4 nodes. --ulimit nofile=1048576 a 32-device clique exhausts the default and stalls silently at "Initialize clique". They ride the per-workload env_vars key #13180 introduced. Why this family needs its own A/B baseline ------------------------------------------ The Megatron baseline is the stock registry image. That cannot work here: rocm/jax-training:maxtext-* ships MaxText but no Primus, so scripts/jax-maxtext/run.sh would not find primus-cli and the baseline phase would die before step 0. So the baseline is built: base image plus the Primus payload, stock RCCL. The candidate is that same Dockerfile plus the librccl swap -- literally a prefix, so the payload layers come out of the cache and only the swap builds twice. The pair therefore differs by the RCCL swap and nothing else, which is tighter than the Megatron A/B rather than looser. The baseline manifest advertises its own Dockerfile. Left at the default a compute node would rebuild the swapped image under the baseline tag, giving an A/B that compares the CI RCCL against itself. Overlay ------- The swap for this family is discovery-based -- find, per-target RUNPATH preservation, and an unresolved-soname set gate -- ported from MAD#228, rather than aimed at the Primus SDK venv paths, which this base lacks. The libamd_smi soname bridge is kept and now runs over every directory the sweep finds, ordered after the copy and before the post-swap measurement so a .27-against-.26 skew is reconciled instead of failing the gate. Unlike MAD#228's dockerfile the Primus payload is gated on primus-cli, not examples/run_pretrain.sh: AMD-AGI/Primus#999 retired that file and run.sh drives primus-cli. v26.7.0 still carries both so MAD's gate passes today, but it would hard-fail on jax-maxtext-v26.8. PRIMUS_REF is pinned here. --skip-overlay-build is refused for this family: its bind-mount targets are Primus-image paths absent from this base, and `docker run -v` creates a missing source as an empty directory rather than refusing, so the run would look healthy while measuring the image's own RCCL. Scheduling ---------- Its own job rather than a third matrix entry: the weekly gate has to be a job-level `if:`, where the matrix context is unavailable, and it is fixed at 4 nodes while the matrix entries take madengine_nodes. WORK_DIR already carries inputs.workload, so they do not collide. Not gated on the matrix job -- a 70B failure should not hide a 405B result. time_limit is 05:00:00 for the A/B pair, against MAD#228's 12:00:00 for a single run: a 4-node exclusive request with a 12 h limit backfills badly, the measured run is ~17 min of training (MAD#228 reports 17.40 s/step) plus prep, and 300 min stays inside the step's existing 330 so the script still reports. The existing 360/330 budget needed no change. Risk to the existing workloads ------------------------------ MAD_REF moves 07ecef61 -> 7acfac7c, which is exactly one commit, ROCm/MAD#279, repointing tools/fetch_primus.sh at a ref that still exists. It touches benchmark/jax_maxtext/README.md, docker/primus_maxdiffusion.ubuntu.amd.Dockerfile and tools/fetch_primus.sh -- nothing under scripts/primus_megatron-lm. I rendered both A/B phase manifests for llama-3.1-70b, llama-4-scout and gpt-oss-120b against this branch and its base: byte-identical, all six. Also verified: the 405B manifest matches MAD#228's validated template field for field; the baseline and candidate manifests differ only in image and dockerfile; both generated Dockerfiles parse under `bash -n`, with the bridge ordered after the copy and before the gate. Not yet run on hardware. The first run has to establish whether the TheRock kpack librccl loads on this base at all, and whether its libamd_smi soname needs the bridge here. JIRA ID AICOMNET-420
i-kosarev
added a commit
that referenced
this pull request
Oct 14, 2026
The stack conflicts when it lands in order, and this comment is why. #12899 reworded it to a one-liner after #13180 branched, so #13180 -- and this branch on top of it -- still carry the old two-line form. Landing #12899 first and then this branch makes both sides changed in the same region. Removing the `timeout-minutes` line from the step is substantive and stays. The explanation of why it is gone does not need to live in that hunk, so it moves up to the job-level timeout it is actually about, and the pre-existing comment is left exactly as #13180 has it. Only one side then touches those lines and the merge resolves on its own. The real staleness is that #13180 sits on an older ci/madengine-same-node-ab; rebasing it is that PR's call, not this one's. This just stops this branch from adding to the collision.
i-kosarev
added a commit
that referenced
this pull request
Oct 15, 2026
…e gate Adds `llama-3.1-405b-maxtext` at 4 nodes (32 GPUs) as the first JAX workload in the gate, and the first whose base image is not rocm/primus. Stacked on Draft until #12899 and #13180 land. The enabling work is ROCm/MAD#228, which validated this workload at 4N. Model family ------------ Until now every workload was PyTorch/Megatron-LM out of MAD's scripts/primus_megatron-lm. A "family" key now dispatches the three places that differ: install_madengine() copies scripts/jax-maxtext and checks Primus out, generate_manifest() emits `--config_path <yaml>` plus a `distributed.primus` block instead of `--model_repo`, and the overlay builder uses a different recipe. Settings lifted from MAD#228's template are each a failure mode, not taste: XLA_GPU_AUTOTUNE_LEVEL=0 Primus defaults to level 4, which at 32 ranks presents as a dead hang -- no steps, no NCCL output -- rather than a slow start. The MAD-native XLA_AUTOTUNE_LEVEL is a silent no-op. per_device_batch_size=1 the shipped config OOMs at 241.25 GiB for jit_train_step at 4 nodes. --ulimit nofile=1048576 a 32-device clique exhausts the default and stalls silently at "Initialize clique". They ride the per-workload env_vars key #13180 introduced. Why this family needs its own A/B baseline ------------------------------------------ The Megatron baseline is the stock registry image. That cannot work here: rocm/jax-training:maxtext-* ships MaxText but no Primus, so scripts/jax-maxtext/run.sh would not find primus-cli and the baseline phase would die before step 0. So the baseline is built: base image plus the Primus payload, stock RCCL. The candidate is that same Dockerfile plus the librccl swap -- literally a prefix, so the payload layers come out of the cache and only the swap builds twice. The pair therefore differs by the RCCL swap and nothing else, which is tighter than the Megatron A/B rather than looser. The baseline manifest advertises its own Dockerfile. Left at the default a compute node would rebuild the swapped image under the baseline tag, giving an A/B that compares the CI RCCL against itself. Overlay ------- The swap for this family is discovery-based -- find, per-target RUNPATH preservation, and an unresolved-soname set gate -- ported from MAD#228, rather than aimed at the Primus SDK venv paths, which this base lacks. The libamd_smi soname bridge is kept and now runs over every directory the sweep finds, ordered after the copy and before the post-swap measurement so a .27-against-.26 skew is reconciled instead of failing the gate. Unlike MAD#228's dockerfile the Primus payload is gated on primus-cli, not examples/run_pretrain.sh: AMD-AGI/Primus#999 retired that file and run.sh drives primus-cli. v26.7.0 still carries both so MAD's gate passes today, but it would hard-fail on jax-maxtext-v26.8. PRIMUS_REF is pinned here. --skip-overlay-build is refused for this family: its bind-mount targets are Primus-image paths absent from this base, and `docker run -v` creates a missing source as an empty directory rather than refusing, so the run would look healthy while measuring the image's own RCCL. Scheduling ---------- Its own job rather than a third matrix entry: the weekly gate has to be a job-level `if:`, where the matrix context is unavailable, and it is fixed at 4 nodes while the matrix entries take madengine_nodes. WORK_DIR already carries inputs.workload, so they do not collide. Not gated on the matrix job -- a 70B failure should not hide a 405B result. time_limit is 05:00:00 for the A/B pair, against MAD#228's 12:00:00 for a single run: a 4-node exclusive request with a 12 h limit backfills badly, the measured run is ~17 min of training (MAD#228 reports 17.40 s/step) plus prep, and 300 min stays inside the step's existing 330 so the script still reports. The existing 360/330 budget needed no change. Risk to the existing workloads ------------------------------ MAD_REF moves 07ecef61 -> 7acfac7c, which is exactly one commit, ROCm/MAD#279, repointing tools/fetch_primus.sh at a ref that still exists. It touches benchmark/jax_maxtext/README.md, docker/primus_maxdiffusion.ubuntu.amd.Dockerfile and tools/fetch_primus.sh -- nothing under scripts/primus_megatron-lm. I rendered both A/B phase manifests for llama-3.1-70b, llama-4-scout and gpt-oss-120b against this branch and its base: byte-identical, all six. Also verified: the 405B manifest matches MAD#228's validated template field for field; the baseline and candidate manifests differ only in image and dockerfile; both generated Dockerfiles parse under `bash -n`, with the bridge ordered after the copy and before the gate. Not yet run on hardware. The first run has to establish whether the TheRock kpack librccl loads on this base at all, and whether its libamd_smi soname needs the bridge here. JIRA ID AICOMNET-420
i-kosarev
added a commit
that referenced
this pull request
Oct 15, 2026
The stack conflicts when it lands in order, and this comment is why. #12899 reworded it to a one-liner after #13180 branched, so #13180 -- and this branch on top of it -- still carry the old two-line form. Landing #12899 first and then this branch makes both sides changed in the same region. Removing the `timeout-minutes` line from the step is substantive and stays. The explanation of why it is gone does not need to live in that hunk, so it moves up to the job-level timeout it is actually about, and the pre-existing comment is left exactly as #13180 has it. Only one side then touches those lines and the merge resolves on its own. The real staleness is that #13180 sits on an older ci/madengine-same-node-ab; rebasing it is that PR's call, not this one's. This just stops this branch from adding to the collision.
i-kosarev
added a commit
that referenced
this pull request
Oct 19, 2026
…e gate Adds `llama-3.1-405b-maxtext` at 4 nodes (32 GPUs) as the first JAX workload in the gate, and the first whose base image is not rocm/primus. Stacked on Draft until #12899 and #13180 land. The enabling work is ROCm/MAD#228, which validated this workload at 4N. Model family ------------ Until now every workload was PyTorch/Megatron-LM out of MAD's scripts/primus_megatron-lm. A "family" key now dispatches the three places that differ: install_madengine() copies scripts/jax-maxtext and checks Primus out, generate_manifest() emits `--config_path <yaml>` plus a `distributed.primus` block instead of `--model_repo`, and the overlay builder uses a different recipe. Settings lifted from MAD#228's template are each a failure mode, not taste: XLA_GPU_AUTOTUNE_LEVEL=0 Primus defaults to level 4, which at 32 ranks presents as a dead hang -- no steps, no NCCL output -- rather than a slow start. The MAD-native XLA_AUTOTUNE_LEVEL is a silent no-op. per_device_batch_size=1 the shipped config OOMs at 241.25 GiB for jit_train_step at 4 nodes. --ulimit nofile=1048576 a 32-device clique exhausts the default and stalls silently at "Initialize clique". They ride the per-workload env_vars key #13180 introduced. Why this family needs its own A/B baseline ------------------------------------------ The Megatron baseline is the stock registry image. That cannot work here: rocm/jax-training:maxtext-* ships MaxText but no Primus, so scripts/jax-maxtext/run.sh would not find primus-cli and the baseline phase would die before step 0. So the baseline is built: base image plus the Primus payload, stock RCCL. The candidate is that same Dockerfile plus the librccl swap -- literally a prefix, so the payload layers come out of the cache and only the swap builds twice. The pair therefore differs by the RCCL swap and nothing else, which is tighter than the Megatron A/B rather than looser. The baseline manifest advertises its own Dockerfile. Left at the default a compute node would rebuild the swapped image under the baseline tag, giving an A/B that compares the CI RCCL against itself. Overlay ------- The swap for this family is discovery-based -- find, per-target RUNPATH preservation, and an unresolved-soname set gate -- ported from MAD#228, rather than aimed at the Primus SDK venv paths, which this base lacks. The libamd_smi soname bridge is kept and now runs over every directory the sweep finds, ordered after the copy and before the post-swap measurement so a .27-against-.26 skew is reconciled instead of failing the gate. Unlike MAD#228's dockerfile the Primus payload is gated on primus-cli, not examples/run_pretrain.sh: AMD-AGI/Primus#999 retired that file and run.sh drives primus-cli. v26.7.0 still carries both so MAD's gate passes today, but it would hard-fail on jax-maxtext-v26.8. PRIMUS_REF is pinned here. --skip-overlay-build is refused for this family: its bind-mount targets are Primus-image paths absent from this base, and `docker run -v` creates a missing source as an empty directory rather than refusing, so the run would look healthy while measuring the image's own RCCL. Scheduling ---------- Its own job rather than a third matrix entry: the weekly gate has to be a job-level `if:`, where the matrix context is unavailable, and it is fixed at 4 nodes while the matrix entries take madengine_nodes. WORK_DIR already carries inputs.workload, so they do not collide. Not gated on the matrix job -- a 70B failure should not hide a 405B result. time_limit is 05:00:00 for the A/B pair, against MAD#228's 12:00:00 for a single run: a 4-node exclusive request with a 12 h limit backfills badly, the measured run is ~17 min of training (MAD#228 reports 17.40 s/step) plus prep, and 300 min stays inside the step's existing 330 so the script still reports. The existing 360/330 budget needed no change. Risk to the existing workloads ------------------------------ MAD_REF moves 07ecef61 -> 7acfac7c, which is exactly one commit, ROCm/MAD#279, repointing tools/fetch_primus.sh at a ref that still exists. It touches benchmark/jax_maxtext/README.md, docker/primus_maxdiffusion.ubuntu.amd.Dockerfile and tools/fetch_primus.sh -- nothing under scripts/primus_megatron-lm. I rendered both A/B phase manifests for llama-3.1-70b, llama-4-scout and gpt-oss-120b against this branch and its base: byte-identical, all six. Also verified: the 405B manifest matches MAD#228's validated template field for field; the baseline and candidate manifests differ only in image and dockerfile; both generated Dockerfiles parse under `bash -n`, with the bridge ordered after the copy and before the gate. Not yet run on hardware. The first run has to establish whether the TheRock kpack librccl loads on this base at all, and whether its libamd_smi soname needs the bridge here. JIRA ID AICOMNET-420
i-kosarev
added a commit
that referenced
this pull request
Oct 19, 2026
The stack conflicts when it lands in order, and this comment is why. #12899 reworded it to a one-liner after #13180 branched, so #13180 -- and this branch on top of it -- still carry the old two-line form. Landing #12899 first and then this branch makes both sides changed in the same region. Removing the `timeout-minutes` line from the step is substantive and stays. The explanation of why it is gone does not need to live in that hunk, so it moves up to the job-level timeout it is actually about, and the pre-existing comment is left exactly as #13180 has it. Only one side then touches those lines and the merge resolves on its own. The real staleness is that #13180 sits on an older ci/madengine-same-node-ab; rebasing it is that PR's call, not this one's. This just stops this branch from adding to the collision.
Contributor
pr-rocm-systems-13180-oci-rccl-pr-buildBuilds
FailuresNo failing tests in this run. 189 tests skipped. pr-rocm-systems-13180-oci-rccl-pr-smokeBuilds
Failures
15 tests skipped. |
Contributor
pr-rocm-systems-13180-ruby64-rccl-pr-smokeBuilds
FailuresNo failing tests in this run. 15 tests skipped. |
Contributor
pr-rocm-systems-13180-oci-rccl-pr-buildBuilds
FailuresNo failing tests in this run. 189 tests skipped. pr-rocm-systems-13180-oci-rccl-pr-smokeBuilds
Failures
15 tests skipped. |
Contributor
pr-rocm-systems-13180-ruby64-rccl-pr-smokeBuilds
FailuresNo failing tests in this run. 15 tests skipped. |
The scheduled caller hardcoded llama-3.1-70b-training, so a second workload needed a second job rather than a list entry. Turn it into a matrix with fail-fast disabled: one workload failing says nothing about the others, and the scale reservation is already paid for by the time it does. Two consequences handled here. WORK_DIR gains the workload name because the caller now runs several workloads from one workflow run, which shares both run_id and run_attempt, and the cleanup step leaves perf.csv behind -- so without it the second workload could parse the first one's results. And the workflow_dispatch input becomes a choice, since a typo there otherwise only surfaces once the job has allocated nodes; workflow_call inputs cannot be typed as choice, so that side keeps a comment pointing at WORKLOAD_CONFIGS. Co-authored-by: Cursor <cursoragent@cursor.com>
MAD#271 (in mad-rccl since 07ecef61, so under the 7a9a25e pin) makes GPT-OSS-120B fit at the 16-GPU minimum and lets the deployment set train_iters and switch the profiler off. The two knobs are unset for the existing workloads, which leaves their command unchanged. The workload sets PRIMUS_TRAIN_ITERS=15 and PRIMUS_DISABLE_PROFILE=1 through a new per-workload env_vars key merged into the container env: the shipped config profiles iterations 6-7, which cost 3x a steady iteration at 2 nodes and would land inside the measured window. It joins the scheduled matrix and the workflow_dispatch choices. Co-authored-by: Cursor <cursoragent@cursor.com>
…ne gate The entry predates the A/B and has never run on ruby, in CI or by hand. It also pins rocm/primus:v26.4 while the MAD template keeps Scout on v26.3, and its FP8 leg is disabled in MAD, which the A/B would score as a missing comparison and fail. It reached the workflow_dispatch choices only because that list mirrors WORKLOAD_CONFIGS. Keep the gate to the two workloads that have been run end to end: llama-3.1-70b and gpt-oss-120b. Co-authored-by: Cursor <cursoragent@cursor.com>
MAD's run.sh trains both on gfx950, and run 38238634668 returned both. Listing them makes a leg that stops running fail the gate instead of leaving it green on the other. Co-authored-by: Cursor <cursoragent@cursor.com>
env_vars was spread last into the container env, but NCCL_SOCKET_IFNAME was read from the cluster env before that, so slurm network_interface kept the cluster value, and GLOO_SOCKET_IFNAME was written after it and always won. Read the socket interface with the workload's value first, and derive GLOO_SOCKET_IFNAME only when the workload does not set it. Co-authored-by: Cursor <cursoragent@cursor.com>
Each leg asks for an exclusive 2-node salloc on meta64. With more than one scale runner both could queue at once, and the one waiting behind the other's allocation spends its --timeout-minutes in the queue and records a timeout for both phases. Serialise the matrix, as the other shared-cluster RCCL workflows do with a concurrency group. Co-authored-by: Cursor <cursoragent@cursor.com>
The nightly madengine gate goes back to the one workload it had before the same-node A/B work: Llama-3.1-70B. GPT-OSS-120B moves to a new weekly job that only runs when test_scope is 'all', which the Saturday cron sets and a dispatch can request. The weekly job is a matrix of include entries, so max-parallel: 1 serialises every weekly workload against the others. Each one holds an exclusive salloc on the same partition, and queue time counts against its own budget. Nodes are set per entry, because each workload was validated at one scale. The job needs the nightly one for ordering only, and !cancelled() stops a failed nightly from skipping it. Co-authored-by: Cursor <cursoragent@cursor.com>
… logs RCCL prints "NCCL WARN Please note that ncclAllToAllv is deprecated" on every call (src/misc/api_trace.cc), and at WARN, so the CI's NCCL_DEBUG keeps it. GPT-OSS-120B's MoE all-to-all produced 777600 of these lines per node per phase: 97% of a 180 MB node log, which madengine stores three times. Run 38238634668 alone left 2.1 GB on /apps, under the quota of the runner account that later failed conda create with "Disk quota exceeded". After the allocation, and before anything reads or uploads the logs, squash_log_spam() keeps the first occurrence of each such line and replaces the repeats with a count. On that run's baseline phase this took 1.1 GB down to 24 MB in 5 s, and the RCCL banners and the parsed results stayed identical. Real warnings are untouched; lowering NCCL_DEBUG would have hidden them too. Co-authored-by: Cursor <cursoragent@cursor.com>
…quota it hit The 2.1 GB that one GPT-OSS run left lived on the work directory's filesystem, which is not the quota that later ran out; that was the runner account's home. Co-authored-by: Cursor <cursoragent@cursor.com>
- therock-test-madengine-weekly now needs test-plan and runs only when the madengine suite is selected, like the nightly job: an accl-profiler-only dispatch with test_scope=all still launched this two-node job. - squash_log_spam rewrites only .out/.err/.log files under slurm_output, not every file found there. Co-authored-by: Cursor <cursoragent@cursor.com>
i-kosarev
force-pushed
the
ci/madengine-gpt-oss-120b
branch
from
November 2, 2026 13:26
731e939 to
d87146d
Compare
Contributor
pr-rocm-systems-13180-oci-rccl-pr-smokeBuilds
FailuresNo failing tests in this run. 15 tests skipped. |
Contributor
pr-rocm-systems-13180-ruby64-rccl-pr-smokeBuilds
FailuresNo failing tests in this run. 15 tests skipped. |
speriaswamy-amd
approved these changes
Nov 3, 2026
speriaswamy-amd
left a comment
speriaswamy-amd
left a comment
Contributor
There was a problem hiding this comment.
LGTM. No blocking correctness or regression issues found. Validated the pinned MAD recipe and rendered manifest settings; 101 local tests passed. One nonblocking test-coverage nit below.
ddurnov
approved these changes
Nov 3, 2026
systems-assistant Bot
pushed a commit
to ROCm/rccl
that referenced
this pull request
Nov 3, 2026
(#13180) ## What this changes Adds `gpt-oss-120b-training` at 2 nodes (16 GPUs) to the MADEngine gate as a second training workload. It runs through the same same-node A/B as llama-3.1-70b. Stacked on #12899 and based on its branch, so the diff shows only the commits below. Merge after #12899. - **Workload matrix.** The scheduled caller hardcoded `llama-3.1-70b-training`, so it becomes a matrix with `fail-fast: false`. `WORK_DIR` now includes the workload name, because matrix jobs share `run_id` and `run_attempt`. The legs run one at a time (`max-parallel: 1`): each wants an exclusive 2-node `salloc` on `meta64`, and queue time counts against its `--timeout-minutes`. The `workflow_dispatch` input becomes a `choice`. - **GPT-OSS-120B entry.** The `mad-rccl` pin from #12899 (`7a9a25e`) contains ROCm/MAD#271 (`07ecef61`), which makes GPT-OSS-120B fit at 16 GPUs and adds `PRIMUS_TRAIN_ITERS` and `PRIMUS_DISABLE_PROFILE`. The workload sets them through a new per-workload `env_vars` key, merged into the container env: 15 iterations with the profiler off. It expects BF16 and FP8, so a leg that stops running fails the gate. A workload's `env_vars` now also wins for `NCCL_SOCKET_IFNAME` and `GLOO_SOCKET_IFNAME`; neither workload sets them. - **Weekly, not nightly.** The nightly gate keeps only `llama-3.1-70b-training`. GPT-OSS-120B moves to a new `therock-test-madengine-weekly` job that runs only when `test_scope` is `all`, which the Saturday cron sets and a dispatch can request. It is a matrix of `include` entries with `max-parallel: 1`, so later weekly workloads queue behind each other rather than compete for exclusive nodes. It `needs` the nightly job for ordering only, and `!cancelled()` keeps a failed nightly from skipping it. Nodes are pinned per entry, because each workload was validated at one scale. - **Log size.** RCCL prints `ncclAllToAllv is deprecated` at WARN on every call. GPT-OSS's MoE all-to-all produced 777600 such lines per node per phase, 97% of a 180 MB node log; run 38238634668 alone left 2.1 GB under the runner account's disk quota. After the allocation, before anything reads or uploads the logs, `squash_log_spam()` keeps the first occurrence of each such line and replaces the repeats with a count: 1.1 GB to 24 MB on that run's baseline phase, with identical RCCL banners and parsed results. Real warnings are untouched. Tested in `tests/test_log_squash.py`. - **llama-4-scout dropped.** The entry predates this work and has never run on ruby, in CI or by hand. It pins `rocm/primus:v26.4`, while the MAD template keeps Scout on v26.3. It reached the dispatch choices only because that list mirrors `WORKLOAD_CONFIGS`. The gate now has exactly the two workloads that have run end to end. ## Why the profiler has to be off The shipped config profiles iterations 6-7. At 2 nodes those cost 3.2x (BF16) and 3.7x (FP8) a steady iteration, and iterations 8-9 stay elevated while the trace is written. Primus averages from iteration 3, so the profiler window would sit inside the measured window. With 15 iterations, 13 are measured, at about 65 s (BF16) and 116 s (FP8) each. ## Risk to the existing workloads - The two knobs are unset for llama-3.1-70b. I rendered its manifest to confirm: no `PRIMUS_*` variables appear, so its command is byte-identical. ## Testing and cost [Run 38238634668](https://github.com/ROCm/rocm-systems/actions/runs/38238634668) on ruby, 2x8 MI355X (job 70409): PASSED, BF16 ratio 0.9993, FP8 0.9924, with results read from `perf_primus-megatron-GPT-OSS-120B.csv` rather than the live-log fallback. Phases took 53 and 61 min, so about 2 h per A/B. `time_limit` is 4 h, which fits inside the existing `--timeout-minutes` 330. On the weekly schedule it adds about 2 h plus runner wait to the Saturday run. Peak HBM at 2 nodes was 191.6 GB, about 60 GB under the smaller nodes in `meta64`, so no exclude list is needed. That margin is arithmetic: the workload has not yet run on a small node. ## Not in this PR - A recorded baseline. The A/B gate is a ratio and needs no absolute reference. - Changing the llama iteration count. JIRA ID AICOMNET-427 [rocm-systems] ROCm/rocm-systems#13180 (commit f5c8b3a)
Contributor
TheRock Submodule Bump ActivityNewest first
|
i-kosarev
added a commit
that referenced
this pull request
Nov 5, 2026
…e gate (#13182) Motivation The MADEngine gate runs PyTorch/Megatron-LM only — every workload comes out of MAD's scripts/primus_megatron-lm on a rocm/primus base. JAX/MaxText drives RCCL differently and is a shipping training path, but nothing in CI runs it. This adds llama-3.1-405b-maxtext at 4 nodes (32 GPUs) as the first JAX workload, through the same same-node A/B as the rest. The enabling work is ROCm/MAD#228, which validated it at 4N. Stacked on #13180.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Adds
gpt-oss-120b-trainingat 2 nodes (16 GPUs) to the MADEngine gate as a second training workload. It runs through the same same-node A/B as llama-3.1-70b.Stacked on #12899 and based on its branch, so the diff shows only the commits below. Merge after #12899.
llama-3.1-70b-training, so it becomes a matrix withfail-fast: false.WORK_DIRnow includes the workload name, because matrix jobs sharerun_idandrun_attempt. The legs run one at a time (max-parallel: 1): each wants an exclusive 2-nodesalloconmeta64, and queue time counts against its--timeout-minutes. Theworkflow_dispatchinput becomes achoice.mad-rcclpin from feat(rccl): score the madengine gate against a same-node baseline #12899 (7a9a25e) contains Run GPT-OSS-120B on 2 nodes, and let the deployment set the perf knobs MAD#271 (07ecef61), which makes GPT-OSS-120B fit at 16 GPUs and addsPRIMUS_TRAIN_ITERSandPRIMUS_DISABLE_PROFILE. The workload sets them through a new per-workloadenv_varskey, merged into the container env: 15 iterations with the profiler off. It expects BF16 and FP8, so a leg that stops running fails the gate. A workload'senv_varsnow also wins forNCCL_SOCKET_IFNAMEandGLOO_SOCKET_IFNAME; neither workload sets them.llama-3.1-70b-training. GPT-OSS-120B moves to a newtherock-test-madengine-weeklyjob that runs only whentest_scopeisall, which the Saturday cron sets and a dispatch can request. It is a matrix ofincludeentries withmax-parallel: 1, so later weekly workloads queue behind each other rather than compete for exclusive nodes. Itneedsthe nightly job for ordering only, and!cancelled()keeps a failed nightly from skipping it. Nodes are pinned per entry, because each workload was validated at one scale.ncclAllToAllv is deprecatedat WARN on every call. GPT-OSS's MoE all-to-all produced 777600 such lines per node per phase, 97% of a 180 MB node log; run 38238634668 alone left 2.1 GB under the runner account's disk quota. After the allocation, before anything reads or uploads the logs,squash_log_spam()keeps the first occurrence of each such line and replaces the repeats with a count: 1.1 GB to 24 MB on that run's baseline phase, with identical RCCL banners and parsed results. Real warnings are untouched. Tested intests/test_log_squash.py.rocm/primus:v26.4, while the MAD template keeps Scout on v26.3. It reached the dispatch choices only because that list mirrorsWORKLOAD_CONFIGS. The gate now has exactly the two workloads that have run end to end.Why the profiler has to be off
The shipped config profiles iterations 6-7. At 2 nodes those cost 3.2x (BF16) and 3.7x (FP8) a steady iteration, and iterations 8-9 stay elevated while the trace is written. Primus averages from iteration 3, so the profiler window would sit inside the measured window. With 15 iterations, 13 are measured, at about 65 s (BF16) and 116 s (FP8) each.
Risk to the existing workloads
PRIMUS_*variables appear, so its command is byte-identical.Testing and cost
Run 38238634668 on ruby, 2x8 MI355X (job 70409): PASSED, BF16 ratio 0.9993, FP8 0.9924, with results read from
perf_primus-megatron-GPT-OSS-120B.csvrather than the live-log fallback. Phases took 53 and 61 min, so about 2 h per A/B.time_limitis 4 h, which fits inside the existing--timeout-minutes330. On the weekly schedule it adds about 2 h plus runner wait to the Saturday run.Peak HBM at 2 nodes was 191.6 GB, about 60 GB under the smaller nodes in
meta64, so no exclude list is needed. That margin is arithmetic: the workload has not yet run on a small node.Not in this PR
JIRA ID AICOMNET-427