[AMD] Verify the nightly ROCm 7.2.4 images trace HIP-graph kernels

[michaelzhang-ai]

Motivation

#30984 landed the rocm724 flavors and the nightly now publishes them to rocm/sgl-dev and lmsysorg/sglang-rocm. Those flavors exist because ROCm 7.2.0 drops kernel-dispatch events for work submitted through hipGraphLaunch (ROCm/ROCm#6102, fixed in 7.2.2), which is what AMD/Silo hit when profiling SGLang decoding on every *-rocm720-* image.

Nothing verifies that the published images actually restore it. A trace with no GPU kernels in it still looks like a successful profile, test/registered/profiling/test_start_profile.py only asserts the trace directory is non-empty, and since #34452 the profiling suite runs without CUDA graphs on ROCm, so CI never exercises the replay path at all. A future base bump could quietly take the fix away again.

Modifications

One job in release-docker-amd-rocm720-nightly.yml. After publish, each rocm724 flavor is probed on the GPU it targets — gfx942 on MI300X, gfx950 on MI355X — because the behaviour lives in the HIP runtime rather than in SGLang, so it is worth confirming per device generation. The job pulls the tag the publish job recorded in its artifact, starts the standard CI container against it, and runs scripts/ci/amd/check_hip_graph_profiling.py inside the image. No dependency install: the point is to exercise the image's own torch and HIP runtime, which is what was just published.

Gating details:

• Per leg, not per workflow. A rocm720 leg failing in publish must not stop the rocm724 images from being checked, so the job does not require all of publish to succeed. A rocm724 leg that never published has no artifact and fails at the download — the same behaviour the mirror job below documents.
• rocm720 legs are excluded, since they are expected to fail; a dispatch selecting a single rocm720 arch skips the job.
• job_select is respected, so a publish-only dispatch does not run it.
• job_select=verify_profiling + probe_image qualifies an already-published tag without a rebuild; pair it with gpu_arch to choose the GPU. That is the quickest way to answer the question for one image, and it is how this should be run first.
• The container is removed at the end, since these runners are shared with the AMD test suites and it would otherwise hold VRAM until something else reclaims it.

This verifies after publish rather than gating it, so the daily images are not held behind a GPU runner queue. A red leg means the day's rocm724 image traces nothing under graph replay and should not be recommended for profiling. Turning it into a real gate means splitting the publish job's rocm/sgl-dev and lmsysorg pushes, which is worth doing separately if you want that.

Accuracy Tests

No model or kernel code changes; this adds a verification job to a release workflow.

Speed Tests and Profiling

Not applicable. The job measures whether profiling works on the published image.

Test plan

• actionlint reports nothing on the new job; the only findings in the file are the pre-existing self-hosted runner labels and docker/login-action@v2 in the publish and mirror jobs.
• Job names pass scripts/lint/check_workflow_job_names.py.
• The job's gating was simulated across the combinations that matter — schedule, all/all, publish-only, a rocm720 arch, a single rocm724 arch, verify_profiling with a skipped publish, and publish failure vs cancelled — confirming it runs exactly the intended legs in each.
• The probe itself is unit-tested in [AMD] Make ROCm 7.2.4 images profile HIP-graph kernels, and check that they do #35390 (trace parsing, every verdict path) and validated for timeout handling and no-GPU behaviour.

workflow_dispatch only offers a workflow that exists on the default branch, so this cannot be exercised until it merges. Once it has, the fastest check is one already-published image on one GPU:

gh workflow run release-docker-amd-rocm720-nightly.yml --ref main \
  -f job_select=verify_profiling \
  -f probe_image=rocm/sgl-dev:v0.5.17-rocm724-mi35x-20260820 \
  -f gpu_arch=gfx950-rocm724

Expected VERDICT: PASS, against the same probe failing on a *-rocm720-* tag. A full gh workflow run ... --ref main then covers build plus both GPUs.

Notes for reviewers

• Depends on [AMD] Make ROCm 7.2.4 images profile HIP-graph kernels, and check that they do #35390, which adds the probe script this job runs. Merge that first.
• This PR previously proposed its own rocm724 stages, nightly workflow, release-matrix entry, and torch Triton-pin patch. [AMD] [Docker] Upgrade Python 3.12 + torch 2.11 + triton 3.7 in ROCm 7.2.4 #30984 landed all four, so only the verification remains.

Measured on hardware

Dispatched against a branch carrying this job plus #35390's probe, targeting today's published image (run, 4.5 minutes end to end on linux-mi35x-gpu-1):

• publish and push_local_registry skipped, one verify_profiling (gfx950-rocm724) leg, artifact download skipped because probe_image was set — the gating behaved as intended.
• Image default: graph replay: 448/512 → step outcome failure, job still green (continue-on-error), and the summary records it as **Image default** (failure).
• ROCm runtime preloaded: graph replay: 512/512 → PASS. --print-ld-preload inside the image produced /opt/rocm/lib/libamdhip64.so.7.2.70204:/opt/rocm/lib/libroctracer64.so.4.1.70204.

workflow_dispatch runs the definition from the selected ref, so this is exactly what the job will do once merged.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ✅ Run #32331314126
Latest PR Test (Extra): ❌ Run #32422817619
Latest PR Test (AMD ROCm 7.2): ➖ No AMD PR run found for this commit.

[michaelzhang-ai]

Hardware results on MI355X

Built and qualified the gfx950-rocm724 flavor on smci355-ccs-aus-m12-17 (8x MI355X / gfx950, KMD 6.16.13), from this branch merged with #35390. The merge is clean — the two rocm.Dockerfile hunks do not overlap.

docker build . -f docker/rocm.Dockerfile \
  --build-arg GPU_ARCH=gfx950-rocm724 --build-arg BRANCH_TYPE=local \
  --build-arg ENABLE_MORI=0 --build-arg ENABLE_NIXL=0 -t sglang-rocm724-candidate

Succeeded in about 46 minutes. Every claim in the description checks out in the resulting image:

claim	observed
7.2.4 base carries the fixed libraries	libamdhip64.so.7.2.70204, libroctracer64.so.4.1.70204
stack unchanged but for the ROCm patch level	torch 2.9.1+rocm7.2.4.git39497456, hip 7.2.53211
relax_torch_triton_pin.py drops the exact pin	triton; platform_system == "Linux" and platform_machine == "x86_64"
AITER's Triton is what is installed	triton 3.7.0+amd.rocm7.2.0.git89002410
device name resolves (needs #35390's libdrm step)	'AMD Instinct MI355X'

The metadata patch is doing real work here: without it the image would record triton==3.5.1+rocm7.2.4.gita272dfa8 while shipping AITER's 3.7.0.

The flavor delivers what it exists for

Real decode trace rather than the microbenchmark — gpt-oss-20b-bf16, triton attention backend, /start_profile with num_steps=10, identical workload across runs (same 3520 cpu_op, 1993 cuda_runtime, 63 graph replays):

image	device kernel events in trace
rocm/sgl-dev:v0.5.17-rocm720-mi35x-20260818, graphs on	513
same image, --disable-cuda-graph	3924
this branch's rocm724 candidate, graphs on	3933

The rocm720 image loses 87% of decode kernels; the rocm724 candidate recovers them with CUDA graphs left on.

One blocker for the nightly gate

The promote job is gated on check_hip_graph_profiling.py, and as written that probe returns VERDICT: PASS on every ROCm 7.2.0 image I tested — the rocm/pytorch base, the rocm/sgl-dev nightly and the lmsysorg release. Its captured graph is 4 nodes, which is below the size where 7.2.0 starts dropping events, and it passes on any nonzero kernel count while the actual failure is partial loss. As it stands the gate would promote an unfixed image.

Details and the fix are in #35390. With the probe capturing 64 kernels and comparing traced against launched, the separation is exact and repeatable — 448/512 on 7.2.0 and 512/512 on 7.2.4, ten runs out of ten on each — and the candidate here passes 5 of 5. Worth landing that before this nightly starts gating on it.

gfx942 is untested; this host is MI355X only.