[AMD] Make ROCm 7.2.4 images profile HIP-graph kernels, and check that they do - #35390
michaelzhang-ai wants to merge 3 commits into
Conversation
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
github-actions
Bot
added
documentation
Hardware results on MI355XRan the test plan on Two things to report: the libdrm prerequisite and the profiling premise both hold up on real hardware, but the probe as written cannot fail a bad image. The libdrm change is confirmed
The built image has The bug is real and ROCm 7.2.4 fixes itReal SGLang decode trace rather than a microbenchmark: gpt-oss-20b-bf16, triton attention backend,
87% of decode kernels never reach the trace on 7.2.0, and the The probe passes every known-bad image
Two independent reasons: The captured graph is too small. The loss only appears at about 16 graph nodes;
An eager control traces 100% at every size on both runtimes, so this is specific to graph replay. The pass condition asks the wrong question. The verdict turns on Worth noting the single-replay case, which is the reported symptom in its sharpest form. At 64 nodes on 7.2.0:
7.2.4 is 100% at every replay count. FixCapturing 64 kernels by default and comparing traced against launched separates the two runtimes with no variance at all:
With that change the matrix comes out as intended — FAIL on the 7.2.0 base and the shipped nightly, PASS on the 7.2.4 base and the candidate — and repeats 5 of 5 on each side with identical counts. The eager control gets the same completeness check so a runtime that drops events generally stays distinguishable from one that only loses graph events. I'll push that to this branch. Caveat: this host is MI355X only, so gfx942 is still untested. |
bf5bd61 to
36a401b
Compare
michaelzhang-ai
changed the title
|
Warning Correction (2026-08-22): Do not use Option A ( The Measured this on an MI355X against a libdrm: confirmed. The image has HIP-graph profiling: the 7.2.4 packages are in the image ( Use the 64-node × 8-replay check from the test plan (4 nodes still passes on 7.2.0). Same split as reported there:
How to force 7.2.4 into
|
…oaded Two corrections from the measurement on #35390. The loss under ROCm 7.2.0 is partial: a 64-node graph reported 448 of 512 kernels, while a 4-node graph came back complete. The probe captured four nodes and passed on any non-zero kernel count, so it would have called the broken runtime healthy. Capture 64 nodes by default and require every dispatch the phases asked for, reporting the shortfall when there is one. Having the fix installed is also not the same as running it. The torch wheels vendor HIP and roctracer under torch/lib and libtorch_hip.so carries RPATH $ORIGIN, so a ROCm 7.2.4 image still profiles through 7.2.0 libraries and LD_LIBRARY_PATH cannot override it. Report the paths torch mapped rather than sonames, flag the ones that did not come from the ROCm install, and print the LD_PRELOAD that switches them -- otherwise the failure looks like an unfixable image instead of a load-order problem. Co-authored-by: quitenode <quitenode@users.noreply.github.com>
…oaded Two corrections from the measurement on #35390. The loss under ROCm 7.2.0 is partial: a 64-node graph reported 448 of 512 kernels, while a 4-node graph came back complete. The probe captured four nodes and passed on any non-zero kernel count, so it would have called the broken runtime healthy. Capture 64 nodes by default and require every dispatch the phases asked for, reporting the shortfall when there is one. Having the fix installed is also not the same as running it. The torch wheels vendor HIP and roctracer under torch/lib and libtorch_hip.so carries RPATH $ORIGIN, so a ROCm 7.2.4 image still profiles through 7.2.0 libraries and LD_LIBRARY_PATH cannot override it. Report the paths torch mapped rather than sonames, flag the ones that did not come from the ROCm install, and print the LD_PRELOAD that switches them -- otherwise the failure looks like an unfixable image instead of a load-order problem. Co-authored-by: quitenode <quitenode@users.noreply.github.com>
6780c46 to
83775b1
Compare
|
Thanks @chuyeh — the 448/512 split is the important part, and it found a hole in the probe. Two fixes pushed:
Docs no longer claim the image alone is enough: #35398 now runs the probe twice per published image — as shipped (informational, expected to fail while torch shadows the ROCm install) and with the ROCm runtime preloaded (the gate) — so both states show up in the nightly summary. On Option B: baking over the two |
…oaded Two corrections from the measurement on #35390. The loss under ROCm 7.2.0 is partial: a 64-node graph reported 448 of 512 kernels, while a 4-node graph came back complete. The probe captured four nodes and passed on any non-zero kernel count, so it would have called the broken runtime healthy. Capture 64 nodes by default and require every dispatch the phases asked for, reporting the shortfall when there is one. Having the fix installed is also not the same as running it. The torch wheels vendor HIP and roctracer under torch/lib and libtorch_hip.so carries RPATH $ORIGIN, so a ROCm 7.2.4 image still profiles through 7.2.0 libraries and LD_LIBRARY_PATH cannot override it. Report the paths torch mapped rather than sonames, flag the ones that did not come from the ROCm install, and print the LD_PRELOAD that switches them -- otherwise the failure looks like an unfixable image instead of a load-order problem. Co-authored-by: quitenode <quitenode@users.noreply.github.com>
1d1cddd to
6cf3eea
Compare
michaelzhang-ai
changed the title
Hardware runsAll on The Dockerfile step, against the published
run 32517262300 · re-run after the fix below, 32517873558. The published images as they stand, both flavors on the GPU each targets: What the first run caught. On the fixed image the probe still printed "torch is using its own HIP/roctracer" and told the reader to redo a CI status
|
…oaded Two corrections from the measurement on #35390. The loss under ROCm 7.2.0 is partial: a 64-node graph reported 448 of 512 kernels, while a 4-node graph came back complete. The probe captured four nodes and passed on any non-zero kernel count, so it would have called the broken runtime healthy. Capture 64 nodes by default and require every dispatch the phases asked for, reporting the shortfall when there is one. Having the fix installed is also not the same as running it. The torch wheels vendor HIP and roctracer under torch/lib and libtorch_hip.so carries RPATH $ORIGIN, so a ROCm 7.2.4 image still profiles through 7.2.0 libraries and LD_LIBRARY_PATH cannot override it. Report the paths torch mapped rather than sonames, flag the ones that did not come from the ROCm install, and print the LD_PRELOAD that switches them -- otherwise the failure looks like an unfixable image instead of a load-order problem. Co-authored-by: quitenode <quitenode@users.noreply.github.com>
f505030 to
053f061
Compare
053f061 to
0cdd5f9
Compare
|
Thanks @devalshahamd — both findings landed.
Your run also caught a wart in the probe: on the fixed image it still said "torch is using its own HIP/roctracer" and told you to redo the copy, because the check keyed on the path and the fix puts the ROCm build at the wheel's path. It now decides by file size instead. |
…efault The ROCm 7.2.4 images fix the roctracer failure that loses kernel-dispatch events under hipGraphLaunch (ROCm/legacy-rocm-build#6102, fixed in 7.2.2), and then do not use it: the torch wheels vendor their own HIP and roctracer under torch/lib and libtorch_hip.so carries RPATH $ORIGIN, so the loader takes those 7.2.0 copies and LD_LIBRARY_PATH cannot override an RPATH. Measured on the published v0.5.17-rocm724-*-20260820 images, 448 of 512 graph-replay kernels reach the trace on MI300X and on MI355X, and a real server trace loses 87% of its decode kernels while looking complete. Replacing the two files the wheel vendors makes the working configuration the default. LD_PRELOAD of the same libraries is not an equivalent workaround: it leaves both copies mapped and only interposes symbols, so a component resolving them independently ends up on the other runtime -- reported as an abort during CUDA-graph capture inside Triton's AMD launcher. Keyed on whether the wheel vendors them rather than on the flavor, so a torch that already links /opt/rocm is left alone and no version guard needs updating. Replaced by rename because the interpreter doing the replacing has the old files mapped. A fresh interpreter then imports torch, so a soname or ABI mismatch fails the build instead of a user's first profile. Co-authored-by: quitenode <quitenode@users.noreply.github.com>
…nels Nothing in CI could tell a working profiler from a broken one: test/registered/profiling/test_start_profile.py only asserts the trace directory is non-empty, and since #34452 the profiling suite runs without CUDA graphs on ROCm, so it never exercises the replay path. Whether an image traces graph replay also cannot be read off its ROCm version, because the torch wheel can shadow the ROCm runtime. The probe captures a graph, replays it under torch.profiler, and counts the device events reaching the exported trace against the dispatches it asked for. Counting is the point: the loss is partial -- 448 of 512 -- and a 4-node graph is traced completely even on 7.2.0, so both a non-empty-trace threshold and a small graph would call a broken runtime healthy. 64 nodes by default is calibrated between two thresholds: 7.2.0 is complete at 4 and lossy from about 16, and 7.2.4 has been seen dropping at 256. An eager control separates "cannot trace graph replay" from "cannot trace anything", and each phase runs in its own process under a timeout because 7.2.0 can wedge inside hipGraphLaunch instead of losing events. The report names the libraries torch actually mapped, decided by file size rather than path so that both a preload and an in-place replacement are recognised, and prints the commands that repair an image. The CPU tests drive main() with canned phase results, since a GPU job only ever exercises whichever failure its own image has: every verdict and exit code, the partial-loss shortfall, runtime detection in each of its shapes, the child command carrying every workload option, and the trace parser, where counting a host-only trace as device work would turn the whole check into a false pass. Co-authored-by: quitenode <quitenode@users.noreply.github.com>
Traces taken on a ROCm 7.2.0 runtime under-report the decode steps, which run from replayed graphs, and the loss is partial rather than total, so nothing about the trace says it is wrong. Record the symptom, how to check an image, the rocm724 images as the answer, how to repair an image that predates the runtime fix, and --disable-cuda-graph where neither applies. Co-authored-by: quitenode <quitenode@users.noreply.github.com>
233c879 to
b4761ba
Compare
4 participants
Motivation
AMD/Silo reported that on the
lmsysorg/sglang*ROCm 7.2 images,torch.profilertraces contain no GPU kernels for work launched through hipGraph. The cause is upstream and fixed upstream — ROCTracer drops kernel-dispatch events underhipGraphLaunch(ROCm/ROCm#6102, fixed in ROCm 7.2.2) — and #30984 has since publishedrocm724images that install ROCm 7.2.4.Installing it turned out not to be the same as using it. The torch wheels vendor their own HIP and roctracer under
torch/lib, andlibtorch_hip.socarriesRPATH $ORIGIN, so the published 7.2.4 images profile through 7.2.0 libraries andLD_LIBRARY_PATHcannot override anRPATH. Measured onrocm/sgl-dev:v0.5.17-rocm724-*-20260820:On a real server trace the gap is larger: gpt-oss-20b-bf16 with
/start_profile num_steps=10records 513 device kernel events with CUDA graphs on, against 3924 with--disable-cuda-graphand 3933 on a working runtime — 87% of decode kernels missing from a trace that otherwise looks complete.So this PR does three things: makes the images use the ROCm runtime they install, adds the check that tells the two states apart, and documents it.
Modifications
docker/rocm.Dockerfile— use the ROCm runtime the image installsOne step at the end of the torch work replaces
torch/lib/libamdhip64.soandtorch/lib/libroctracer64.sowith the ROCm install's versioned copies, plus a verification step. Three details worth review:/opt/rocm— the rocm720 stack on torch 2.9.1 — there is nothing to replace and the step says so and moves on, so there is no version guard to update for the next flavor. A wheel that vendors them with no ROCm copy available is a broken image, and the build fails.os.replaces.LD_PRELOADof the same two libraries is not an equivalent workaround: @devalshahamd found it aborts server startup on 7.2.4 inside Triton's AMD launcher during CUDA-graph capture, while this replacement leaves both the probe and a real workload with clean profiles. That fits the mechanism — a preload leaves both HIP copies mapped and only interposes symbols, so a component resolving them independently can end up on the other runtime with its own state.scripts/ci/amd/check_hip_graph_profiling.py— the checkCaptures a graph, replays it under
torch.profiler, and counts device events in the exported trace against the dispatches it asked for. Counting matters: the loss is partial (448 of 512), and a 4-node graph is traced correctly even on 7.2.0, so both a "non-empty trace" threshold and a small graph would call a broken runtime healthy. The default is 64 nodes replayed 8 times — a calibrated point, since 7.2.0 is complete at 4 and lossy from about 16 while 7.2.4 has been seen dropping at 256.An eager control run separates "cannot trace graph replay" from "cannot trace anything", and each phase runs in its own process under a timeout because 7.2.0 can also wedge inside
hipGraphLaunch. It reports the paths of the libraries torch mapped, flags any the ROCm install is not backing, and prints the commands to fix that;--print-ld-preloadexposes the value for a single-run check.Not registered in the AMD suites: on a rocm720 image it is expected to fail and, per #34452, can take the process down with it.
test/registered/unit/ci/test_hip_graph_profiling_probe.pyThe probe's value is that a red run says which failure happened, and a GPU job only ever exercises whichever case its image is in. 18 CPU tests drive
main()with canned phase results: every verdict and exit code, the partial-loss shortfall, vendored-runtime detection including the half-preloaded case, the preload value naming real files rather than symlinks, and the trace parser, where counting a host-only trace as device work would turn the check into a false pass. Registered asbase-a-test-cpu.docs/docs/developer_guide/benchmark_and_profiling.mdxRecords the symptom, how to check an image, the
rocm724images as the answer, how to repair an older image, and--disable-cuda-graphwhere neither applies.Accuracy Tests
No model or kernel code changes. The Dockerfile step swaps two shared libraries for the same libraries at the version the image already installs.
Speed Tests and Profiling
The point of the change is that
torch.profilerreports what the GPU ran. No throughput effect is expected: HIP and roctracer are replaced with the ROCm 7.2.4 build the image ships and that the rest of the stack was validated against.Measured
The fix, on the published image. The Dockerfile step's own code, extracted from
docker/rocm.Dockerfileat runtime, applied torocm/sgl-dev:v0.5.17-rocm724-mi35x-20260820on MI355X, probing either side of it with nothing preloaded — run 32517873558:torch/libcopiestorchstill imports afterwards (2.11.0+rocm7.2,hip 7.2.26015), and the step copied 27,193,296 and 347,896 bytes — the ROCm file sizes exactly.Both published flavors, on the GPU each targets,
448/512FAIL as shipped and512/512PASS with the ROCm runtime in use: MI355X 32337394284, MI300X 32422850894.torch.cuda.get_device_name(0)resolves on both, and torch reportship 7.2.26015— the 7.2.0 build — inside a 7.2.4 image.Real server trace, gpt-oss-20b-bf16 with
/start_profile num_steps=10, identical workload across runs: 513 device kernel events with CUDA graphs on against 3924 with--disable-cuda-graphand 3933 on a working runtime, i.e. 87% of decode kernels missing.Independently, @devalshahamd ran the probe on ROCm 7.0.0, 7.2.0 and 7.2.4 and reported it behaves as expected on all three, and that replacing the wheel's libraries gives a real SGLang workload a clean profile.
The Dockerfile step's logic exercised against a simulated
torch/liband/opt/rocm/lib: both vendored, idempotent re-run, nothing vendored, vendored with no ROCm copy (fails loudly), symlinks never chosen as the source.CI: gate green and
base-a-test-cpugreen with the probe's tests collected and run (32421463756 for the earlier 18; 22 now).PR Test Extra/PR Test Extra (AMD)fail on the missingrun-ci-extralabel, not on tests.Review findings addressed
LD_PRELOADaborts server startup on 7.2.4 (@devalshahamd) — inside Triton's AMD launcher during CUDA-graph capture. That is why the Dockerfile replaces the files instead of documenting a preload, and why the probe marks the preload as a single-run check. [AMD] Make ROCm 7.2.4 images profile correctly without LD_PRELOAD #35806 proposed the Dockerfile change separately and is folded in here.run_childdropped--graph-nodes(@devalshahamd) — the parent accepted the flag and the children used their own default, so a--graph-nodes 4calibration run measured 64. The child command is now built from a declared option set, with a test that fails if a future option is neither forwarded nor parent-only. Default runs were unaffected, so the numbers above stand.kernels > 0on a 4-node graph and would have called every broken image healthy. It now counts every dispatch at 64 nodes.Checklist
Notes
Review and Merge Process
CI States
Latest PR Test (Base): ❌ Run #32682707842
Latest PR Test (Extra): ❌ Run #32682707619
Latest PR Test (AMD ROCm 7.2): ❌ Run #32682707786