[AMD][CI] Publish ROCm 10 release images and kernel wheel - #38763
Merged
Merged
Conversation
ROCm 10 is now the primary flavor for AMD PR and nightly tests (#38659) and the nightly workflow already publishes rocm10-mi30x / rocm10-mi35x, but a version tag only produced rocm700, rocm720 and rocm724 images. Add rocm10 to the release matrix and the manual flavor selector, mapping it to the gfx942-rocm1000 / gfx950-rocm1000 Dockerfile stages. gfx1250 (mi45x) stays out of the release matrix: it is bring-up only and the nightly keeps it in a separate job for that reason. Co-authored-by: quitenode <quitenode@users.noreply.github.com>
The kernel release only covered ROCm 7.0 and 7.2, so the flavor AMD CI now gates on has no published wheel. Add a 1000 leg to the ROCm build matrix and a release-rocm1000 job that publishes it under the rocm1000 wheel index. ROCm 10 is distributed as pip packages that unpack into site-packages, so the build container has no /opt/rocm tree for CMakeLists_rocm.txt to find hip-lang under and, being a runtime image, no devel tree to compile against. build_rocm.sh applies the same fixups rocm.Dockerfile's rocm1000-base stage makes. That also leaves no /opt/rocm-<version> directory for rename_wheels_rocm.sh to read the wheel's local version tag from, so it now accepts one from the caller and keeps the probe as the fallback for the 7.x flavors. Co-authored-by: quitenode <quitenode@users.noreply.github.com>
michaelzhang-ai
requested review from
Fridge003,
HaiShaw,
Kangyan-Zhou,
bingxche,
ispobock and
merrymercy
as code owners
September 10, 2026 00:33
Every ROCm leg is an image build on the shared AMD node, and its release job appends the wheel to the index again, so a dispatch naming one flavor should not drag the other two along. Build only the named flavor and publish only what was built; a push still runs the full set. Co-authored-by: quitenode <quitenode@users.noreply.github.com>
michaelzhang-ai
temporarily deployed
to
prod
September 10, 2026 00:41 — with
GitHub Actions
Inactive
michaelzhang-ai
temporarily deployed
to
prod
September 10, 2026 00:41 — with
GitHub Actions
Inactive
HaiShaw
approved these changes
Sep 10, 2026
HaiShaw
left a comment
Collaborator
There was a problem hiding this comment.
LGTM.
Please purge rocm700 next.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Follow-up to #38659, which made ROCm 10 the default flavor for AMD PR and nightly tests. The release side was left behind: the two release workflows called out there still produce 7.x artifacts only.
release-docker-amd.ymlbuildsrocm700,rocm720androcm724on a version tag, solmsysorg/sglanghas nov<version>-rocm10-mi30x/-mi35xtag even thoughrelease-docker-amd-rocm10.ymlpublishes the same images nightly torocm/sgl-devandlmsysorg/sglang-rocm.release-whl-kernel.ymlbuildssglang-kernelwheels for ROCm 7.0 and 7.2 only, so the flavor AMD CI now gates on has no published wheel and norocm1000wheel index.Modifications
release-docker-amd.yml— addrocm10to the release matrix and to the manual flavor selector, mapping it to thegfx942-rocm1000/gfx950-rocm1000Dockerfile stages and therocm10-mi30x/rocm10-mi35xtags the nightly images already use. gfx1250 (mi45x) is deliberately left out: it is bring-up only, which is why the nightly keeps it in its own job.release-whl-kernel.yml— add a1000leg to the ROCm build matrix, arocm1000option to the dispatch target selector, and arelease-rocm1000job that publishes the wheel and updates therocm1000index (scripts/update_kernel_whl_index.py --rocm 1000, no change needed there). A dispatch that names one ROCm flavor now builds and publishes only that flavor: each leg is an image build on the shared AMD node, and each release job appends its wheel to the index again, so pulling all three along was both expensive and a source of duplicate index lines. A push still runs the full set.3rdparty/amd/wheel/sgl-kernel/build_rocm.sh— teach it ROCm 10, building againstrocm/pytorch:rocm10.0_ubuntu24.04_py3.12_pytorch_release_2.11.0. That image matches the Ubuntu 24.04 / Python 3.12 / torch 2.11 stack the released ROCm 10 images carry and is built withAMDGPU_FAMILY=device-all, so a single wheel still covers gfx942 and gfx950. ROCm 10 ships as pip packages that unpack into site-packages, so the container has no/opt/rocmtree forCMakeLists_rocm.txtto find hip-lang under and, being a runtime image, no devel tree to compile against; the new setup block applies the same fixupsdocker/rocm.Dockerfile'srocm1000-basestage makes for the images.3rdparty/amd/wheel/sgl-kernel/rename_wheels_rocm.sh— accept the ROCm version for the wheel's local version tag from the caller, since the pip-installed SDK leaves no/opt/rocm-<version>directory to read it from. The 7.x flavors pass nothing and keep the existing probe.Compatibility
700and720is byte-identical to today's apart from one blank line and a trailing space, andrename_wheels_rocm.shfalls back to the same/opt/rocm-*probe when no version is passed.rocm700/rocm720/rocm724still publish on every version tag.Validation
ROCm 10 kernel wheel — dispatched and green. Run #34422339330 on this branch with
target: rocm1000. Only the1000leg was created, confirming the new flavor scoping, and bothbuild-rocm-matrix (3.10, 1000)andrelease-rocm1000passed. The build log confirms each piece of the ROCm 10 path:rocm-sdk-devel==10.0.0installed, so the runtime image did need the devel tree added./opt/rocmsymlink:Found HIP: /opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel.Using AMDGPU_TARGET from environment: gfx942;gfx950/Multi-arch build: Enabling both HIP_FP8_TYPE_FNUZ (gfx942) and HIP_FP8_TYPE_E4M3 (gfx950), so one wheel covers both archs despite the Python 3.12 container.sglang_kernel-0.4.6.post1+rocm1000-cp310-abi3-manylinux2014_x86_64.whl.The artifact is live: it is attached to the
v0.4.6.post1release insgl-project/whlnext to the+rocm700and+rocm720wheels,rocm1000/sglang-kernel/index.htmlongh-pageshas the matching entry, and the wheel contains the builtsgl_kernel/common_ops.abi3.so.ROCm 10 release image — dispatched and green. Run #34422374614 on this branch with
rocm_version: rocm10,version: 0.5.19. The flavor selector expanded to exactlypublish (rocm10, gfx942, all)andpublish (rocm10, gfx950, all)and both passed, publishinglmsysorg/sglang:v0.5.19-rocm10-mi30xand-mi35x(new tags, colliding with nothing). Inspecting the published image configs confirms the flavor-to-stage mapping: the mi30x image carriesGPU_ARCH=gfx942-rocm1000and the mi35x imageGPU_ARCH=gfx950-rocm1000, both withROCM_HOMEundersite-packages/_rocm_sdk_develas the ROCm 10 stack expects.Local: captured the in-container script
build_rocm.shgenerates for700,720and1000, syntax-checked each, and diffed the700/720output against the pre-change script to confirm ROCm 10 support does not alter them.pre-commit run --filesover the changed files passes.Follow-ups
3rdparty/amd/wheel/sglang/pyproject.tomlcarriesrocm700androcm720extras that pin a publishedsglang-kernelwheel URL. Now that a+rocm1000wheel exists, arocm1000extra can be added in a follow-up.Accuracy Tests
N/A — release workflow change; model outputs are unaffected.
Speed Tests and Profiling
N/A — this changes which release artifacts are built.
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ci