Skip to content

[AMD][CI] Publish ROCm 10 release images and kernel wheel - #38763

Merged
HaiShaw merged 3 commits into
mainfrom
cursor/amd-rocm10-release-artifacts-767a
Sep 10, 2026
Merged

HaiShaw merged 3 commits into
mainfrom
cursor/amd-rocm10-release-artifacts-767a

Conversation

@michaelzhang-ai

@michaelzhang-ai michaelzhang-ai commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

Motivation

Follow-up to #38659, which made ROCm 10 the default flavor for AMD PR and nightly tests. The release side was left behind: the two release workflows called out there still produce 7.x artifacts only.

  • release-docker-amd.yml builds rocm700, rocm720 and rocm724 on a version tag, so lmsysorg/sglang has no v<version>-rocm10-mi30x / -mi35x tag even though release-docker-amd-rocm10.yml publishes the same images nightly to rocm/sgl-dev and lmsysorg/sglang-rocm.
  • release-whl-kernel.yml builds sglang-kernel wheels for ROCm 7.0 and 7.2 only, so the flavor AMD CI now gates on has no published wheel and no rocm1000 wheel index.

Modifications

release-docker-amd.yml — add rocm10 to the release matrix and to the manual flavor selector, mapping it to the gfx942-rocm1000 / gfx950-rocm1000 Dockerfile stages and the rocm10-mi30x / rocm10-mi35x tags the nightly images already use. gfx1250 (mi45x) is deliberately left out: it is bring-up only, which is why the nightly keeps it in its own job.

release-whl-kernel.yml — add a 1000 leg to the ROCm build matrix, a rocm1000 option to the dispatch target selector, and a release-rocm1000 job that publishes the wheel and updates the rocm1000 index (scripts/update_kernel_whl_index.py --rocm 1000, no change needed there). A dispatch that names one ROCm flavor now builds and publishes only that flavor: each leg is an image build on the shared AMD node, and each release job appends its wheel to the index again, so pulling all three along was both expensive and a source of duplicate index lines. A push still runs the full set.

3rdparty/amd/wheel/sgl-kernel/build_rocm.sh — teach it ROCm 10, building against rocm/pytorch:rocm10.0_ubuntu24.04_py3.12_pytorch_release_2.11.0. That image matches the Ubuntu 24.04 / Python 3.12 / torch 2.11 stack the released ROCm 10 images carry and is built with AMDGPU_FAMILY=device-all, so a single wheel still covers gfx942 and gfx950. ROCm 10 ships as pip packages that unpack into site-packages, so the container has no /opt/rocm tree for CMakeLists_rocm.txt to find hip-lang under and, being a runtime image, no devel tree to compile against; the new setup block applies the same fixups docker/rocm.Dockerfile's rocm1000-base stage makes for the images.

3rdparty/amd/wheel/sgl-kernel/rename_wheels_rocm.sh — accept the ROCm version for the wheel's local version tag from the caller, since the pip-installed SDK leaves no /opt/rocm-<version> directory to read it from. The 7.x flavors pass nothing and keep the existing probe.

Compatibility

  • Existing flavors are untouched. The generated in-container build script for 700 and 720 is byte-identical to today's apart from one blank line and a trailing space, and rename_wheels_rocm.sh falls back to the same /opt/rocm-* probe when no version is passed.
  • Workflow names, job IDs, and the tag scheme for existing images are unchanged. rocm700 / rocm720 / rocm724 still publish on every version tag.

Validation

ROCm 10 kernel wheel — dispatched and green. Run #34422339330 on this branch with target: rocm1000. Only the 1000 leg was created, confirming the new flavor scoping, and both build-rocm-matrix (3.10, 1000) and release-rocm1000 passed. The build log confirms each piece of the ROCm 10 path:

  • rocm-sdk-devel==10.0.0 installed, so the runtime image did need the devel tree added.
  • CMake resolved hip-lang through the /opt/rocm symlink: Found HIP: /opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel.
  • Using AMDGPU_TARGET from environment: gfx942;gfx950 / Multi-arch build: Enabling both HIP_FP8_TYPE_FNUZ (gfx942) and HIP_FP8_TYPE_E4M3 (gfx950), so one wheel covers both archs despite the Python 3.12 container.
  • Renamed to sglang_kernel-0.4.6.post1+rocm1000-cp310-abi3-manylinux2014_x86_64.whl.

The artifact is live: it is attached to the v0.4.6.post1 release in sgl-project/whl next to the +rocm700 and +rocm720 wheels, rocm1000/sglang-kernel/index.html on gh-pages has the matching entry, and the wheel contains the built sgl_kernel/common_ops.abi3.so.

ROCm 10 release image — dispatched and green. Run #34422374614 on this branch with rocm_version: rocm10, version: 0.5.19. The flavor selector expanded to exactly publish (rocm10, gfx942, all) and publish (rocm10, gfx950, all) and both passed, publishing lmsysorg/sglang:v0.5.19-rocm10-mi30x and -mi35x (new tags, colliding with nothing). Inspecting the published image configs confirms the flavor-to-stage mapping: the mi30x image carries GPU_ARCH=gfx942-rocm1000 and the mi35x image GPU_ARCH=gfx950-rocm1000, both with ROCM_HOME under site-packages/_rocm_sdk_devel as the ROCm 10 stack expects.

Local: captured the in-container script build_rocm.sh generates for 700, 720 and 1000, syntax-checked each, and diffed the 700 / 720 output against the pre-change script to confirm ROCm 10 support does not alter them. pre-commit run --files over the changed files passes.

Follow-ups

3rdparty/amd/wheel/sglang/pyproject.toml carries rocm700 and rocm720 extras that pin a published sglang-kernel wheel URL. Now that a +rocm1000 wheel exists, a rocm1000 extra can be added in a follow-up.

Accuracy Tests

N/A — release workflow change; model outputs are unaffected.

Speed Tests and Profiling

N/A — this changes which release artifacts are built.

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

cursoragent and others added 2 commits September 10, 2026 00:30
ROCm 10 is now the primary flavor for AMD PR and nightly tests (#38659) and
the nightly workflow already publishes rocm10-mi30x / rocm10-mi35x, but a
version tag only produced rocm700, rocm720 and rocm724 images. Add rocm10 to
the release matrix and the manual flavor selector, mapping it to the
gfx942-rocm1000 / gfx950-rocm1000 Dockerfile stages.

gfx1250 (mi45x) stays out of the release matrix: it is bring-up only and the
nightly keeps it in a separate job for that reason.

Co-authored-by: quitenode <quitenode@users.noreply.github.com>
The kernel release only covered ROCm 7.0 and 7.2, so the flavor AMD CI now
gates on has no published wheel. Add a 1000 leg to the ROCm build matrix and
a release-rocm1000 job that publishes it under the rocm1000 wheel index.

ROCm 10 is distributed as pip packages that unpack into site-packages, so the
build container has no /opt/rocm tree for CMakeLists_rocm.txt to find hip-lang
under and, being a runtime image, no devel tree to compile against. build_rocm.sh
applies the same fixups rocm.Dockerfile's rocm1000-base stage makes. That also
leaves no /opt/rocm-<version> directory for rename_wheels_rocm.sh to read the
wheel's local version tag from, so it now accepts one from the caller and keeps
the probe as the fallback for the 7.x flavors.

Co-authored-by: quitenode <quitenode@users.noreply.github.com>
Every ROCm leg is an image build on the shared AMD node, and its release job
appends the wheel to the index again, so a dispatch naming one flavor should
not drag the other two along. Build only the named flavor and publish only what
was built; a push still runs the full set.

Co-authored-by: quitenode <quitenode@users.noreply.github.com>

@HaiShaw HaiShaw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
Please purge rocm700 next.

@HaiShaw
HaiShaw merged commit f315243 into main Sep 10, 2026
107 of 111 checks passed
@HaiShaw
HaiShaw deleted the cursor/amd-rocm10-release-artifacts-767a branch September 10, 2026 06:57
mqhc2020 pushed a commit to mqhc2020/sglang that referenced this pull request Sep 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants