Skip to content

[ROCm] ROCm7.2.2 + profiler fix + AITER 0.1.12.post2 #41386

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gshtras merged 15 commits into from
May 4, 2026
Merged

[ROCm] ROCm7.2.2 + profiler fix + AITER 0.1.12.post2 #41386

Changes from 14 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
acf8f1b
Bump ROCm to 7.2.2; rebuild CLR to fix profiling
Rohan138 Apr 15, 2026
339c381
Bump ROCm to 7.2.2; rebuild CLR to fix profiling
Rohan138 Apr 15, 2026
ebb8f5b
remove kineto patch to switch to rocprofiler-sdk; keep roctracer for now
Rohan138 Apr 16, 2026
12336f1
pin rocm-system repo and commit directly
Rohan138 Apr 16, 2026
fb18435
Update comment
Rohan138 Apr 16, 2026
05e6a06
Update comment
Rohan138 Apr 16, 2026
344ad5a
Add CLR deps
Rohan138 Apr 16, 2026
5241f3f
Add -y
Rohan138 Apr 16, 2026
e4947ea
Add apt update
Rohan138 Apr 16, 2026
b90e631
Bump torch to 2.11
gshtras Apr 13, 2026
14fa1d1
Bump aiter to v0.1.12.post1
Rohan138 Apr 20, 2026
99f15d3
patch fix AITER
Rohan138 Apr 21, 2026
adaa73b
Restore torch 2.10 and triton
gshtras Apr 22, 2026
b4fdd2e
Aiter 0.1.12
gshtras Apr 24, 2026
85a6f6a
rocm version in release pipeline
gshtras May 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 24 additions & 4 deletions docker/Dockerfile.rocm_base
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
ARG BASE_IMAGE=rocm/dev-ubuntu-22.04:7.2.1-complete
ARG BASE_IMAGE=rocm/dev-ubuntu-22.04:7.2.2-complete
ARG TRITON_BRANCH="ba5c1517"
ARG TRITON_REPO="https://github.com/ROCm/triton.git"
ARG PYTORCH_BRANCH="8514f051" # release/2.10 as of 3/17
Expand All @@ -9,7 +9,7 @@ ARG PYTORCH_AUDIO_BRANCH="v2.9.0"
ARG PYTORCH_AUDIO_REPO="https://github.com/pytorch/audio.git"
ARG FA_BRANCH="0e60e394"
ARG FA_REPO="https://github.com/Dao-AILab/flash-attention.git"
ARG AITER_BRANCH="v0.1.10.post3"
ARG AITER_BRANCH="v0.1.12.post2"
ARG AITER_REPO="https://github.com/ROCm/aiter.git"
ARG MORI_BRANCH="v1.1.0"
ARG MORI_REPO="https://github.com/ROCm/mori.git"
Expand Down Expand Up @@ -104,6 +104,28 @@ ENV SCCACHE_REGION=${USE_SCCACHE:+${SCCACHE_REGION_NAME}}
ENV SCCACHE_S3_NO_CREDENTIALS=${USE_SCCACHE:+${SCCACHE_S3_NO_CREDENTIALS}}
ENV SCCACHE_IDLE_TIMEOUT=${USE_SCCACHE:+0}

# torch profiler hotfix for 7.2.2: rebuild CLR with https://github.com/ROCm/rocm-systems/pull/5062
# will be removed once we move to ROCm 7.2.3
RUN apt-get update && apt-get install -y rocm-llvm-dev
RUN pip install CppHeaderParser
RUN git clone --no-checkout --filter=blob:none https://github.com/ROCm/rocm-systems /tmp/rocm-systems \
&& cd /tmp/rocm-systems \
&& git sparse-checkout init --cone \
&& git sparse-checkout set projects/hip projects/clr \
&& git checkout 35e8c7bf8911862e5389509800e65fdf125412b3 \
&& export CLR_DIR=/tmp/rocm-systems/projects/clr \
&& export HIP_DIR=/tmp/rocm-systems/projects/hip \
&& mkdir -p $CLR_DIR/build && cd $CLR_DIR/build \
&& cmake \
-DHIP_COMMON_DIR=$HIP_DIR \
-DCMAKE_PREFIX_PATH="/opt/rocm/" \
-DCLR_BUILD_HIP=ON \
-DCLR_BUILD_OCL=OFF \
-DHIP_PLATFORM=amd \
.. \
&& make -j$(nproc) \
&& make install \
&& rm -rf /tmp/rocm-systems
Comment on lines +109 to +128

@gemini-code-assist gemini-code-assist Bot Apr 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The hotfix for the torch profiler has a critical functional issue and an efficiency concern:

  1. Functional Issue: The cmake command is missing -DCMAKE_INSTALL_PREFIX="/opt/rocm/". By default, CMake installs to /usr/local. However, the LD_LIBRARY_PATH (defined on line 29) prioritizes /opt/rocm/lib over /usr/local/lib. This means the system will continue to load the original libraries from the base image instead of the hotfixed ones, rendering the fix ineffective.
  2. Image Efficiency: The hotfix is implemented across three separate RUN layers and does not clean up the apt cache. Consolidating these into a single layer and cleaning up /var/lib/apt/lists/* is standard practice to minimize image size and improve build performance.
RUN apt-get update && apt-get install -y --no-install-recommends rocm-llvm-dev \
    && pip install CppHeaderParser \
    && git clone --no-checkout --filter=blob:none https://github.com/ROCm/rocm-systems /tmp/rocm-systems \
    && cd /tmp/rocm-systems \
    && git sparse-checkout init --cone \
    && git sparse-checkout set projects/hip projects/clr \
    && git checkout 35e8c7bf8911862e5389509800e65fdf125412b3 \
    && export CLR_DIR=/tmp/rocm-systems/projects/clr \
    && export HIP_DIR=/tmp/rocm-systems/projects/hip \
    && mkdir -p $CLR_DIR/build && cd $CLR_DIR/build \
    && cmake \
        -DHIP_COMMON_DIR=$HIP_DIR \
        -DCMAKE_PREFIX_PATH="/opt/rocm/" \
        -DCMAKE_INSTALL_PREFIX="/opt/rocm/" \
        -DCLR_BUILD_HIP=ON \
        -DCLR_BUILD_OCL=OFF \
        -DHIP_PLATFORM=amd \
        .. \
    && make -j$(nproc) \
    && make install \
    && cd /app \
    && rm -rf /tmp/rocm-systems \
    && rm -rf /var/lib/apt/lists/*


###
### Triton Build
Expand Down Expand Up @@ -153,8 +175,6 @@ RUN git clone ${PYTORCH_REPO} pytorch
RUN cd pytorch && git checkout ${PYTORCH_BRANCH}
RUN cd pytorch \
&& pip install -r requirements.txt && git submodule update --init --recursive
RUN cd pytorch/third_party/kineto \
&& git remote add rocm https://github.com/ROCm/kineto && git fetch rocm && git checkout 2d73be3
RUN cd pytorch && python3 tools/amd_build/build_amd.py \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
Expand Down
Loading