Adds debug utility functions for CDNA3 MMA ops. by diptorupd · Pull Request #3 · ROCm/flashinfer

diptorupd · 2025-09-29T19:09:02Z

This PR adds debug utility functions specifically for CDNA3 MMA (Matrix Multiply Accumulate) operations in the HIP backend. The utilities provide debugging capabilities for matrix operations including initialization, loading, printing, and writing operations on shared memory arrays.

Key changes:

Adds comprehensive debug utilities for CDNA3 MMA operations
Provides functions for matrix fragment loading with A and B layout patterns
Includes utilities for printing and visualizing matrix data in shared memory

Copilot

Pull Request Overview

This PR adds debug utility functions specifically for CDNA3 MMA (Matrix Multiply Accumulate) operations in the HIP backend. The utilities provide comprehensive debugging capabilities for matrix operations in GPU shared memory.

Adds functions for initializing, loading, and printing matrix data in shared memory arrays
Implements matrix fragment loading with distinct A and B layout patterns for CDNA3 architecture
Provides utilities for visualizing and writing matrix data back to shared memory

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

libflashinfer/include/gpu_iface/backend/hip/mma_debug_utils_hip.h

This PR introduces a patch to includes Tested with Unit Tests: Test project /root/amd_eng/flashinfer/libflashinfer/tests/hip/build Start 1: MathTest 1/4 Test #1: MathTest ......................... Passed 3.25 sec Start 2: PosEncTest 2/4 Test #2: PosEncTest ....................... Passed 3.25 sec Start 3: CascadeTest 3/4 Test #3: CascadeTest ...................... Passed 3.24 sec Start 4: PageTest 4/4 Test #4: PageTest ......................... Passed 161.15 sec 100% tests passed, 0 tests failed out of 4

This PR fixes some of the unit test failures that occur in Single Decode. It also disables clang formatting of headers. The clang format of headers causes compilation issues. The compiler is unable to find `HIP WARP SYNC INTRINSICS` causing failures. Disabling clang format fixes these issues ``` Start 1: MathTest 1/6 Test #1: MathTest ......................... Passed 3.31 sec Start 2: PosEncTest 2/6 Test #2: PosEncTest ....................... Passed 3.36 sec Start 3: CascadeTest 3/6 Test #3: CascadeTest ...................... Passed 3.35 sec Start 4: PageTest 4/6 Test #4: PageTest ......................... Passed 114.08 sec Start 5: SingleDecodeTest 5/6 Test #5: SingleDecodeTest ................. Passed 35.22 sec Start 6: BatchDecodeTest 6/6 Test #6: BatchDecodeTest .................. Passed 559.75 sec 100% tests passed, 0 tests failed out of 6 Total Test time (real) = 719.07 sec ```

CPP test suite was using `hipified` headers. In this PR, we port over unit tests to use `gpu_iface`. This is necessary for us as the next step is to move the build infrastructure to use `gpu_iface` This PR has been tested locally ``` Test project /root/flashinfer/libflashinfer/tests/hip/build Start 1: MathTest 1/6 Test #1: MathTest ......................... Passed 3.40 sec Start 2: PosEncTest 2/6 Test #2: PosEncTest ....................... Passed 3.40 sec Start 3: CascadeTest 3/6 Test #3: CascadeTest ...................... Passed 985.27 sec Start 4: PageTest 4/6 Test #4: PageTest ......................... Passed 112.40 sec Start 5: SingleDecodeTest 5/6 Test #5: SingleDecodeTest ................. Passed 35.46 sec Start 6: BatchDecodeTest 6/6 Test #6: BatchDecodeTest .................. Passed 556.81 sec 100% tests passed, 0 tests failed out of 6 ``` To replicate the tests ``` cd flashinfer/libflashinfer/tests/hip ``` ``` mkdir build && cd build/ ``` ``` cmake -DCMAKE_PREFIX_PATH=/root/libtorch -DCMAKE_CXX_COMPILER:PATH=/opt/rocm/bin/amdclang++ -DFLASHINFER_INCLUDE_DIRS=/root/flashinfer/libflashinfer/include/ .. ``` ``` make ``` ``` ctest ```

In this PR I remove the `libtorch` dependency and removed `test_page.cpp`. `test_page.cpp` is the only unit test that uses libtorch. However, we also have a pytest for testing page. We will use that for validation. Removing the libtorch dependency will help us speed docker builds and remove additional dependencies. ```Test project /root/flashinfer/libflashinfer/tests/hip/build Start 1: MathTest 1/8 Test #1: MathTest ............................ Passed 0.31 sec Start 2: PosEncTest 2/8 Test #2: PosEncTest .......................... Passed 0.31 sec Start 3: CascadeTest 3/8 Test #3: CascadeTest ......................... Passed 1369.12 sec Start 4: SingleDecodeTest 4/8 Test #4: SingleDecodeTest .................... Passed 7726.35 sec Start 5: BatchDecodeTest 5/8 Test #5: BatchDecodeTest ..................... Passed 811.61 sec Start 6: test_mfma_fp32_16x16x16fp16 6/8 Test #6: test_mfma_fp32_16x16x16fp16 ......... Passed 0.30 sec Start 7: test_transpose_4x4_half_registers 7/8 Test #7: test_transpose_4x4_half_registers ... Passed 0.28 sec Start 8: test_rowsum 8/8 Test #8: test_rowsum ......................... Passed 0.27 sec 100% tests passed, 0 tests failed out of 8 ```

* Adds debug utility functions for CDNA3 MMA ops.

This PR introduces a patch to includes Tested with Unit Tests: Test project /root/amd_eng/flashinfer/libflashinfer/tests/hip/build Start 1: MathTest 1/4 Test ROCm#1: MathTest ......................... Passed 3.25 sec Start 2: PosEncTest 2/4 Test ROCm#2: PosEncTest ....................... Passed 3.25 sec Start 3: CascadeTest 3/4 Test ROCm#3: CascadeTest ...................... Passed 3.24 sec Start 4: PageTest 4/4 Test ROCm#4: PageTest ......................... Passed 161.15 sec 100% tests passed, 0 tests failed out of 4

This PR fixes some of the unit test failures that occur in Single Decode. It also disables clang formatting of headers. The clang format of headers causes compilation issues. The compiler is unable to find `HIP WARP SYNC INTRINSICS` causing failures. Disabling clang format fixes these issues ``` Start 1: MathTest 1/6 Test ROCm#1: MathTest ......................... Passed 3.31 sec Start 2: PosEncTest 2/6 Test ROCm#2: PosEncTest ....................... Passed 3.36 sec Start 3: CascadeTest 3/6 Test ROCm#3: CascadeTest ...................... Passed 3.35 sec Start 4: PageTest 4/6 Test ROCm#4: PageTest ......................... Passed 114.08 sec Start 5: SingleDecodeTest 5/6 Test ROCm#5: SingleDecodeTest ................. Passed 35.22 sec Start 6: BatchDecodeTest 6/6 Test ROCm#6: BatchDecodeTest .................. Passed 559.75 sec 100% tests passed, 0 tests failed out of 6 Total Test time (real) = 719.07 sec ```

CPP test suite was using `hipified` headers. In this PR, we port over unit tests to use `gpu_iface`. This is necessary for us as the next step is to move the build infrastructure to use `gpu_iface` This PR has been tested locally ``` Test project /root/flashinfer/libflashinfer/tests/hip/build Start 1: MathTest 1/6 Test ROCm#1: MathTest ......................... Passed 3.40 sec Start 2: PosEncTest 2/6 Test ROCm#2: PosEncTest ....................... Passed 3.40 sec Start 3: CascadeTest 3/6 Test ROCm#3: CascadeTest ...................... Passed 985.27 sec Start 4: PageTest 4/6 Test ROCm#4: PageTest ......................... Passed 112.40 sec Start 5: SingleDecodeTest 5/6 Test ROCm#5: SingleDecodeTest ................. Passed 35.46 sec Start 6: BatchDecodeTest 6/6 Test ROCm#6: BatchDecodeTest .................. Passed 556.81 sec 100% tests passed, 0 tests failed out of 6 ``` To replicate the tests ``` cd flashinfer/libflashinfer/tests/hip ``` ``` mkdir build && cd build/ ``` ``` cmake -DCMAKE_PREFIX_PATH=/root/libtorch -DCMAKE_CXX_COMPILER:PATH=/opt/rocm/bin/amdclang++ -DFLASHINFER_INCLUDE_DIRS=/root/flashinfer/libflashinfer/include/ .. ``` ``` make ``` ``` ctest ```

In this PR I remove the `libtorch` dependency and removed `test_page.cpp`. `test_page.cpp` is the only unit test that uses libtorch. However, we also have a pytest for testing page. We will use that for validation. Removing the libtorch dependency will help us speed docker builds and remove additional dependencies. ```Test project /root/flashinfer/libflashinfer/tests/hip/build Start 1: MathTest 1/8 Test ROCm#1: MathTest ............................ Passed 0.31 sec Start 2: PosEncTest 2/8 Test ROCm#2: PosEncTest .......................... Passed 0.31 sec Start 3: CascadeTest 3/8 Test ROCm#3: CascadeTest ......................... Passed 1369.12 sec Start 4: SingleDecodeTest 4/8 Test ROCm#4: SingleDecodeTest .................... Passed 7726.35 sec Start 5: BatchDecodeTest 5/8 Test ROCm#5: BatchDecodeTest ..................... Passed 811.61 sec Start 6: test_mfma_fp32_16x16x16fp16 6/8 Test ROCm#6: test_mfma_fp32_16x16x16fp16 ......... Passed 0.30 sec Start 7: test_transpose_4x4_half_registers 7/8 Test ROCm#7: test_transpose_4x4_half_registers ... Passed 0.28 sec Start 8: test_rowsum 8/8 Test ROCm#8: test_rowsum ......................... Passed 0.27 sec 100% tests passed, 0 tests failed out of 8 ```

* Adds debug utility functions for CDNA3 MMA ops.

This PR introduces a patch to includes Tested with Unit Tests: Test project /root/amd_eng/flashinfer/libflashinfer/tests/hip/build Start 1: MathTest 1/4 Test #1: MathTest ......................... Passed 3.25 sec Start 2: PosEncTest 2/4 Test #2: PosEncTest ....................... Passed 3.25 sec Start 3: CascadeTest 3/4 Test #3: CascadeTest ...................... Passed 3.24 sec Start 4: PageTest 4/4 Test #4: PageTest ......................... Passed 161.15 sec 100% tests passed, 0 tests failed out of 4

This PR fixes some of the unit test failures that occur in Single Decode. It also disables clang formatting of headers. The clang format of headers causes compilation issues. The compiler is unable to find `HIP WARP SYNC INTRINSICS` causing failures. Disabling clang format fixes these issues ``` Start 1: MathTest 1/6 Test #1: MathTest ......................... Passed 3.31 sec Start 2: PosEncTest 2/6 Test #2: PosEncTest ....................... Passed 3.36 sec Start 3: CascadeTest 3/6 Test #3: CascadeTest ...................... Passed 3.35 sec Start 4: PageTest 4/6 Test #4: PageTest ......................... Passed 114.08 sec Start 5: SingleDecodeTest 5/6 Test #5: SingleDecodeTest ................. Passed 35.22 sec Start 6: BatchDecodeTest 6/6 Test #6: BatchDecodeTest .................. Passed 559.75 sec 100% tests passed, 0 tests failed out of 6 Total Test time (real) = 719.07 sec ```

CPP test suite was using `hipified` headers. In this PR, we port over unit tests to use `gpu_iface`. This is necessary for us as the next step is to move the build infrastructure to use `gpu_iface` This PR has been tested locally ``` Test project /root/flashinfer/libflashinfer/tests/hip/build Start 1: MathTest 1/6 Test #1: MathTest ......................... Passed 3.40 sec Start 2: PosEncTest 2/6 Test #2: PosEncTest ....................... Passed 3.40 sec Start 3: CascadeTest 3/6 Test #3: CascadeTest ...................... Passed 985.27 sec Start 4: PageTest 4/6 Test #4: PageTest ......................... Passed 112.40 sec Start 5: SingleDecodeTest 5/6 Test #5: SingleDecodeTest ................. Passed 35.46 sec Start 6: BatchDecodeTest 6/6 Test #6: BatchDecodeTest .................. Passed 556.81 sec 100% tests passed, 0 tests failed out of 6 ``` To replicate the tests ``` cd flashinfer/libflashinfer/tests/hip ``` ``` mkdir build && cd build/ ``` ``` cmake -DCMAKE_PREFIX_PATH=/root/libtorch -DCMAKE_CXX_COMPILER:PATH=/opt/rocm/bin/amdclang++ -DFLASHINFER_INCLUDE_DIRS=/root/flashinfer/libflashinfer/include/ .. ``` ``` make ``` ``` ctest ```

In this PR I remove the `libtorch` dependency and removed `test_page.cpp`. `test_page.cpp` is the only unit test that uses libtorch. However, we also have a pytest for testing page. We will use that for validation. Removing the libtorch dependency will help us speed docker builds and remove additional dependencies. ```Test project /root/flashinfer/libflashinfer/tests/hip/build Start 1: MathTest 1/8 Test #1: MathTest ............................ Passed 0.31 sec Start 2: PosEncTest 2/8 Test #2: PosEncTest .......................... Passed 0.31 sec Start 3: CascadeTest 3/8 Test #3: CascadeTest ......................... Passed 1369.12 sec Start 4: SingleDecodeTest 4/8 Test #4: SingleDecodeTest .................... Passed 7726.35 sec Start 5: BatchDecodeTest 5/8 Test #5: BatchDecodeTest ..................... Passed 811.61 sec Start 6: test_mfma_fp32_16x16x16fp16 6/8 Test #6: test_mfma_fp32_16x16x16fp16 ......... Passed 0.30 sec Start 7: test_transpose_4x4_half_registers 7/8 Test #7: test_transpose_4x4_half_registers ... Passed 0.28 sec Start 8: test_rowsum 8/8 Test #8: test_rowsum ......................... Passed 0.27 sec 100% tests passed, 0 tests failed out of 8 ```

* Adds debug utility functions for CDNA3 MMA ops.

diptorupd requested review from Copilot, demandal25 and rtmadduri September 29, 2025 19:09

Copilot AI reviewed Sep 29, 2025

View reviewed changes

libflashinfer/include/gpu_iface/backend/hip/mma_debug_utils_hip.h Show resolved Hide resolved

demandal25 approved these changes Oct 1, 2025

View reviewed changes

Adds debug utility functions for CDNA3 MMA ops.

68aa735

diptorupd force-pushed the feature/mma_debug_utils branch from fb6e23a to 68aa735 Compare October 1, 2025 16:21

Pre-commit fixes

82188b0

diptorupd merged commit 4518727 into amd-integration Oct 1, 2025
1 check passed

diptorupd deleted the feature/mma_debug_utils branch October 2, 2025 16:45

diptorupd added a commit that referenced this pull request Dec 5, 2025

Adds debug utility functions for CDNA3 MMA ops. (#3)

9b3363e

* Adds debug utility functions for CDNA3 MMA ops.

zhenhantech pushed a commit to zhenhantech/flashinfer that referenced this pull request Jan 9, 2026

Adds debug utility functions for CDNA3 MMA ops. (ROCm#3)

98bf10e

* Adds debug utility functions for CDNA3 MMA ops.

diptorupd added a commit that referenced this pull request Feb 2, 2026

Adds debug utility functions for CDNA3 MMA ops. (#3)

118cdc2

* Adds debug utility functions for CDNA3 MMA ops.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds debug utility functions for CDNA3 MMA ops.#3

Adds debug utility functions for CDNA3 MMA ops.#3
diptorupd merged 2 commits intoamd-integrationfrom
feature/mma_debug_utils

diptorupd commented Sep 29, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

diptorupd commented Sep 29, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants