Update Changelog by diptorupd · Pull Request #1 · AMD-Ecosystem/flashinfer

diptorupd · 2025-09-29T18:10:34Z

Adds an updated Changelog entry for the v0.2.5+rocm release

Copilot

Pull Request Overview

This PR adds a changelog entry for the v0.2.5+rocm release, documenting the initial technical preview of a ROCm port of FlashInfer. The update replaces extensive existing changelog content with a concise summary of ROCm-specific changes.

Key Changes

Replaces comprehensive changelog with v0.2.5+rocm release notes
Documents the initial ROCm port including decode kernels and infrastructure changes
Lists specific contributions related to HIP support, build system updates, and GPU interoperability

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

rtmadduri · 2025-09-29T20:41:49Z

LGTM

This PR introduces a patch to includes Tested with Unit Tests: Test project /root/amd_eng/flashinfer/libflashinfer/tests/hip/build Start 1: MathTest 1/4 Test #1: MathTest ......................... Passed 3.25 sec Start 2: PosEncTest 2/4 Test #2: PosEncTest ....................... Passed 3.25 sec Start 3: CascadeTest 3/4 Test #3: CascadeTest ...................... Passed 3.24 sec Start 4: PageTest 4/4 Test #4: PageTest ......................... Passed 161.15 sec 100% tests passed, 0 tests failed out of 4

This PR fixes some of the unit test failures that occur in Single Decode. It also disables clang formatting of headers. The clang format of headers causes compilation issues. The compiler is unable to find `HIP WARP SYNC INTRINSICS` causing failures. Disabling clang format fixes these issues ``` Start 1: MathTest 1/6 Test #1: MathTest ......................... Passed 3.31 sec Start 2: PosEncTest 2/6 Test #2: PosEncTest ....................... Passed 3.36 sec Start 3: CascadeTest 3/6 Test #3: CascadeTest ...................... Passed 3.35 sec Start 4: PageTest 4/6 Test #4: PageTest ......................... Passed 114.08 sec Start 5: SingleDecodeTest 5/6 Test #5: SingleDecodeTest ................. Passed 35.22 sec Start 6: BatchDecodeTest 6/6 Test #6: BatchDecodeTest .................. Passed 559.75 sec 100% tests passed, 0 tests failed out of 6 Total Test time (real) = 719.07 sec ```

CPP test suite was using `hipified` headers. In this PR, we port over unit tests to use `gpu_iface`. This is necessary for us as the next step is to move the build infrastructure to use `gpu_iface` This PR has been tested locally ``` Test project /root/flashinfer/libflashinfer/tests/hip/build Start 1: MathTest 1/6 Test #1: MathTest ......................... Passed 3.40 sec Start 2: PosEncTest 2/6 Test #2: PosEncTest ....................... Passed 3.40 sec Start 3: CascadeTest 3/6 Test #3: CascadeTest ...................... Passed 985.27 sec Start 4: PageTest 4/6 Test #4: PageTest ......................... Passed 112.40 sec Start 5: SingleDecodeTest 5/6 Test #5: SingleDecodeTest ................. Passed 35.46 sec Start 6: BatchDecodeTest 6/6 Test #6: BatchDecodeTest .................. Passed 556.81 sec 100% tests passed, 0 tests failed out of 6 ``` To replicate the tests ``` cd flashinfer/libflashinfer/tests/hip ``` ``` mkdir build && cd build/ ``` ``` cmake -DCMAKE_PREFIX_PATH=/root/libtorch -DCMAKE_CXX_COMPILER:PATH=/opt/rocm/bin/amdclang++ -DFLASHINFER_INCLUDE_DIRS=/root/flashinfer/libflashinfer/include/ .. ``` ``` make ``` ``` ctest ```

In this PR I remove the `libtorch` dependency and removed `test_page.cpp`. `test_page.cpp` is the only unit test that uses libtorch. However, we also have a pytest for testing page. We will use that for validation. Removing the libtorch dependency will help us speed docker builds and remove additional dependencies. ```Test project /root/flashinfer/libflashinfer/tests/hip/build Start 1: MathTest 1/8 Test #1: MathTest ............................ Passed 0.31 sec Start 2: PosEncTest 2/8 Test #2: PosEncTest .......................... Passed 0.31 sec Start 3: CascadeTest 3/8 Test #3: CascadeTest ......................... Passed 1369.12 sec Start 4: SingleDecodeTest 4/8 Test #4: SingleDecodeTest .................... Passed 7726.35 sec Start 5: BatchDecodeTest 5/8 Test #5: BatchDecodeTest ..................... Passed 811.61 sec Start 6: test_mfma_fp32_16x16x16fp16 6/8 Test #6: test_mfma_fp32_16x16x16fp16 ......... Passed 0.30 sec Start 7: test_transpose_4x4_half_registers 7/8 Test #7: test_transpose_4x4_half_registers ... Passed 0.28 sec Start 8: test_rowsum 8/8 Test #8: test_rowsum ......................... Passed 0.27 sec 100% tests passed, 0 tests failed out of 8 ```

This PR introduces a patch to includes Tested with Unit Tests: Test project /root/amd_eng/flashinfer/libflashinfer/tests/hip/build Start 1: MathTest 1/4 Test #1: MathTest ......................... Passed 3.25 sec Start 2: PosEncTest 2/4 Test #2: PosEncTest ....................... Passed 3.25 sec Start 3: CascadeTest 3/4 Test #3: CascadeTest ...................... Passed 3.24 sec Start 4: PageTest 4/4 Test #4: PageTest ......................... Passed 161.15 sec 100% tests passed, 0 tests failed out of 4

This PR fixes some of the unit test failures that occur in Single Decode. It also disables clang formatting of headers. The clang format of headers causes compilation issues. The compiler is unable to find `HIP WARP SYNC INTRINSICS` causing failures. Disabling clang format fixes these issues ``` Start 1: MathTest 1/6 Test #1: MathTest ......................... Passed 3.31 sec Start 2: PosEncTest 2/6 Test #2: PosEncTest ....................... Passed 3.36 sec Start 3: CascadeTest 3/6 Test #3: CascadeTest ...................... Passed 3.35 sec Start 4: PageTest 4/6 Test #4: PageTest ......................... Passed 114.08 sec Start 5: SingleDecodeTest 5/6 Test #5: SingleDecodeTest ................. Passed 35.22 sec Start 6: BatchDecodeTest 6/6 Test #6: BatchDecodeTest .................. Passed 559.75 sec 100% tests passed, 0 tests failed out of 6 Total Test time (real) = 719.07 sec ```

CPP test suite was using `hipified` headers. In this PR, we port over unit tests to use `gpu_iface`. This is necessary for us as the next step is to move the build infrastructure to use `gpu_iface` This PR has been tested locally ``` Test project /root/flashinfer/libflashinfer/tests/hip/build Start 1: MathTest 1/6 Test #1: MathTest ......................... Passed 3.40 sec Start 2: PosEncTest 2/6 Test #2: PosEncTest ....................... Passed 3.40 sec Start 3: CascadeTest 3/6 Test #3: CascadeTest ...................... Passed 985.27 sec Start 4: PageTest 4/6 Test #4: PageTest ......................... Passed 112.40 sec Start 5: SingleDecodeTest 5/6 Test #5: SingleDecodeTest ................. Passed 35.46 sec Start 6: BatchDecodeTest 6/6 Test #6: BatchDecodeTest .................. Passed 556.81 sec 100% tests passed, 0 tests failed out of 6 ``` To replicate the tests ``` cd flashinfer/libflashinfer/tests/hip ``` ``` mkdir build && cd build/ ``` ``` cmake -DCMAKE_PREFIX_PATH=/root/libtorch -DCMAKE_CXX_COMPILER:PATH=/opt/rocm/bin/amdclang++ -DFLASHINFER_INCLUDE_DIRS=/root/flashinfer/libflashinfer/include/ .. ``` ``` make ``` ``` ctest ```

In this PR I remove the `libtorch` dependency and removed `test_page.cpp`. `test_page.cpp` is the only unit test that uses libtorch. However, we also have a pytest for testing page. We will use that for validation. Removing the libtorch dependency will help us speed docker builds and remove additional dependencies. ```Test project /root/flashinfer/libflashinfer/tests/hip/build Start 1: MathTest 1/8 Test #1: MathTest ............................ Passed 0.31 sec Start 2: PosEncTest 2/8 Test #2: PosEncTest .......................... Passed 0.31 sec Start 3: CascadeTest 3/8 Test #3: CascadeTest ......................... Passed 1369.12 sec Start 4: SingleDecodeTest 4/8 Test #4: SingleDecodeTest .................... Passed 7726.35 sec Start 5: BatchDecodeTest 5/8 Test #5: BatchDecodeTest ..................... Passed 811.61 sec Start 6: test_mfma_fp32_16x16x16fp16 6/8 Test #6: test_mfma_fp32_16x16x16fp16 ......... Passed 0.30 sec Start 7: test_transpose_4x4_half_registers 7/8 Test #7: test_transpose_4x4_half_registers ... Passed 0.28 sec Start 8: test_rowsum 8/8 Test #8: test_rowsum ......................... Passed 0.27 sec 100% tests passed, 0 tests failed out of 8 ```

…raph, return_lse (#234) ## Summary The AITER PA v1 decode backend on `amd-integration` has three call patterns that produce wrong output, hard crashes, or unhelpful `NotImplementedError`s. This PR fixes each one at the level it's actually broken at, rather than blanket-disabling AITER. | Case | Behavior on `amd-integration` | This PR | |---|---|---| | Sliding-window attention (`window_left >= 0`) | AITER selected. Wrapper passes `sliding_window = window_left` (off-by-one), and `window_left = 0` collides with AITER's "disabled" sentinel — silently wrong output. | **AITER runs** with corrected convention mapping (`window_left + 1`). | | `use_cuda_graph=True` with explicit `backend="aiter"` | AITER selected. Per-plan scalars (`max_kv_len`, `max_blocks_per_seq`) are baked into the captured graph; replay against a larger batch launches with an under-sized grid. | Clear `ValueError` at `plan()` time. (auto-select already routes to `fa2`.) | | `run(return_lse=True)` | Raises `NotImplementedError("AITER decode backend does not currently return LSE")`. | Transparent dispatch through a pre-built FA2 shadow plan; one-time warning at `plan()` time so the per-call backend switch is not silent. | ## Why each fix ### 1. Sliding window — wrapper convention bug, not a kernel gap AITER PA v1's kernel *does* implement window masking (`csrc/cpp_itfs/pa/pa_kernels.cuh:457`): ```cpp if (local_token_idx + i < context_len - sliding_window) tmp = -FLT_MAX; ``` gated by the `sliding_window_enabled` template flag (set by the compile step at `csrc/cpp_itfs/pa/pa_v1.py:144`). The wrapper already plumbs `sliding_window` through `_aiter_pa_v1_resolve` and the run-time call. The bug on trunk is a **convention difference**: - FlashInfer: `window_left = W` → query at position `kv_len-1` sees positions `[kv_len-1-W, kv_len-1]` = `W+1` tokens. - AITER: `sliding_window = S` (with `S > 0` enabling the mask) → admits `S` tokens. - `S = 0` is AITER's compile-time "disabled" sentinel. Trunk passes `sliding_window = window_left` — off by one, plus `window_left = 0` (one visible token) collides with AITER's disabled sentinel and silently returns full attention. Fixed: `sliding_window = window_left + 1` when `window_left >= 0`, else 0. This keeps AITER on the hot path for sliding-window models (Gemma, Mistral, etc.) instead of giving up perf to FA2. ### 2. CUDA graph — wrapper-level limitation, hard-rejected in explicit path AITER's launch grid is computed at `plan()` time from `max_kv_len` and `max_blocks_per_seq` of the *current* batch and passed by value to the kernel launch. Under CUDA-graph capture these scalars are baked into the captured graph and can't be widened on replay against a larger batch. Supporting this properly would require capturing with worst-case dimensions — a new API parameter (e.g. `max_seq_len_per_request`) — which is out of scope for this PR. The auto-select fallback to FA2 stays in place; the explicit `backend="aiter"` path (which on trunk silently produces broken launches) now raises: ``` ValueError: AITER decode backend is incompatible with CUDA-graph capture: the kernel's launch grid is sized from per-plan scalars (max_kv_len, max_blocks_per_seq) that are baked into the captured graph at capture time. Use backend='fa2' for CUDA-graph workflows, or backend='auto' which routes around this automatically. ``` ### 3. return_lse — replace NotImplementedError with transparent fallback AITER PA v1 does not output LSE (only `out`; the kernel computes per-partition `exp_sums`/`max_logits` internally for split-K but does not expose normalized LSE). Trunk raises `NotImplementedError` at `run()` time, breaking any caller that needs LSE under an AITER plan. This PR pre-builds an FA2 decode plan at AITER `plan()` time and dispatches through it whenever `return_lse=True` arrives at `run()`. This is the only correct option since `return_lse` is per-call, not per-plan. Two details worth flagging: - The shadow plan uses the real `window_left` (and the corresponding template flag), so it produces correct LSE under sliding-window AITER plans (now supported per fix #1). - A one-time-per-device warning is emitted at AITER plan() time. Without it, a user toggling `return_lse=True` on a hot path would silently move from AITER → FA2 with no signal. ## Tests added `tests/rocm_tests/test_batch_decode_aiter_hip.py`: - `test_batch_decode_aiter_sliding_window_vs_fa2` — AITER↔FA2 parity over `window_left ∈ {0, 31, 127, 1023}`, fp16/bf16, batch sizes, GQA ratios, including the saturation regime (`window_left >= max_kv_len-1`) that exercises the kernel's no-op masking branch. - `test_batch_decode_aiter_return_lse_via_fa2` — verifies (a) `return_lse=False` still runs through AITER, (b) `return_lse=True` falls back to the shadow FA2 plan and returns `(output, lse)` matching the pure-FA2 reference, with and without sliding window. - `test_batch_decode_auto_routes_cuda_graph_to_fa2` — `backend="auto"` + `use_cuda_graph=True` resolves to `fa2`. - Extended `test_batch_decode_aiter_rejects_invalid_config` — explicit `backend="aiter"` + `use_cuda_graph=True` raises a `ValueError` mentioning "CUDA-graph". ## Test plan - [x] `pytest tests/rocm_tests/test_batch_decode_aiter_hip.py -v` — **178 passed** in 69 s. New parity + LSE-fallback + CUDA-graph rejection cases. - [x] `pytest tests/rocm_tests/test_sliding_window_hip.py -m "not slow"` — **1248 passed** in 60 s. Exercises the AITER path on every sliding-window decode shape that was previously silently wrong on trunk. - [x] `pytest tests/rocm_tests/test_batch_decode_kernels_hip.py -m "not slow" -n auto --reruns 2` — **1872 passed** in 174 s. Covers the broader decode matrix including `return_lse=True` (which on trunk raised `NotImplementedError` under AITER) and CUDA-graph wrappers (which now route to FA2 cleanly). ## API impact - `BatchDecodeWithPagedKVCacheWrapper` docstring updated to document the AITER-specific constraints (CUDA-graph incompatible; sliding-window supported transparently; `return_lse` falls back to FA2). - No public signature changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

diptorupd requested review from Copilot and rtmadduri September 29, 2025 18:11

Copilot AI reviewed Sep 29, 2025

View reviewed changes

Comment thread CHANGELOG.md

Comment thread CHANGELOG.md Outdated

Comment thread CHANGELOG.md Outdated

demandal25 self-requested a review September 29, 2025 18:13

diptorupd and others added 3 commits September 29, 2025 16:31

Update Changelog

3ff6b9a

Update CHANGELOG.md

dfdd841

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update CHANGELOG.md

e49c9e5

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

diptorupd force-pushed the docs/changelog branch from 3e3b6dc to e49c9e5 Compare September 29, 2025 20:31

rtmadduri approved these changes Sep 29, 2025

View reviewed changes

Fix trailing whitespace

606f3a4

demandal25 approved these changes Sep 29, 2025

View reviewed changes

rtmadduri merged commit db27195 into amd-integration Sep 29, 2025
1 check passed

rtmadduri deleted the docs/changelog branch September 29, 2025 21:01

rtmadduri restored the docs/changelog branch September 29, 2025 21:02

rtmadduri deleted the docs/changelog branch September 29, 2025 21:02

demandal25 mentioned this pull request May 20, 2026

fix(rocm): correct AITER decode backend gaps — sliding window, CUDA graph, return_lse #234

Merged

3 tasks

Navigation Menu

Uh oh!

Update Changelog - #1

Update Changelog#1
rtmadduri merged 4 commits into
amd-integrationfrom
docs/changelog

diptorupd commented Sep 29, 2025 •

edited by demandal25

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rtmadduri commented Sep 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

diptorupd commented Sep 29, 2025 • edited by demandal25 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Key Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rtmadduri commented Sep 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

diptorupd commented Sep 29, 2025 •

edited by demandal25

Loading