Repository navigation
fix(rocke): gfx942 dense KDB/VPF LSE max, knob types, shared softmax/PV (child of ROCm#13205) - #4
Merged
Parthiv911 merged 2 commits intoOct 12, 2026
Conversation
- The K-double-buffer pipeline folds its last KV tile in after the loop, but the LSE epilogue (develop #13140) read the loop's stale row max. The epilogue now reads the final max on both paths; the single-buffer IR is unchanged. - Spec rejects non-bool use_cfvst and non-int lds_num_buffers instead of coercing them; the V^T pad branch reads the resolved CFVST decision, so the pad and the CFVST choice can no longer disagree. - One softmax_pv helper serves tile_body and the pipeline (byte-identical IR). - Numeric coverage for every knob the bf16 CFVST gate opens, with and without the pipeline, plus an LSE x pipeline case (zero-trip and multi-tile, both grids); CPU test for the type rejections.
No default-knob case reaches the opt-in bf16 CFVST body or its K-double-buffer + V-prefetch pipeline, so their IR had no drift tripwire. Adds CFVST, pipeline (block_n 64/32, persistent) and LSE x pipeline cases for every LLVM flavor. Existing entries are unchanged.
Parthiv911
merged commit Oct 12, 2026
e151739
into
Parthiv911:users/Parthiv911/rocke/gfx942-dense-cfvst-kdb-vpf
23 of 27 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
JIRA ID: AICK-1900
@Parthiv911, this is a child PR for ROCm#13205. It targets your PR branch, so merging it here updates ROCm#13205. Please review it before you take it in. I have already pushed one commit to your branch: a merge of
develop(18f75d42f4a). That merge resolves the conflict intest_attention_dense_gfx942_numeric.py: both your strict test and develop's LSE tests are kept, each with its own GPU guard. The two commits in this PR come after that merge.What was wrong
loop.results[0]as the final row max. The K-double-buffer pipeline adds its last KV tile after the loop, so withemit_lse=True, lds_num_buffers=2the LSE paired a stale max with the final sum. On gfx942 silicon the LSE was off by 1.18 at S=512 and by 6.9e29 atn_up == 1, where the stale max is the −inf seed. The output O was still correct.use_cfvstwas read throughbool(), souse_cfvst="false"built a CFVST kernel. The pad resolver usedis True, so the pad and the CFVST decision could disagree.block_n=64row had numeric tests.finish_pipeline_tilecopiedtile_body's softmax/PV block. That copy is how the LSE state diverged.What changed
kernels/gfx942/attention_dense.py__post_init__rejects a non-booluse_cfvstand a non-intlds_num_buffers. The pad branch usesresolved_use_cfvst(). A singlesoftmax_pvhelper servestile_bodyand the pipeline, replacing the copy.tests/test_attention_dense_gfx942_numeric.pytest_cfvst_knob_matrix_numeric: 15 knob rows ×lds_num_buffers1/2.test_kdbvpf_pipeline_lse: zero-trip and multi-tile, on both grids.tests/test_attention_dense_gfx942.pytests/test_attention_dense_gfx942_golden.py,golden/*.jsonblock_n64/32, the persistent pipeline, and LSE × pipeline. That is 15 entries across LLVM 20/22/23.Why it works
Both branches now leave the final running max in
m_i, andlse_storereadsm_i. On the single-buffer path that is stillloop.results[0], so its IR does not change. Because both paths call the samesoftmax_pv, they cannot drift apart in mask, exp2, rescale or PV handling. Rejecting wrong types in the spec stops a mistyped value from silently selecting a different kernel.How we validated
n_up == 1dropped from 1.18 / 6.9e29 to 9.5e-7 / 4.8e-7.18f75d42f4a)rocke_engine).Notes
lds_num_buffers=1instead of loosening toward 4e-2.NOT_YET_IMPLEMENTED, share the gate predicate betweensupports_attention_denseanddense_rules.py, fix the stale_use_v_swizzledocstrings andvswz1tag, and updatedispatch/AGENTS.md:186. Also note in the PR description that the production sweep grows: fp16 D128 goes from 98 to 182 specs.