Repository navigation
feat(hipdnn): ragged tensors with per-batch origins, lengths and strides (ALMIOPEN-3151) - #15080
Merged
BrianHarrisonAMD merged 12 commits intoNov 6, 2026
Merged
Conversation
✅ All Checks Passed — Ready for Review
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
Pre-commit check failed⛔ pre-commit failed Please run locally:
This repo uses |
|
🎉 All checks passed! This PR is ready for review. |
…des (ALMIOPEN-3151) Ragged tensors in the hipDNN test stack follow cuDNN's layout rules: each batch has its own origin and length, and layouts may be head-major, strided or padded. - Data SDK RaggedTensor: per-batch origins and lengths (optional length tensor), element count and addressed span computed on the dense view, aliasing layouts rejected, offset and length tensors read at their batch stride. RaggedIterationInfo exposes origins and lengths; rowOffsets() returns the offset table of a contiguous, in-order layout. - SDPA seq_len_q / seq_len_kv bind per-batch lengths to ragged roles. - CPU and GPU SDPA references, the ragged-to-dense path and the bundle loader accept every container-valid layout. - Golden-data generator and verifier write and check the new layouts; a ragged tensor file holds exactly the addressed elements. - RFC 0014, RFC 0011, KNOWN_DIVERGENCES.md and the SDPA reference docs describe the rules.
BrianHarrisonAMD
force-pushed
the
users/bharriso/hipdnn-ragged-footprint
branch
from
November 4, 2026 20:54
d6a55db to
1e1eafd
Compare
- The dense rewrite declines a ragged role whose batch differs from Q's, and a seq_len tensor that is not INT32. - RaggedIterationInfo::rowOffsets() skips zero-length batches in its contiguity check. - The golden verifier ignores the unused batch-axis stride and rejects a ragged tensor bound to two seq_len tensors, as the C++ side does. - Tests: the off-axis offset test asserts the shape check; a bundle whose offsets use a doubled multiplier is rejected at load. - RFC 0014 specifies the length tensor under section 4.5 and documents the gate and S_max-cap limits; RFC status lines reference this PR.
- One helper checks the [count, 1, 1, 1] shape of ragged_offset and seq_len tensors in the container, the dense rewrite and the GPU reference. - The dense rewrite finds attribute-referenced tensors through the generated node-operand visitor instead of a hand-written list. - The golden verifier rejects ragged tensors that are not rank 4 and returns an empty footprint for a zero-extent axis before checking strides; the backward generator rejects ragged-only options given without --ragged-sides. - Shared graph-edit helper for the dense-rewrite tests; tests cover both seq_len tensors off the batch axis and an offset table named by a backward attribute. - RFC 0014 states the aux shapes and that ragged_offset[B] is unused with lengths; stale comments and the SEM-011 wording are updated. - KNOWN_DIVERGENCES.md is unchanged by this PR: the ragged rules are hipDNN behaviour documented in RFC 0014.
…rop an unused test parameter
…low-rank Q/K - The dense rewrite reads INT64 ragged_offset tables, as RFC 0014 documents, and declines a graph whose non-ragged Q or K is not rank 4 instead of throwing. - Tests cover INT64 offsets, both seq_len roles, a ragged role whose batch differs from Q's, rowOffsets() with leading and all-empty batches, and the verifier's seq_len binding errors. - The forward-support tests use the shared graph-edit helpers. - RFC 0014 states the [B, 1, 1, 1] length shape, the rowOffsets() rule for zero-length batches and the INT32-only seq_len of the dense rewrite; SEM-011 no longer implies the CPU reference lacks a ragged LSE.
…dnn-ragged-footprint
- The dense rewrite declines a ragged primary or offset table whose data type has no native mapping (for example INT8) instead of throwing, and no longer carries unreachable empty-dims checks. - A test reads INT64 offset tables through the strided path and checks the byte size and origins against INT32. - RFC 0014 and the reference docs list the graphs the dense rewrite declines and state the all-empty rowOffsets() case; two test comments now match their tests.
BrianHarrisonAMD
marked this pull request as ready for review
November 5, 2026 02:13
Bottom-right causal masking leaves one query row of the ragged Max / Sum_exp test with no visible key, so its Stats is -inf. EXPECT_NEAR on two -inf values computes NaN and fails with GoogleTest releases that do not special-case equal infinities. The test now checks such rows exactly (Stats and Max are -inf, Sum_exp is 0) and keeps the tolerance check for finite rows.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## develop #15080 +/- ##
===========================================
- Coverage 77.77% 77.77% -0.00%
===========================================
Files 2905 2906 +1
Lines 491621 491613 -8
Branches 74545 74533 -12
===========================================
- Hits 382350 382335 -15
+ Misses 84277 84276 -1
- Partials 24994 25002 +8
*This pull request uses carry forward flags. Click here to find out more.
🚀 New features to boost your workflow:
|
CMiservaAMD
approved these changes
Nov 5, 2026
Contributor
There was a problem hiding this comment.
No issues were found when merging these changes with my harness changes and generating test cases. One small question found by AI (apparently added as a separate review, below). I'm assuming the answer is yes but felt it was worth double-checking. Overall LGTM.
GarbyGilbertAMD
approved these changes
Nov 5, 2026
CMiservaAMD
reviewed
Nov 5, 2026
…egister SEM-026 records that the ragged container rejects overlapping token rows, sorted-stride violations, negative strides, origins that are not whole tokens and seq_len values outside [0, S_max], which the cuDNN frontend does not validate. RFC 0014 known limitation 8 and the reference support table link to it.
jerehartAMD
reviewed
Nov 5, 2026
…RFC 0014 Section 2.1 no longer calls [B, X, ...] the physical layout: the batch axis is logical, and token-major [T, H, D] and head-major [H, T, D] buffers are placed by per-batch origins and strides.
…dnn-ragged-footprint
jerehartAMD
approved these changes
Nov 5, 2026
dpeabody-amd
added a commit
that referenced
this pull request
Nov 7, 2026
…1151WmmaAttention hipDNN's paged K/V contract (SdpaSemanticsRegister SEM-008/009, #14358) differs from the one this engine was written against: - Every paged graph carries seq_len_q and seq_len_kv, also with a dense Q. The matcher now requires both and always binds seq_len_q. The dense-Q form without seq_len_q is gone: GraphLayout keeps PAGED_DENSE_Q (dense Q, live rows seq_len_q) and PAGED_RAGGED_Q, and an object without seqused_q serves no graph. - A dense Q's rows past seq_len_q, up to Sq, are padding with O = 0 and LSE = -inf. The kernel now writes them (an earlier commit); the GPU test expects that rather than untouched rows. - A ragged Q follows RFC 0014 (#15080): origins may come in any order, ragged_offset[B] is neither used nor validated, and the rows of an offset block past seq_len_q belong to no sequence. The kernel reads only each sequence's origin and writes only its live rows; the GPU test lays the sequences out in reverse order, poisons ragged_offset[B] and expects those rows untouched. - Separate K and V page tables (page_table_k, page_table_v) are valid graphs and are now served: the matcher binds both, and the launch passes them as the kernel's block_table and block_table_v (one table bound to both sides is passed twice). The two tables must have one width and, with more than one sequence, one row stride. - One-sided paging, per-side block sizes or page counts, K and V tables of different widths or row strides, and packed page tables are valid graphs the kernels cannot read, so each is declined with a logged cause. - An unset max_seq_len_kv is valid. The kernels use it only as the bound on seq_len_kv, so it is inferred as the table width times the block size, as cuDNN does. - A graph may declare a device-pointer alignment below the default 16 bytes (engine API 1.3.0, which the plugin's 1.4.0 admits). The kernels load Q / K / V / O rows as 16-byte vectors, INT32 operands as 4-byte words and the sink with 16-bit loads, so the matcher declines a graph declaring less for any of them (cause 'alignment', dense and paged) where hipDNN can still pick another engine, rather than the plan failing at execute. - left_bound follows cuDNN (L keys ending at the diagonal); maskFor already translates it to the kernel's window_left = L - 1. The GPU test's windows follow it (32-, 16- and 128-key windows). Catalog: the 48 paged objects with dense_q but without seqused_q (24 plain, 24 with sinks) could serve no hipDNN graph and are dropped; 96 paged objects remain (dense Q and ragged Q, each with and without sinks), 144 prefill objects in all. Retained descriptor ids are unchanged. The census is regenerated. Decline logs: the paged checks log the cause keys #14247 uses for the same contract rules, so the shared checks it moves into SdpaGraphMatch.hpp keep every key this engine logs: one_sided_paging; page_table (a packed table, K and V tables of different widths or row strides, a table that is not INT32 [B, 1, M, 1] with a unit entry stride); block_size (different K and V block sizes, a page size that is not a power of two, and one no paged object ships for, which used to find no object without a log); shape (different K and V page counts); seq_len_q, seq_len_kv and ragged (a missing or malformed sequence-length or offset tensor, declined silently before); attn_mask (additive bias on a paged graph). Each reason names the operand and values involved. The dense layout logs 'padding' for seq_len_q, seq_len_kv, max_seq_len_kv or padding_mask. Each line is headed by the declining engine's name (sdpa_graph::DeclinedBy): the decode engine runs the shared dense matcher too, and used to log under the prefill engine's name; it now also logs why it declines a paged graph. Tests: the matcher scaffolding both unit tests share (engine symbols, the graph fixture, object metadata, the decline-log reader, the ABI helpers) moves into Gfx1151WmmaAttentionTestSupport.hpp. The unit tests bind separate K and V tables (dense and ragged Q) and a V container with its own strides, and assert the engine, cause and reason of each decline: every paged check in isolation, the unshipped page sizes, alignment (sink included), FP8, dropout, a forced implementation, the other unserved features and ALiBi; the dense declines are asserted for both engines. The GPU test checks against test_sdk's CpuFpReferenceSdpa (gatherPagedKv per page table, sequence lengths, band, sink and LSE) instead of its own reference, adds two cases whose V pages sit in other blocks than their K pages, through a second page table, two whose V page, head and token strides all differ from K's, and runs the 96 and 256 buckets at pages 32 and 64 and the 128 bucket at page 64. Sinks: the engine still accepts a sink_token in the Q data type and declines a FLOAT one. hipDNN's frontend accepts only a FLOAT sink, so the GPU test's sink cases now skip with a message when the frontend rejects a Q-dtype sink; the paged-rules probe that skipped every case is removed, and the other cases run. Re-pin the gfx1151_wmma_attention.yaml digest in the IngestorGenerator fixture unslotted_render_golden.json (test_slots.py): the contract change alters that config's render by design.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ragged tensors in the hipDNN test stack now follow cuDNN's layout rules. Each batch has its own origin and length, and the layout can be head-major, strided or padded. Before this change, a ragged batch had to be one contiguous, in-order, token-major run whose length came from the offset differences. That ruled out layouts cuDNN, PyTorch and AOTriton use, such as head-major Stats
[H,T], fused-QKV views and per-batch lengths fromseq_len. The references, bundle loader and golden-data scripts can now express and validate those layouts.JIRA ID : ALMIOPEN-3151
Risk Assessment
Risk 3. The change is confined to test infrastructure and the Data SDK ragged container: engines, kernels, the graph API and the FlatBuffers schema do not change. Behaviour changes that reach users of the container:
RaggedIterationInfoexposes per-batchoriginsandlengths.rowOffsets()is now a method that returns the old offset table for in-order, contiguous layouts and throws otherwise.ragged_offsettensor must have shape[B + 1, 1, 1, 1]and aseq_lentensor[B, 1, 1, 1], with the values read at the batch-axis stride. Other shapes are rejected.RaggedLayoutTestSupport.hpp(shared ragged layout cases) andRaggedLengthUids.hpp(maps each ragged tensor of an SDPA node to itsseq_lentensor).All ragged golden bundles on
developload and verify under the new rules.ASIC Coverage
No product kernel changes. The HIP SDPA GPU reference kernel used by the integration tests now addresses per-batch origins, lengths and strides. It is verified on gfx942 (MI325X) and gfx950 (MI355X).
Testing Summary
HIPDNN_ENABLE_SDPA=ON): CPU reference, densify path and gates on head-major, strided, padded and shifted-origin layouts, each compared against the dense reference on the same data.seq_lentensors.develop's 8 forward ragged bundles checked against their golden outputs.Testing Checklist
hipdnn_data_sdk_tests- Status: Passed (1608)hipdnn_test_sdk_tests- Status: Passed (2682, 3 skipped)hipdnn_integration_tests_unit_tests- Status: Passed (1213, 2 skipped)pytest dnn-providers/integration-tests/reference-data-scripts/tests- Status: Passed (240)hipdnn_gpu_ref_tests, plus the integration unit tests on device - Status: Passed (54, 0 skipped; 1213)developforward ragged golden bundles -hipdnn_golden_data_tests --reference gpu --validator cpuon gfx942 and gfx950 - Status: Passed (8/8)Technical Changes
RaggedTensor: a ragged tensor is a per-batch origin (offset[b] × multiplier), a per-batch length and ordinary strides. Lengths come from an optional length tensor, or from the offset blocks when there is none. The element count, the addressed span and the non-aliasing rule are computed on the dense view. Aliasing layouts are rejected. Offset and length tensors are shape-checked and read at their batch stride.seq_len_q/seq_len_kvbind per-batch lengths to its ragged roles.seq_len_qcovers Q, O, Stats, Max and Sum_exp in forward, and Q, O, dO, dQ and Stats in backward;seq_len_kvcovers K, V, dK and dV. No schema change. A tensor bound to two differentseq_lentensors is an error.seq_lenwith ragged tensors. The dense path accepts INT32 and INT64 offset tables and declines graphs it cannot rewrite instead of failing.