docs: fix CUDA Attention skill-doc guidance to match bottom-right causal + fully-masked errata - #28988
Closed
titaiwangms wants to merge 1 commit into
Closed
docs: fix CUDA Attention skill-doc guidance to match bottom-right causal + fully-masked errata#28988titaiwangms wants to merge 1 commit into
titaiwangms wants to merge 1 commit into
Conversation
… fully-masked Y=0 + mode-3=0 The cuda-attention-kernel-patterns skill doc gave guidance opposite to the shipped behavior in #28958 (onnx/onnx#8068 errata): - is_causal=1 with an external/static KV cache (nonpad_kv_seqlen, no past_key) uses bottom-right (offset-aware) alignment and IS valid for decode — the doc previously called it spec-invalid and told models to use is_causal=0. - Fully-masked query rows output Y=0 on BOTH EPs (CPU Bug-2 guard + CUDA ZeroFullyMaskedRowsKernel); removed the stale mean(V) 'spec reference' claim and the resolved cross-EP TODO. - Documented qk_matmul_output mode-3 fully-masked row = 0 (mandated, consistent with Y), and that CUDA returns NOT_IMPLEMENTED for mode-3 (CPU-only path). - Corrected the bottom-right key-range bullet (was backwards) to the j <= i + offset form matching the rest of the doc and the kernel. Docs-only; no code/kernel change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Ti-Tai Wang <titaiwang@microsoft.com>
Contributor
Author
|
Folding this docs change into #28958 since it documents the same kernel behavior (bottom-right is_causal, fully-masked Y=0, mode-3=0) — keeping it atomic with the kernel change for review/merge. Closing in favor of #28958, which now carries the SKILL.md correction as a dedicated docs commit (1964d9b). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Docs-only correction to the hand-written agent skill doc
.agents/skills/cuda-attention-kernel-patterns/SKILL.md. No code or kernel change.Why
The skill doc gave guidance opposite to the behavior shipped in #28958 (which implements the onnx/onnx#8068 errata). It would actively misdirect agents working on the ONNX-domain Attention op:
is_causal=1+ external/static KV cache (nonpad_kv_seqlen, nopast_key): the doc called this spec-invalid and told models to useis_causal=0. It is in fact valid and supported, using bottom-right (offset-aware) alignment —offset[b] = nonpad_kv_seqlen[b] - q_sequence_length(clamped>= 0). Corrected, including the bottom-right key-range bullet that was written backwards.Y = 0on BOTH EPs (CPU Bug-2 guard + CUDAZeroFullyMaskedRowsKernel). Removed the stalemean(V)'spec reference' claim and the now-resolved cross-EP TODO.qk_matmul_outputmode-3: documented fully-masked row =0(mandated, consistent withY), and that CUDA returnsNOT_IMPLEMENTEDfor mode-3 (it is a CPU-only path).Grounding
Matches the landed kernel:
core/providers/cuda/llm/attention.cc(bottom-right dispatch, mode-3NOT_IMPLEMENTED),attention_mask_impl.cu(ZeroFullyMaskedRowsKernel),core/providers/cpu/llm/attention.cc(Bug-2 row-zeroing). See #28958 and onnx/onnx#8068.Risk
Documentation only — no build/runtime impact.