Add bool mask vs float additive bias guidance to debugging-memcpy skill - #272
Merged
Conversation
Performance Comparison
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
37 tasks
Contributor
|
LGTM with three small suggestions. Verified the technical claims against 1. Scope the section to the ONNX Attention op (suggested replacement for the section header on line 332): ## Bool mask vs float additive bias for attention
> Applies to the ONNX-domain `Attention` op (opset 23/24) that mobius emits.
> Contrib `MultiHeadAttention` takes a pre-built float `attention_bias`;
> contrib `GroupQueryAttention` does not accept any bias.The bool-mask + 2. Wording nit on the "double-applies" bullet (lines 358–359): - **`is_causal=1` + bool mask** is redundant and easy to get out of sync —
the `is_causal` flag adds its own causal mask on top of the explicit one,
so any off-by-one between them silently changes resultsIn practice MEA ( 3. Flash prerequisites + Bool masks do **NOT** enable Flash Attention. Flash Attention requires
`attn_mask=nullptr` (no mask at all). Both bool and float masks route
to Memory-Efficient Attention (MEA) or unfused attention. If you need
Flash Attention, use `is_causal=1` with no explicit mask. Note that
Flash also requires fp16/bf16, `head_size == v_head_size` (≤ 256),
and SM ≥ 8.0 (Ampere+); otherwise the op falls to MEA or unfused
regardless of how the mask is expressed. For variable-length padding,
prefer `nonpad_kv_seqlen` (opset 24) over a mask — it preserves Flash
eligibility, while a mask hard-disables Flash.
|
titaiwangms
approved these changes
May 6, 2026
…py skill Document when to use float additive bias vs bool mask for attention: - Float bias recommended for complex patterns (sliding window, KV-shared, dual head_dim, padding) - Bool mask only for simple causal-only patterns - Common misconception: bool masks do NOT enable Flash Attention - Decision table and code example using create_attention_bias() Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Add 5 dispatch reference tables to debugging-memcpy skill: - Contrib MultiHeadAttention cascade (7 kernels) - Contrib GroupQueryAttention cascade (4 kernels, rejects bias) - ONNX Attention MHA cascade (3 kernels) - ONNX Attention GQA cascade (3 kernels) - GQA + float mask conditions (MEA vs unfused) Explains why Gemma4 KV-shared layers fall to unfused (asymmetric head_size) and key rules (Flash requires nullptr mask, SM>=8.0). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
justinchuby
force-pushed
the
skill-attn-mask-guidance
branch
from
May 6, 2026 17:09
9c9790d to
9f73366
Compare
…ng-memcpy Extract attention content from debugging-memcpy into a new standalone attention-optimization skill covering: - Bool mask vs float additive bias (when to use each) - Flash Attention requirements and nonpad_kv_seqlens - 5 ORT CUDA attention kernel dispatch tables - GQA vs ONNX Attention tradeoffs - Key takeaways for model builders debugging-memcpy now focuses purely on memcpy debugging with a cross-reference to the new skill. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Add top-level decision table mapping scenarios (causal, padding, sliding window, complex, custom) to recommended mask types with rationale. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
- nonpad_kv_seqlens is best for padding (enables Flash + shared buffer) - Flash primarily helps prefill, not decode (memory-bandwidth bound) - Gemma4 can't use Flash for any layer (sliding needs mask, full has head_dim=512) — MEA is the effective best kernel - Bool and float masks have identical dispatch (bool→float internally) - Updated decision table with rationale Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Bool mask is equally precise as float for sliding window. We chose float for Gemma4 because of a bug in our bool mask construction for complex multi-constraint patterns (KV-shared + sliding + dual head_dim), not a fundamental limitation. For simpler models, bool is fine and uses less memory. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a section to the debugging-memcpy skill documenting when to use float additive bias vs bool mask for attention.
Key points
is_causal=1sufficesattn_mask=nullptrcreate_attention_bias()recommendationLearned from Gemma4 debugging where bool mask construction for mixed sliding-window + KV-shared + dual head_dim was error-prone.