Skip to content

Commit 9b4463b

Browse files
justinchubyCopilot
andcommitted
docs: correct sliding window guidance — bool is equally precise
Bool mask is equally precise as float for sliding window. We chose float for Gemma4 because of a bug in our bool mask construction for complex multi-constraint patterns (KV-shared + sliding + dual head_dim), not a fundamental limitation. For simpler models, bool is fine and uses less memory. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
1 parent 69c1291 commit 9b4463b

1 file changed

Lines changed: 22 additions & 22 deletions

File tree

  • .agents/skills/attention-optimization

.agents/skills/attention-optimization/SKILL.md

Lines changed: 22 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -24,8 +24,8 @@ Use this skill when:
2424
|----------|------------|-----|
2525
| Causal only | `attn_mask=None` + `is_causal=1` | Enables Flash (fastest for prefill) |
2626
| Padding (batch>1) | `nonpad_kv_seqlens` (best) or bool mask | `nonpad_kv_seqlens` enables Flash + shared buffer with no mask |
27-
| Sliding window | Float additive bias | Precise window control |
28-
| Complex (causal+sliding+padding) | Float additive bias | Most flexible, avoids construction bugs |
27+
| Sliding window (simple) | Bool mask | Equally precise as float, uses less memory |
28+
| Complex (sliding+KV-shared+dual head_dim) | Float additive bias | Avoids mask construction bugs in multi-constraint patterns |
2929
| Custom pattern | Float additive bias | Arbitrary values |
3030

3131
## Bool mask vs float additive bias
@@ -35,31 +35,31 @@ Use this skill when:
3535
| Pattern | Recommended mask type |
3636
|---------|----------------------|
3737
| Simple causal-only | No mask — use `is_causal=1` (enables Flash) |
38-
| Sliding window | Float additive bias |
38+
| Sliding window (simple model) | Bool mask (precise, less memory) |
3939
| KV-shared layers | Float additive bias |
4040
| Mixed head_dim (e.g. Gemma4) | Float additive bias |
41-
| Padding + causal | Float additive bias |
41+
| Padding + causal | `nonpad_kv_seqlens` or bool mask |
42+
| Multiple constraints combined | Float additive bias |
4243

4344
### Why float additive bias is safer for complex patterns
4445

45-
ONNX `Attention` supports bool mask (`True`=attend, `False`=ignore).
46-
ORT correctly converts bool→float internally via
47-
`ConvertAttnMaskToBias()`. However, constructing correct bool masks
48-
for complex patterns is error-prone:
49-
50-
- **Sliding window boundaries** must align with KV cache positions —
51-
off-by-one errors silently produce wrong attention patterns
52-
- **KV-shared layers** borrow K/V from other layers — the mask shape
53-
must match the borrowed KV dimensions, not the current layer's
54-
- **`is_causal=1` + bool mask** double-applies constraints — the
55-
`is_causal` flag adds its own causal mask on top of the explicit one
56-
- **Dual head_dim** (e.g. Gemma4 local=128, global=256) means mask
57-
shapes differ per layer type
58-
59-
Float additive bias gives explicit control:
60-
- `0.0` for "attend" positions
61-
- `-inf` (or `-10000.0`) for "ignore" positions
62-
- No ambiguity in kernel interpretation
46+
Bool mask and float additive bias are **equally precise** — both can
47+
represent any attention pattern. ORT converts bool→float internally
48+
via `ConvertAttnMaskToBias()`, so they have identical kernel dispatch.
49+
50+
The reason we use float bias for complex models is **bug avoidance**,
51+
not a fundamental limitation of bool masks. Constructing correct bool
52+
masks for multi-constraint patterns is error-prone:
53+
54+
- **Sliding window + KV-shared** — mask shape must match borrowed KV
55+
dimensions, not the current layer's. Off-by-one errors are silent.
56+
- **`is_causal=1` + bool mask** — double-applies causal constraints
57+
- **Dual head_dim** (e.g. Gemma4 local=128, global=256) — mask shapes
58+
differ per layer type, increasing construction complexity
59+
60+
For simpler models (e.g. Mistral with only sliding window), bool mask
61+
is fine and uses less memory. Float bias is recommended when multiple
62+
constraints interact.
6363

6464
### Common misconception: bool masks and Flash Attention
6565

0 commit comments

Comments
 (0)