@@ -24,8 +24,8 @@ Use this skill when:
2424| ----------| ------------| -----|
2525| Causal only | ` attn_mask=None ` + ` is_causal=1 ` | Enables Flash (fastest for prefill) |
2626| Padding (batch>1) | ` nonpad_kv_seqlens ` (best) or bool mask | ` nonpad_kv_seqlens ` enables Flash + shared buffer with no mask |
27- | Sliding window | Float additive bias | Precise window control |
28- | Complex (causal+ sliding+padding ) | Float additive bias | Most flexible, avoids construction bugs |
27+ | Sliding window (simple) | Bool mask | Equally precise as float, uses less memory |
28+ | Complex (sliding+KV-shared+dual head_dim ) | Float additive bias | Avoids mask construction bugs in multi-constraint patterns |
2929| Custom pattern | Float additive bias | Arbitrary values |
3030
3131## Bool mask vs float additive bias
@@ -35,31 +35,31 @@ Use this skill when:
3535| Pattern | Recommended mask type |
3636| ---------| ----------------------|
3737| Simple causal-only | No mask — use ` is_causal=1 ` (enables Flash) |
38- | Sliding window | Float additive bias |
38+ | Sliding window (simple model) | Bool mask (precise, less memory) |
3939| KV-shared layers | Float additive bias |
4040| Mixed head_dim (e.g. Gemma4) | Float additive bias |
41- | Padding + causal | Float additive bias |
41+ | Padding + causal | ` nonpad_kv_seqlens ` or bool mask |
42+ | Multiple constraints combined | Float additive bias |
4243
4344### Why float additive bias is safer for complex patterns
4445
45- ONNX ` Attention ` supports bool mask (` True ` =attend, ` False ` =ignore).
46- ORT correctly converts bool→float internally via
47- ` ConvertAttnMaskToBias() ` . However, constructing correct bool masks
48- for complex patterns is error-prone:
49-
50- - ** Sliding window boundaries** must align with KV cache positions —
51- off-by-one errors silently produce wrong attention patterns
52- - ** KV-shared layers** borrow K/V from other layers — the mask shape
53- must match the borrowed KV dimensions, not the current layer's
54- - ** ` is_causal=1 ` + bool mask** double-applies constraints — the
55- ` is_causal ` flag adds its own causal mask on top of the explicit one
56- - ** Dual head_dim** (e.g. Gemma4 local=128, global=256) means mask
57- shapes differ per layer type
58-
59- Float additive bias gives explicit control:
60- - ` 0.0 ` for "attend" positions
61- - ` -inf ` (or ` -10000.0 ` ) for "ignore" positions
62- - No ambiguity in kernel interpretation
46+ Bool mask and float additive bias are ** equally precise** — both can
47+ represent any attention pattern. ORT converts bool→float internally
48+ via ` ConvertAttnMaskToBias() ` , so they have identical kernel dispatch.
49+
50+ The reason we use float bias for complex models is ** bug avoidance** ,
51+ not a fundamental limitation of bool masks. Constructing correct bool
52+ masks for multi-constraint patterns is error-prone:
53+
54+ - ** Sliding window + KV-shared** — mask shape must match borrowed KV
55+ dimensions, not the current layer's. Off-by-one errors are silent.
56+ - ** ` is_causal=1 ` + bool mask** — double-applies causal constraints
57+ - ** Dual head_dim** (e.g. Gemma4 local=128, global=256) — mask shapes
58+ differ per layer type, increasing construction complexity
59+
60+ For simpler models (e.g. Mistral with only sliding window), bool mask
61+ is fine and uses less memory. Float bias is recommended when multiple
62+ constraints interact.
6363
6464### Common misconception: bool masks and Flash Attention
6565
0 commit comments