Skip to content
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
e4228dc
Change default from CUTLASS MLA to FlashInfer MLA
MatthewBonanni Jan 19, 2026
30cd686
Change log lines from debug to info
MatthewBonanni Jan 19, 2026
0b4c746
Merge remote-tracking branch 'upstream/main'
MatthewBonanni Jan 19, 2026
2cfba3b
First pass
MatthewBonanni Jan 19, 2026
5417186
Fix typo
MatthewBonanni Jan 19, 2026
c427ff8
Fix pre-commit
MatthewBonanni Jan 19, 2026
b80db7e
Merge branch 'main' into mla_prefill_abstraction
MatthewBonanni Jan 29, 2026
370d66d
Use device_config
MatthewBonanni Jan 29, 2026
5151bd9
Remove dead code
MatthewBonanni Jan 30, 2026
6922185
Bump deprecation version
MatthewBonanni Jan 30, 2026
f0a0acd
Update docs
MatthewBonanni Jan 30, 2026
3dc0fba
Cleanup
MatthewBonanni Jan 30, 2026
0225644
Format name
MatthewBonanni Jan 30, 2026
8013037
Fix dagger inside quotes
MatthewBonanni Jan 30, 2026
6688a67
Add selector test
MatthewBonanni Jan 30, 2026
a962bee
Update table
MatthewBonanni Jan 30, 2026
25d976f
Comment
MatthewBonanni Jan 30, 2026
f30cb13
Add model dtype support
MatthewBonanni Jan 30, 2026
eb27ec8
Add type annotation to fix docs build
MatthewBonanni Jan 30, 2026
17c9a9e
Merge branch 'main' into mla_prefill_abstraction
MatthewBonanni Feb 2, 2026
0f42a95
Fix rebase issue
MatthewBonanni Feb 2, 2026
97c9aa7
Introduce hashable config
MatthewBonanni Feb 2, 2026
fb7bead
Fix test
MatthewBonanni Feb 2, 2026
3b25d20
Pass device capability directly
MatthewBonanni Feb 2, 2026
0086b94
Fix hashing
MatthewBonanni Feb 2, 2026
5051737
Fix tests
MatthewBonanni Feb 3, 2026
115a75b
Merge branch 'main' into mla_prefill_abstraction
MatthewBonanni Mar 16, 2026
88d0be8
Fix pre-commit
MatthewBonanni Mar 16, 2026
0cd22de
Clean up FA import
MatthewBonanni Mar 16, 2026
7bf5ae8
Merge branch 'main' into mla_prefill_abstraction
MatthewBonanni Apr 2, 2026
d4c77ab
wip
LucasWilkinson Apr 7, 2026
2f0b2ad
cleanup
LucasWilkinson Apr 8, 2026
1d04d07
cleanup
LucasWilkinson Apr 9, 2026
9077fe2
Merge branch 'main' into mla_prefill_abstraction
MatthewBonanni Apr 28, 2026
26d38ac
Fixes
MatthewBonanni Apr 28, 2026
f1f47f9
Fix pre-commit
MatthewBonanni Apr 28, 2026
de1c30f
Cleanup
MatthewBonanni Apr 28, 2026
b4be44d
Merge branch 'main' into mla_prefill_abstraction
MatthewBonanni Apr 29, 2026
6ca7954
Fix test
MatthewBonanni Apr 29, 2026
a8edfd1
Fix old flag handling
MatthewBonanni Apr 29, 2026
d75923c
Clean up
MatthewBonanni Apr 29, 2026
d71ddcf
FA4 default
MatthewBonanni Apr 30, 2026
37399f7
Clean up
MatthewBonanni Apr 30, 2026
0433138
Remove cuDNN
MatthewBonanni Apr 30, 2026
2255459
Eliminate metadata classes, cleanup
MatthewBonanni Apr 30, 2026
eada601
Merge impl into backend
MatthewBonanni Apr 30, 2026
8489336
Update docs
MatthewBonanni Apr 30, 2026
21874a4
Clean up
MatthewBonanni Apr 30, 2026
b14ba7e
Clean up
MatthewBonanni Apr 30, 2026
83d9580
Update test
MatthewBonanni May 1, 2026
7bf83a2
Fix test
MatthewBonanni May 1, 2026
710581e
Put the prefill backend in the metadata
MatthewBonanni May 1, 2026
f42e555
Merge branch 'main' into mla_prefill_abstraction
mergify[bot] May 1, 2026
0a1feb2
Fix test
MatthewBonanni May 1, 2026
f73e58b
Clean up and fix test
MatthewBonanni May 1, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/design/attention_backends.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,10 +189,10 @@ configuration.

| Backend | Description | Compute Cap. | Enable | Disable | Notes |
|---------|-------------|--------------|--------|---------|-------|
| TRT-LLM Ragged‡ | TensorRT-LLM ragged attention | 10.x | Default on SM100 | `-ac.use_trtllm_ragged_deepseek_prefill=0` | DeepSeek R1 dims only |
| FlashInfer | FlashInfer CUTLASS backend | 10.x | `-ac.disable_flashinfer_prefill=0` | `-ac.disable_flashinfer_prefill=1` | DeepSeek R1 dims only |
| cuDNN | cuDNN-based attention | 10.x | `-ac.use_cudnn_prefill=1` | `-ac.use_cudnn_prefill=0` | |
| FlashAttention | FlashAttention varlen (FA2/FA3) | Any | Default fallback | Use other backends | FA3 on SM90, FA2 otherwise |
| `TRTLLM_RAGGED_PREFILL`‡ | TensorRT-LLM ragged attention | 10.x | `-ac.mla_prefill_backend=TRTLLM_RAGGED` | `-ac.mla_prefill_backend=FLASH_ATTN` | DeepSeek R1 dims only |
| `FLASHINFER_PREFILL` | FlashInfer CUTLASS backend | 10.x | `-ac.mla_prefill_backend=FLASHINFER` | `-ac.mla_prefill_backend=FLASH_ATTN` | DeepSeek R1 dims only |
| `CUDNN_PREFILL` | cuDNN-based attention | 10.x | `-ac.mla_prefill_backend=CUDNN` | `-ac.mla_prefill_backend=FLASH_ATTN` | DeepSeek R1 dims only |
| `FLASH_ATTN_PREFILL` | FlashAttention varlen (FA2/FA3) | Any | Default fallback | Use other backends | FA3 on SM90, FA2 otherwise |
Comment thread
MatthewBonanni marked this conversation as resolved.
Outdated

> **‡** TRT-LLM Ragged is the default on Blackwell (SM100).
> On other GPUs, FlashAttention is used as the default.
Expand Down
Loading