[hnyls2002]

• With attention TP > 1 and a FlashMLA backend that pads the query to 64 heads (not trtllm, not flashmla_mega), DeepSeek-V4.1 caches the zeroed padded query buffer on the step's attention metadata (q_pad_buffer) and replaced it whenever the shape changed.
• A decoder tail graph step captures every row bucket on that one metadata, largest first. Each smaller bucket replaced the buffer, so the larger buckets' graphs kept the address of a freed one: their replays wrote the query into memory the allocator had reused, and after flush_cache (which empties the CUDA cache) they hit an MMU fault in fused_q_norm_rope.
• The metadata now keeps one buffer that only grows, and each call takes a row slice (MqaAttentionBase._zero_padded_q). Every bucket a tail step captures shares the largest one, which the graphs keep through the metadata they hold. The padding is still zeroed once and never rewritten. The graphs keep that one largest buffer alive (128 MiB at 2048 rows with TP8 heads padded to 64), where before each bucket's buffer was freed under it.
• The buffer must not grow after a CUDA graph has captured it, so a capture marks the metadata (q_pad_captured) and a later growth fails an assert instead of freeing a captured buffer. HIP's breakable-graph metadata keeps the mark across replays like the buffer.
• Paths without query padding are unaffected: DP attention (attention TP 1), trtllm, and flashmla_mega.
• Verification: DeepSeek-V4.1-Flash, TP4, DSpark, --dsv4-attn-backend flashmla (pads the query on this GPU too), the external-linker load-back KL cases, which flush the cache between requests.
  • Before: in the encoder SWA replay class (eager prefill + tail graphs), the second case fails with a CUDA error after a flush, in both of two runs.
  • After: both classes pass, the paged-SWA one (prefill graph, tail graphs, decode graphs) and the encoder SWA replay one; the new assert never fires.
  • The same failure shows on 8xH200 CI, where the default backend pads the query.

---

CI States

Latest PR Test (Base): 🚫 Run #39368034936
Latest PR Test (Extra): ❌ Run #39368034911
Latest PR Test (AMD ROCm 10): ⏳ Run #39368035086

[hnyls2002]

/rerun-test registered/attention/unittests/dsv4/test_deepseek_v4.py test_deepseek_v41_flash_gb300_encoder_bcg.py

[hnyls2002]

/tag-and-rerun-ci

[github-actions]

🚀 4-gpu-b200 (1 test): ✅ View workflow run

cd test/ && python3 registered/attention/unittests/dsv4/test_deepseek_v4.py

🚀 1-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/attention/unittests/dsv4/test_deepseek_v4.py

🚀 4-gpu-gb300 (1 test): ❌ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v41_flash_gb300_encoder_bcg.py