Skip to content

feat: add indexed attention training kernels - #8

Open
S1ro1 wants to merge 4 commits into
feat/mxfp8-moe-runtimefrom
feat/indexed-attention
Open

feat: add indexed attention training kernels#8
S1ro1 wants to merge 4 commits into
feat/mxfp8-moe-runtimefrom
feat/indexed-attention

Conversation

@S1ro1

@S1ro1 S1ro1 commented Aug 30, 2026

Copy link
Copy Markdown

Stacked on #9. This PR contains only the indexed-attention commits.

Adds a Python-only TileLang implementation for token-indexed grouped-query attention, including forward, backward, block scoring, and radix selection.

The package registers fake and autograd implementations, supports the sentinel and padding contract used by Qwen3.8 Flash Next, and caches successful kernel module loads so compiled model code can use the registry cleanly.

Sparse-index selection is standalone TileLang. The scorer reads Prime's contiguous packed training keys and computes BF16 tensor-core scores into a chunked workspace capped at 1 GiB. The radix selector finds the exact FP32 cutoff, resolves cutoff ties by lower block index, and writes global block indices directly. No compiled selector extension or Triton runtime dependency remains.

Validation:

  • BF16 indexed-attention forward and backward numerics against an FP32 PyTorch reference on H200
  • production index width and scattered-key smoke runs on H200
  • fullgraph torch.compile forward and backward
  • selective activation checkpointing forward and backward
  • two-rank context-parallel equivalence
  • Python-only wheel build with only the TileLang indexed-attention surface
  • scorer outputs matched the inference reference bit-for-bit, and selected sets matched exactly, for two 16K sequences, ragged 32K packing, smaller packed batches, and sub-block sequences
  • H200 production-shape selector benchmark: 5.598 ms for the complete TileLang selector; score computation was 2.852 ms
  • 4-layer EP8 forward profile with two 16K sequences, compile, selective activation checkpointing, and no LM head: 331 ms (Perfetto trace)

@S1ro1
S1ro1 changed the base branch from main to feat/mxfp8-moe-runtime August 30, 2026 19:45
@S1ro1
S1ro1 marked this pull request as ready for review August 30, 2026 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant