[garrett361]

SFT now pads packed rows only to a multiple of cp, as RL does, instead of to seq_len. The fp8 indexer drops its autotuning, which re-benchmarked configs on every new row length and made unpadded rows slower than padded ones on DeepSeek-V4 (#3726).

• The indexer pads keys to a multiple of 16 and runs one fixed config, within 1.05x of the best.
• Every rank, including RL on DeepSeek-V4 and GLM-5, runs that config with bitwise-identical top-k.
• SFT and RL throughput count trained tokens, not seq_len × rows or the first row × count.
• The SFT seq_len % (2 * cp) check is gone: CP only needs seq_len % cp, which configs enforce.

Same samples per arm, time summed over steps 5 to 20, one H200 node:

Workload	Padding on main	main	This PR	Speedup
DeepSeek-V4-Flash 6L, scaleswe, 256K rows, cp 8	19.8%	61.2 s	38.2 s	1.60x
GLM-4.5-Air 8L, math, 131K rows, dp 4 cp 2	7.3%	223.8 s	200.1 s	1.12x
GLM-4.5-Air 8L, scaleswe, 131K rows, dp 4 cp 2	19.7%	701.0 s	623.0 s	1.13x

On DeepSeek-V4, the 1.60x exceeds the 1.25x that removing 19.8% of tokens predicts because two effects multiply: main's token-0 pads crowd onto the same experts, and the padding itself costs compute. The DeepSeek-V4 ablation section below separates the two.

On GLM with scaleswe, removing 19.7% of the tokens would predict about 1.25x, but it measured 1.13x. Its 4 data-parallel ranks now get rows of different lengths, and each step likely waits for the rank with the longest rows.

Loss matches main to within 1e-3 per step. This overlaps #3932 in CatDataset._finalize_pack. Harness: benchmarks/scripts/sft_pad_to_cp/ (removed before merge).

DeepSeek-V4 ablation

Each row changes one thing on the DeepSeek-V4 workload above, to separate the cost of the padding length, of token-0 pads crowding onto the same experts, and of indexer re-tuning:

Arm	Time	Speedup vs main
main: pads every row to seq_len with token 0	61.2 s	1.00x
main, but pads with the row's own tokens (#3932)	49.6 s	1.23x
pads only to a multiple of cp, indexer still autotuned	73.6 s	0.83x
pads only to a multiple of cp, fixed-config indexer (this PR)	38.2 s	1.60x

Alternatives considered

• Bucketing the autotune key on next_power_of_2(S_q) reached 1.47x on DeepSeek-V4. Each bucket kept its first row's config (up to 1.44x slower later), short rows paid extra tunes, and each rank still tuned on its own rows, so ranks could run different kernels.

🤖 Generated with Claude Code

[mikasenghaas]

great great gerat