[valarLip]

Why

#2452 found two MiniMax-M3 mono issues. The first, stores not drained before their done flags, is fixed on main (every flag goes through the framework's publish, s_waitcnt vmcnt(0) first). The second, a CUDA-graph pad row changing real rows' MoE results, is still open. #2452 keeps a pad row out of the expert set by its seq_len.

Testing that fix showed the issue is wider than pad rows. With the same probe rows and only the step's other rows changed, the probes' MoE output (ar_out) differs when the others are other requests, pad rows, or long-context requests. On main this happens at 11 and 16 rows. The cause is the wide down stage. It deals each expert's 6 k-chunks to the row group's 4 waves by the expert's rank in the step's expert union (j = wave + 4 i), and the 4 waves' partials are summed apart. Another row bringing in an expert shifts the ranks, which regroups this row's sums. In serving this shows up as answers that change with the batch: two GSM8K runs of the same server agree on 26% of the answers.

What

• Down, batch-invariant: wave q takes chunks q and q + 4 of every expert, and unit i is expert i // 2 for all 4 waves (lockstep, so they read one expert at a time). Which wave adds a chunk, and in what order, follows from the expert alone.
• Pad rows out of the union: a pad row is identified by batch_id_per_q_token, not by its seq_len.
  • The M3 decode publishes it through the shared publish_batch_ids, with -1 on the graph's pad tail; the capture metadata points at the same buffer.
  • K4 takes it, and row_live(batch_ids, k) (atom/mono/device/ops.py) lets a pad row set no bit in the expert -> token mask, so its experts are neither read nor computed.
  • The dispatch takes a step only with the ids published.

Tried, and slower

Variant	us a layer
Per-wave unit lists balanced by expert-id parity (1.5 units an expert a wave)	+4 / +6 (runtime expert indices)
The same chunk assignment with the waves running ahead independently	+5 at 8 rows (waves on different experts)
Branching past empty units	no gain

Test

4 x MI355X, TP4.

	main	this
Probe rows, other rows changed, one sparse layer (fused K1 + K4), widths 1-16	differs at 11, 16	bit-identical at every width
Graph replay, 16 rows: 9 real + 7 pads	118.1 us	110.5 us
Graph replay, all rows real, 5 / 8 / 12 / 16 rows	64.8 / 83.2 / 100.7 / 123.4	+2.4 / +1.4 / 0 / +1.3 us (the busiest wave does 2 units an expert, not 1.5)
GSM8K chat 8-shot, max-num-seqs 16, 12 concurrent, two runs	0.9393 / 0.9371	0.9386 / 0.9416
Answers identical between the two runs	25.9%	79.6%

The remaining ~20% that differ between runs come from the original-path parts (prefill, the 3 dense layers), which are not batch-invariant.

The probe test (batch_invariance.py) and its V4.1 counterpart live in the run-m3-mono / run-v41-mono skills. V4.1's K2b is batch-invariant by construction: a pick's down runs its whole K on one wave, and down_combine sums each token's top-k in its own order.

tests/test_minimax_m3_mono_dispatch.py adds three refusals: no batch ids, int64 ids, too few ids. All three fail with the dispatch check reverted.

[github-actions]

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

• ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

• ✅ Run after the PR is approved and Pre Checkin passes
• ✅ Run immediately when an approval review is submitted
• ✅ Can be requested before approval with labels

Label	Tests
ci:full	Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom	Run native ATOM model accuracy tests
ci:vllm	Run ATOM vLLM OOT model accuracy tests
ci:sglang	Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2485 --add-label <label>