[MengqingCao]

MiniMax-M3.1-Flash P/D with Mooncake Store needs to preserve the complete M3 PAGE layout, including sparse index and draft regions. This PR adds that Store support and a TP4-prefill / TP4-decode recipe with accuracy, AgentX benchmark, timed P/D trace, and C32 chunked-transfer comparison cases.

The recipe adapts #2747 to eight MI355X GPUs and includes the runtime changes ported from #2686 and #2743–#2746. The branch is rebased onto main at ebd5f84f3, including the chunked P/D protocol from #2425. The AgentX launcher conflicts are resolved using the upstream PyPI installer while retaining this recipe's version 1.0.12 and CLI defaults.

Changes

• Support complete M3 PAGE packing/restoration in Mooncake Store, including index-scale and draft regions. Fingerprint the codec, NVFP4 layout and draft PTPC mode in Store keys.
• Include fixed-DSpark Mono support ([MiniMax-M3.1] Enable Mono decode with fixed DSpark verify #2686), draft PTPC FP8 requantization ([MiniMax-M3.1] DSpark draft follows --online_quant_config #2743), SGLang gfx950 NVFP4 kernels ([MiniMax-M3.1] FP4 index cache and NVFP4 sparse KV cache #2744), FP4 index cache ([MiniMax-M3.1] FP4 index-K cache: --index-cache-dtype fp4 (2/3) #2745), and NVFP4 sparse KV / fused verify top-k ([MiniMax-M3.1] NVFP4 sparse KV cache + fused verify top-k (3/3) #2746). Source revisions are recorded in the P/D recipe.
• Resolve DSpark checkpoints in target Hugging Face subfolders. Add model-specific image selection, served-model-name support and recipe AgentX CLI handling to CI.
• Require complete trace exports from every expected rank before stopping a timed trace workload.
• Add a C32 case enabling enable_chunked_transfer on the producer, with a matching unchanged C32 baseline. Add CPU-buffer coverage of incremental transfer for target NVFP4/scales, FP4 index/scales, draft FP8, cached-prefix skipping and final DSpark handoff.

Experiment configuration

Setting	Value
Model	amd/MiniMax-M3.1-Flash-Quark-MXFP4
Topology	One MI355X node: P TP4 on GPUs 0–3, D TP4 on GPUs 4–7
KV / index	Target NVFP4 KV + FP4 index using SGLang Gluon; draft FP8 KV
DSpark	K=7; synthetic AL 4.04 for performance; real acceptance for accuracy
Graphs / batch budget	Graphs on both roles; max-num-seqs 64/96/128/176 for C32/C48/C64/C88; 32768 batch tokens
Store / transfer	1 TiB Mooncake Store, two 512 GiB owners; Mooncake RDMA P→D
Workload	agentx-harness==1.0.12, semianalysis_cc_traces_weka_062126
Performance	C32/C48/C64/C88; 3600 seconds formal profiling after warmup, no torch trace
C32 comparison	Same head, node, image and recipe; only producer enable_chunked_transfer differs
Trace	C32; wait 1800 seconds after formal Profiling begins, capture P/D for 30 seconds, stop after all rank exports
Accuracy	GSM8K, full dataset, 5-shot, chat template
Image	rocm/atom-dev:nightly_202610261558, branch mounted; matching ATOMesh router built from source

The dense cached-prefill workspace feature remains reverted; this sparse-attention recipe does not depend on it. Traces are stored under each job's shared torch_traces/<worker>/rank_<tp>/ directory. P/D throughput/GPU divides by all eight allocated GPUs.

For chunked transfer, compare client/route-level TTFT and throughput. Decode queue/KV-wait timers start earlier with parallel P/D dispatch and can include prefill work. A prefill completing in one compute chunk offers limited compute/transfer overlap, and earlier destination reservation can increase decode KV pressure. The first comparison keeps the 32768-token budget fixed.

Validation

• Current head: 21224a248c282d8c19fc6c4ff9eeb596089cc744; base: ebd5f84f3a6af6a99d23f85e5c1107b238d1cdaf.
• 562 tests passed, 1 skipped, 134 subtests passed across chunked transfer, DCP transfer, MHA pools, Store and ATOMesh configuration/launcher tests.
• Both C32 cells passed expanded-configuration equality checks (excluding identity and the chunked switch) and Slurm dry-run. Bash syntax, Ruff, Black and whitespace checks passed.
• C32 baseline/chunked comparison submitted at the current head on pit2-p03-g12, 3600 seconds each. GPU results are pending.
• Before rebase, e73246e80 completed C32, C48 and C64. Total throughput was 231,882.38 / 310,065.13 / 377,440.25 tok/s. C48/C64 each cancelled three requests at drain; C64 also reached a 300-second cancellation-confirmation timeout before exporting successfully. These are individual successful jobs, not a claim that the entire sweep succeeded.
• The earlier 30-second P/D trace completed at cc113529a. It predates the workspace revert and this rebase. Synthetic-acceptance performance runs do not establish GSM8K accuracy.

[github-actions]

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

• ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

• ✅ Run after the PR is approved and Pre Checkin passes
• ✅ Run immediately when an approval review is submitted
• ✅ Can be requested before approval with labels

Label	Tests
ci:full	Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom	Run native ATOM model accuracy tests
ci:vllm	Run ATOM vLLM OOT model accuracy tests
ci:sglang	Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2750 --add-label <label>