Skip to content

[None][test] Switch the MiniMax-M3 Eagle3 CI to the GQA draft head - #17514

Closed
zheyuf wants to merge 1 commit into
NVIDIA:feat/m3_with_msafrom
zheyuf:feat/m3-eagle3-gqa-ci
Closed

[None][test] Switch the MiniMax-M3 Eagle3 CI to the GQA draft head#17514
zheyuf wants to merge 1 commit into
NVIDIA:feat/m3_with_msafrom
zheyuf:feat/m3-eagle3-gqa-ci

Conversation

@zheyuf

@zheyuf zheyuf commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Description

Switches the MiniMax-M3 Eagle3 CI to the GQA draft head
(Inferact/MiniMax-M3-EAGLE3-GQA): the same drafter retrained on the same
data with its attention changed from 64 to 4 KV heads, so the draft KV cache
is 16x smaller and matches the target's head count. Both CI arms switch;
nothing else changes. One commit, one file.

#17457 has landed, so this is no longer stacked — it rebases onto the merged
feat/m3_with_msa. That PR contributes the disaggregated CI arm, the shared
acceptance probe, and the equal-heads fix in _build_per_layer_num_kv_heads
that this head requires: with 4 KV heads the drafter has the target's head
count, and the old equality shortcut returned a scalar per-layer list, so the
appended draft layers went unnoticed by the shared manager and the drafter was
routed through the target's attention machinery (it died with "Trtllm-gen
kernels not found").

Measurements

One 4-GPU node, InferenceMAX accuracy eval plus the shared chat-GSM8K
acceptance probe (200 questions, chat template, greedy, 512 tokens).

arm draft KV path acceptance rate accuracy
aggregated (ADP off) separate manager 3.515 0.838 95.45
disaggregated (ADP on) unified sub-page view 3.369 0.790 95.15
disaggregated (repeat) unified sub-page view 3.395 0.798 95.45

Reference points: the MHA card publishes GSM8K 3.518 / 0.839 and the MHA head
measured 3.524 / 0.841 on the disaggregated arm of #17457. The GQA card
publishes no GSM8K figure, but the two cards agree on the benchmark they share
(MT-Bench 2.698 vs 2.668), and the aggregated measurement above is
indistinguishable from the MHA reference — so the MHA card's GSM8K entry
stands in as the reference for both heads.

Thresholds. The aggregated arm keeps its margins (3.515 vs the 3.3 floor).
The disaggregated arm measures ~0.14 lower for this head across two runs, so
the rate floor moves 0.78 -> 0.76 for headroom; the length floor stays at 3.3,
which still sits between the measured healthy range and the 3.33 plateau that a
drafter-KV transfer regression produced on this workload for the MHA head.

Head-to-head outside CI (context for reviewers)

An earlier revision of this description reported the GQA head ~11% below the
MHA head on MT-Bench against the production serving configs and called it a
head property. That was wrong and is retracted. Follow-up single-variable
experiments:

In clean configs the two heads are equal, and both match their cards
(same harness, TP4/EP4, MSA, draft_len 3, greedy, MT-Bench 80 conversations x
2 turns; only speculative_model changes):

target MHA GQA card
NVFP4 2.658 2.665 MHA 2.698 (n=80)
MXFP8 (what the cards used) 2.692 2.680 GQA 2.668 (n=64)

The production-config gap came from a CUDA-graph defect that is specific to
this head geometry
, not from the head's quality: in disaggregated serving the
GQA drafter's acceptance collapses at small generation batch when the captured
graph batch sizes include 1 and 2 (acceptance 2.00 at batch 1, 2.21 at batch 2,
versus 3.48 / 3.57 for MHA; flat for MHA at every size). Restricting the
generation server's cuda_graph_config.batch_sizes to >= 4 with padding
removes it completely at no throughput cost (round-1 acceptance 1.91 -> 3.36,
steady-state 182 vs 180 tok/s), and puts the GQA head within ~3% of MHA — the
same margin the two model cards show. Accuracy is unaffected either way
(gsm8k_inferencemax 95.07-95.45), so it costs speculative speedup only.

The CI configuration here does not pin batch_sizes, so its disaggregated
numbers above are measured with the default list (which includes 1 and 2);
they are the honest floor for that configuration. The defect itself is a
runtime/kernel issue outside this PR and is being reported separately.

PR Checklist

  • PR title follows the [JIRA/NVBUG/None][type] format
  • Commits are signed off (DCO)
  • Rebased onto merged feat/m3_with_msa (single commit)
  • CI (/bot run)

Both arms now run Inferact/MiniMax-M3-EAGLE3-GQA, the head production
is moving to: same training data as the MHA head with the drafter's
attention changed from 64 to 4 KV heads, so its draft KV cache is 16x
smaller and matches the target's head count.

Measured on this branch (4-GPU node, InferenceMAX eval + the shared
chat-GSM8K acceptance probe):

  aggregated  (separate draft manager)  3.515 / 0.838, accuracy 95.45
  disaggregated (unified sub-page view) 3.369 / 0.790, accuracy 95.15
                                        3.395 / 0.798, accuracy 95.45

The aggregated figure is indistinguishable from the MHA card's GSM8K
reference (3.518 / 0.839), which is why that reference stands in for
the GQA card's missing GSM8K entry; the two cards agree on the
benchmark they do share (MT-Bench 2.698 vs 2.668). The disaggregated
arm measures ~0.14 lower, so the rate floor drops 0.78 -> 0.76 for
headroom while the length floor stays at 3.3 — still between the
measured healthy range and the 3.33 plateau a drafter-KV transfer
regression produced on this workload.

Signed-off-by: Zheyu Fu <zheyuf@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant