[None][test] Switch the MiniMax-M3 Eagle3 CI to the GQA draft head - #17514
Closed
zheyuf wants to merge 1 commit into
Closed
[None][test] Switch the MiniMax-M3 Eagle3 CI to the GQA draft head#17514zheyuf wants to merge 1 commit into
zheyuf wants to merge 1 commit into
Conversation
Both arms now run Inferact/MiniMax-M3-EAGLE3-GQA, the head production
is moving to: same training data as the MHA head with the drafter's
attention changed from 64 to 4 KV heads, so its draft KV cache is 16x
smaller and matches the target's head count.
Measured on this branch (4-GPU node, InferenceMAX eval + the shared
chat-GSM8K acceptance probe):
aggregated (separate draft manager) 3.515 / 0.838, accuracy 95.45
disaggregated (unified sub-page view) 3.369 / 0.790, accuracy 95.15
3.395 / 0.798, accuracy 95.45
The aggregated figure is indistinguishable from the MHA card's GSM8K
reference (3.518 / 0.839), which is why that reference stands in for
the GQA card's missing GSM8K entry; the two cards agree on the
benchmark they do share (MT-Bench 2.698 vs 2.668). The disaggregated
arm measures ~0.14 lower, so the rate floor drops 0.78 -> 0.76 for
headroom while the length floor stays at 3.3 — still between the
measured healthy range and the 3.33 plateau a drafter-KV transfer
regression produced on this workload.
Signed-off-by: Zheyu Fu <zheyuf@nvidia.com>
zheyuf
force-pushed
the
feat/m3-eagle3-gqa-ci
branch
from
August 12, 2026 05:54
7b1bd18 to
af6ddb1
Compare
This was referenced Aug 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Switches the MiniMax-M3 Eagle3 CI to the GQA draft head
(
Inferact/MiniMax-M3-EAGLE3-GQA): the same drafter retrained on the samedata with its attention changed from 64 to 4 KV heads, so the draft KV cache
is 16x smaller and matches the target's head count. Both CI arms switch;
nothing else changes. One commit, one file.
#17457 has landed, so this is no longer stacked — it rebases onto the merged
feat/m3_with_msa. That PR contributes the disaggregated CI arm, the sharedacceptance probe, and the equal-heads fix in
_build_per_layer_num_kv_headsthat this head requires: with 4 KV heads the drafter has the target's head
count, and the old equality shortcut returned a scalar per-layer list, so the
appended draft layers went unnoticed by the shared manager and the drafter was
routed through the target's attention machinery (it died with "Trtllm-gen
kernels not found").
Measurements
One 4-GPU node, InferenceMAX accuracy eval plus the shared chat-GSM8K
acceptance probe (200 questions, chat template, greedy, 512 tokens).
Reference points: the MHA card publishes GSM8K 3.518 / 0.839 and the MHA head
measured 3.524 / 0.841 on the disaggregated arm of #17457. The GQA card
publishes no GSM8K figure, but the two cards agree on the benchmark they share
(MT-Bench 2.698 vs 2.668), and the aggregated measurement above is
indistinguishable from the MHA reference — so the MHA card's GSM8K entry
stands in as the reference for both heads.
Thresholds. The aggregated arm keeps its margins (3.515 vs the 3.3 floor).
The disaggregated arm measures ~0.14 lower for this head across two runs, so
the rate floor moves 0.78 -> 0.76 for headroom; the length floor stays at 3.3,
which still sits between the measured healthy range and the 3.33 plateau that a
drafter-KV transfer regression produced on this workload for the MHA head.
Head-to-head outside CI (context for reviewers)
An earlier revision of this description reported the GQA head ~11% below the
MHA head on MT-Bench against the production serving configs and called it a
head property. That was wrong and is retracted. Follow-up single-variable
experiments:
In clean configs the two heads are equal, and both match their cards
(same harness, TP4/EP4, MSA, draft_len 3, greedy, MT-Bench 80 conversations x
2 turns; only
speculative_modelchanges):The production-config gap came from a CUDA-graph defect that is specific to
this head geometry, not from the head's quality: in disaggregated serving the
GQA drafter's acceptance collapses at small generation batch when the captured
graph batch sizes include 1 and 2 (acceptance 2.00 at batch 1, 2.21 at batch 2,
versus 3.48 / 3.57 for MHA; flat for MHA at every size). Restricting the
generation server's
cuda_graph_config.batch_sizesto >= 4 with paddingremoves it completely at no throughput cost (round-1 acceptance 1.91 -> 3.36,
steady-state 182 vs 180 tok/s), and puts the GQA head within ~3% of MHA — the
same margin the two model cards show. Accuracy is unaffected either way
(gsm8k_inferencemax 95.07-95.45), so it costs speculative speedup only.
The CI configuration here does not pin
batch_sizes, so its disaggregatednumbers above are measured with the default list (which includes 1 and 2);
they are the honest floor for that configuration. The defect itself is a
runtime/kernel issue outside this PR and is being reported separately.
PR Checklist
[JIRA/NVBUG/None][type]formatfeat/m3_with_msa(single commit)/bot run)