Skip to content

[None][test] Switch MiniMax-M3 Eagle3 accuracy test to the GQA draft head - #17406

Closed
zheyuf wants to merge 1 commit into
NVIDIA:feat/m3_with_msafrom
zheyuf:feat/m3-eagle3-gqa-head
Closed

[None][test] Switch MiniMax-M3 Eagle3 accuracy test to the GQA draft head#17406
zheyuf wants to merge 1 commit into
NVIDIA:feat/m3_with_msafrom
zheyuf:feat/m3-eagle3-gqa-head

Conversation

@zheyuf

@zheyuf zheyuf commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Description

Switches TestMiniMaxM3::test_nvfp4_eagle3 from the MHA EAGLE3 draft head
(Inferact/MiniMax-M3-EAGLE3) to the GQA retrain
(Inferact/MiniMax-M3-EAGLE3-GQA).

Per the model card the GQA head is trained on the same data (kimi-mtp, OpenCodeInstruct, SWE-bench,
SWE-bench-Pro); the only architectural change is the draft attention, num_key_value_heads 64 → 4.

Why

The EAGLE3 draft is a single dense attention layer, so its KV cache is disproportionately expensive
next to MiniMax-M3's sparse-attention target. From the published configs (head_dim 128, BF16 KV),
draft KV per token per rank at TP4:

draft KV per token/rank
MHA (num_key_value_heads 64) 64/4 × 128 × 2 × 2 B = 8192 B
GQA (num_key_value_heads 4) 4/4 × 128 × 2 × 2 B = 512 B

A 16× smaller draft cache leaves substantially more of the KV pool for the target, and in
disaggregated serving it proportionally shrinks the draft share of the KV transfer.

Acceptance is comparable: the GQA card reports MT-Bench 2.668 / 55.62% and SPEED-Bench (qualitative)
2.561 / 52.04% mean accepted length / accept rate at num_speculative_tokens=3.

Why no runtime change is needed

The draft KV cache manager and the drafter's attention both read num_key_value_heads from the
drafter's own config, not the target's:

  • SpecDecOneEngineForCausalLM builds the draft config via
    ModelConfig.from_pretrained(spec_config.speculative_model) (modeling_speculative.py)
  • Eagle3Attention passes config.num_key_value_heads straight into Attention
    (modeling_speculative.py)
  • KvCacheCreator._get_effective_draft_config() returns that draft config, and the V2 KV cache
    manager derives per-rank KV heads from it (_util.py, kv_cache_manager_v2.py)

This is the same path the existing GQA Llama EAGLE3 heads already exercise (e.g.
EAGLE3-LLaMA3.1-Instruct-8B is 32 Q / 8 KV heads) — the MiniMax-M3 MHA head was the unusual case,
not this one.

Test Coverage

accuracy/test_llm_api_pytorch.py::TestMiniMaxM3::test_nvfp4_eagle3 — both l0_dgx_b200_m3 variants
(attention_dp False/True). Test ids are unchanged, so no test-list edits are required.

Verification done

  • Swept the branch (tests/, docs/, examples/, jenkins/): this test is the only reference to
    the MHA head, on both the original and current feat/m3_with_msa.
  • The MiniMax-M3 deployment guide does not pin an EAGLE3 head, so no doc change is needed.
  • Confirmed the two checkpoints' configs are identical apart from num_key_value_heads, and that the
    GQA weights match it (k_proj is [512, 12288] = 4 × 128, all BF16).
  • Smoke-tested a 4-GPU TP4 deployment: the GQA head loads and completes engine warmup with no code
    changes and no errors.

Acceptance thresholds

The GQA card publishes MT-Bench / SPEED-Bench but no chat-GSM8K figures, so the MHA-derived
references (rate 0.839, length 3.518) and thresholds (rate > 0.78, length > 3.3) are carried over and
labelled provisional in the test comments. They should be recalibrated from the first green CI run on
this head.

Dependencies

  • MiniMax-M3-EAGLE3-GQA added to the CI model store (ftp/llm-models!583, merged) — allow the
    usual sync delay to the CI scratch before running the test

…head

The AgentX MiniMax-M3 submission moves its EAGLE3 draft head from the
MHA checkpoint (Inferact/MiniMax-M3-EAGLE3) to the GQA retrain
(Inferact/MiniMax-M3-EAGLE3-GQA): same training data, attention changed
from MHA to GQA (num_key_value_heads 64 -> 4), giving a 16x smaller
draft KV cache and a lighter draft share in the disagg KV transfer.

test_nvfp4_eagle3 now loads MiniMax-M3-EAGLE3-GQA from LLM_MODELS_ROOT.
No runtime change is needed: the draft KV cache manager reads
num_key_value_heads from the drafter config, the same path the GQA
Llama EAGLE3 heads use.

The GQA card publishes no chat-GSM8K figures, so the MHA-derived
acceptance thresholds (rate > 0.78, length > 3.3) are carried over and
marked provisional pending calibration on the first CI run.

Signed-off-by: Zheyu Fu <zheyuf@nvidia.com>
zheyuf added a commit to zheyuf/TensorRT-LLM that referenced this pull request Aug 11, 2026
…t disagg acceptance

Supersedes NVIDIA#17406. The drafter becomes Inferact/MiniMax-M3-EAGLE3-GQA
in both arms, with thresholds recalibrated from the first GQA run:
disagg chat-GSM8K measures length 2.773 / rate 0.591 (consistent with
the GQA card's MT-Bench 2.668; the MHA head measured 3.44-3.48 here),
so the MHA-derived floors do not carry over. The disagg arm now also
asserts acceptance through the generation worker's /metrics iteration
stats (enable_iter_perf_stats; the fixture's DuckLLM gains a
router_url field to locate the worker via /cluster_info) — closing
the gap where a drafter-KV transfer regression was invisible to the
accuracy assert.

Signed-off-by: Zheyu Fu <zheyuf@nvidia.com>
@zheyuf zheyuf closed this Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant