Skip to content

[None][test] Switch MiniMax-M3 Eagle3 accuracy test to the GQA draft head #17406

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zheyuf wants to merge 1 commit into from
Closed

[None][test] Switch MiniMax-M3 Eagle3 accuracy test to the GQA draft head #17406

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 14 additions & 8 deletions tests/integration/defs/accuracy/test_llm_api_pytorch.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7904,7 +7904,7 @@ def test_nvfp4_eagle3(self, tp_size, ep_size, attention_dp,
max_draft_len = 3
spec_config = Eagle3DecodingConfig(
max_draft_len=max_draft_len,
speculative_model=f"{llm_models_root()}/MiniMax-M3-EAGLE3",
speculative_model=f"{llm_models_root()}/MiniMax-M3-EAGLE3-GQA",
)
# The runtime forces tokens_per_block per implementation (128 MSA / 32
# reference).
Expand Down Expand Up @@ -7968,8 +7968,8 @@ def drain_spec_stats(llm):
task.evaluate(llm)

# Chat-format acceptance — the drafter's training distribution
# (Inferact/MiniMax-M3-EAGLE3 card: 0.839 / 3.518). Reuses the
# live engine and the cached dataset; ~20 s under CUDA graphs.
# (Inferact/MiniMax-M3-EAGLE3-GQA). Reuses the live engine and
# the cached dataset; ~20 s under CUDA graphs.
questions = [
r["question"]
for r in load_dataset("gsm8k", "main", split="test")
Expand All @@ -7990,12 +7990,18 @@ def drain_spec_stats(llm):
assert steps > 0, "no speculative iterations recorded"
chat_rate = accepted / drafted
chat_length = 1 + accepted / steps
# Published reference (Inferact/MiniMax-M3-EAGLE3 drafter card):
# rate 0.839, length 3.518. Our testing thresholds: rate > 0.78,
# length > 3.3.
# The GQA drafter card (Inferact/MiniMax-M3-EAGLE3-GQA) publishes
# MT-Bench 2.668 / 55.62% and SPEED-Bench 2.561 / 52.04% (vLLM,
# MXFP8 target, draft_len=3) but no chat-GSM8K figures. The head
# is retrained on the same data as the MHA head with only the
# attention changed (64 -> 4 KV heads), so the MHA-derived
# references (its card: rate 0.839, length 3.518) and thresholds
# (rate > 0.78, length > 3.3) are carried over pending
# calibration on the first GQA CI run.
ref_rate, ref_length = 0.839, 3.518
ref_source = "the Inferact/MiniMax-M3-EAGLE3 drafter model card"
print(f"MiniMax-M3 Eagle3 chat-GSM8K acceptance: rate="
ref_source = ("the Inferact/MiniMax-M3-EAGLE3 (MHA) drafter "
"card; provisional for the GQA head")
print(f"MiniMax-M3 Eagle3-GQA chat-GSM8K acceptance: rate="
f"{chat_rate:.3f}, mean acceptance length="
f"{chat_length:.3f} ({steps} spec iterations)")
assert chat_rate > 0.78, \
Expand Down
Loading