Skip to content

[Spec] Consolidate the verify step into eagle_worker_common.run_eagle_verify - #31380

Merged
hnyls2002 merged 2 commits into
mainfrom
lsyin/spec-run-verify
Jul 16, 2026
Merged

hnyls2002 merged 2 commits into
mainfrom
lsyin/spec-run-verify

Conversation

@hnyls2002

@hnyls2002 hnyls2002 commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Consolidate the verify step of EAGLEWorkerV2 and MultiLayerEagleWorkerV2 into one eagle_worker_common.run_eagle_verify function. The single-layer eagle verify() (the superset) is the source of truth; both workers' verify() become thin argument-binding wrappers, and multi-layer's stale partial copy is deleted. The two compaction helpers (_finalize_accept_tree_path, _compact_accept_to_front) move along as module-private functions.

The two real differences (preserved verbatim behind switches)

Switch EAGLE Multi-layer
metadata_ready_pre_pad False — relies on eagle_prepare_for_verify marking metadata ready only when the cuda-graph load_batch path ran; the non-graph path re-inits post-pad True — marks ready pre-pad unconditionally (its pre-existing behavior, kept verbatim)
finalize_tree_path True — compacts the accepted tree path to the front of each per-req block for topk > 1 False — has never run this compaction (kept verbatim; whether topk > 1 multi-layer should adopt it is left as a follow-up)

What multi-layer gains (each a gated no-op for its configurations)

Feature No-op evidence
Grammar bitmask (generate_token_bitmask + vocab_mask) Entire block behind batch.has_grammar; eagle_sample's vocab_mask defaults to None, so the old 3-arg call was already equivalent
Mamba state commit commit_mamba_states_after_verify early-returns unless the model is mambaish and the backend has the commit hook
c128 draft-state clear getattr(..., "clear_unaccepted_c128_draft_states", None) is None for other KV caches
NPU mrope position recompute Gated on _is_npu and model_config.model_is_mrope

Verification

Verification script: https://gist.github.com/hnyls2002/ee26d74ec93ddcd45aa4c485a7c0b996 — runs against the pinned base/head commits:

  1. Relocation proof: the committed run_eagle_verify is AST-equivalent to the base commit's eagle verify body plus exactly three declared edits — the finalize_tree_path gate prefix, the gated pre-pad metadata block, and the deletion of the num_tokens_per_req re-stamp (mirroring [Spec] Converge DP-attention spec width scaling onto num_tokens_per_req #31244, which removed it from the base copy after this branch was cut). No other line differs beyond the declared parameter renames.
  2. Helper relocation proof: both compaction helpers match their declared moves.

Cross-checks: no test patches any of the moved names; _finalize_accept_tree_path / _compact_accept_to_front have no external callers; StandaloneWorkerV2 / FrozenKVMTPWorkerV2 inherit the eagle wrapper (same switch values as the body they inherited before). compile + ruff clean across speculative/; black reflow AST-equivalence checked.

GPU accuracy validation (eagle + MTP configs) to follow before merge, in addition to the CI reruns.


CI States

Latest PR Test (Base): 🚫 Run #29472859914
Latest PR Test (Extra): ❌ Run #29472859780

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@hnyls2002

Copy link
Copy Markdown
Collaborator Author

/rerun-test registered/spec/eagle/test_spec_eagle.py registered/spec/eagle/test_spec_eagle_topk_page.py registered/spec/eagle/test_eagle_constrained_decoding.py registered/spec/eagle/test_eagle_reject_sampling.py registered/spec/test_gemma4_mtp_31b_extra.py registered/spec/test_spec_standalone.py registered/spec/test_frozen_kv_mtp.py registered/unit/spec/test_eagle_worker_v2_topk1_fastpath.py

@github-actions

github-actions Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Results for /rerun-test registered/spec/eagle/test_spec_eagle.py registered/spec/eagle/test_spec_eagle_topk_page.py registered/spec/eagle/test_eagle_constrained_decoding.py registered/spec/eagle/test_eagle_reject_sampling.py registered/spec/test_gemma4_mtp_31b_extra.py registered/spec/test_spec_standalone.py registered/spec/test_frozen_kv_mtp.py registered/unit/spec/test_eagle_worker_v2_topk1_fastpath.py:

🚀 1-gpu-5090 (3 tests): ✅ View workflow run

cd test/ && python3 registered/spec/eagle/test_spec_eagle.py
cd test/ && python3 registered/spec/eagle/test_spec_eagle_topk_page.py
cd test/ && python3 registered/unit/spec/test_eagle_worker_v2_topk1_fastpath.py

🚀 1-gpu-h100 (3 tests): ✅ View workflow run

cd test/ && python3 registered/spec/eagle/test_eagle_constrained_decoding.py
cd test/ && python3 registered/spec/test_spec_standalone.py
cd test/ && python3 registered/spec/test_frozen_kv_mtp.py

🚀 2-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/spec/eagle/test_eagle_reject_sampling.py
cd test/ && python3 registered/spec/test_gemma4_mtp_31b_extra.py

# Conflicts:
#	python/sglang/srt/speculative/eagle_worker_v2.py
@hnyls2002

Copy link
Copy Markdown
Collaborator Author

/tag-and-rerun-ci

@hnyls2002

Copy link
Copy Markdown
Collaborator Author

/rerun-test registered/models_e2e/test_mimo_v2.py registered/models_e2e/test_deepseek_v3_mtp.py registered/models_e2e/test_dsa_glm52_tp_mtp.py registered/models_e2e/test_qwen3_next_models_mtp.py registered/spec/eagle/test_eagle_dp_attention.py registered/spec/eagle/test_adaptive_speculative.py

@github-actions

github-actions Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Results for /rerun-test registered/models_e2e/test_mimo_v2.py registered/models_e2e/test_deepseek_v3_mtp.py registered/models_e2e/test_dsa_glm52_tp_mtp.py registered/models_e2e/test_qwen3_next_models_mtp.py registered/spec/eagle/test_eagle_dp_attention.py registered/spec/eagle/test_adaptive_speculative.py:

🚀 8-gpu-h200 (3 tests): ✅ View workflow run

cd test/ && python3 registered/models_e2e/test_mimo_v2.py
cd test/ && python3 registered/models_e2e/test_deepseek_v3_mtp.py
cd test/ && python3 registered/models_e2e/test_dsa_glm52_tp_mtp.py

🚀 4-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/models_e2e/test_qwen3_next_models_mtp.py
cd test/ && python3 registered/spec/eagle/test_eagle_dp_attention.py

🚀 1-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/spec/eagle/test_adaptive_speculative.py

@hnyls2002
hnyls2002 merged commit b55228c into main Jul 16, 2026
215 of 257 checks passed
@hnyls2002
hnyls2002 deleted the lsyin/spec-run-verify branch July 16, 2026 07:56
Chronostasys pushed a commit to MindLab-Research/sglang that referenced this pull request Aug 24, 2026
jakki-amd pushed a commit to jakki-amd/sglang that referenced this pull request Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant