Add EAGLE-3 Speculative Decoding Support for Qwen3 MoE

[rahul-tuli]

This PR adds support for EAGLE-3 speculative decoding to the Qwen3MoeForCausalLM model, enabling faster inference with draft models like nm-testing/Mockup-qwen235-eagle3-fp16.

Changes

Modified Files

• vllm/model_executor/models/qwen3_moe.py

Implementation Details

1. Added SupportsEagle3 Interface

   • Imported and added SupportsEagle3 to Qwen3MoeForCausalLM class inheritance
   • Implements required methods: set_aux_hidden_state_layers() and get_eagle3_aux_hidden_state_layers()

2. Updated Qwen3MoeModel

   • Added aux_hidden_state_layers attribute to track layers that output auxiliary hidden states
   • Modified forward() method to collect auxiliary hidden states at specified layers
   • Returns tuple of (hidden_states, aux_hidden_states) when auxiliary states are collected

3. Updated Qwen3MoeForCausalLM

   • Implements get_eagle3_aux_hidden_state_layers() to return auxiliary layer indices (2, mid-layer, and n-3)
   • Implements set_aux_hidden_state_layers() to configure which layers output auxiliary states

Testing

Tested with Qwen3-235B-A22B MoE model and EAGLE-3 drafter:

from vllm import LLM, SamplingParams

# Initialize with EAGLE-3 speculative decoding
llm = LLM(
    model="Qwen/Qwen3-235B-A22B-Instruct-2507-FP8",
    tensor_parallel_size=4,
    speculative_config={
        "model": "nm-testing/Mockup-qwen235-eagle3-fp16",
        "method": "eagle3",
        "num_speculative_tokens": 3,
    },
    max_model_len=16384,
)

# Generate with speculative decoding
prompts = [
    "Hello, my name is",
    "The capital of France is",
]

sampling_params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=100)
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(f"Prompt: {output.prompt}")
    print(f"Generated: {output.outputs[0].text}")

Related

This implementation follows the same pattern as existing EAGLE-3 support in:

• Qwen2ForCausalLM
• Qwen3ForCausalLM
• LlamaForCausalLM

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

[benchislett]

LGTM

[zeroorhero]

@rahul-tuli Hi, did the Speculative Decoding actually make Qwen3 MoE faster? What kind of acceptance rate did you see on average?

[gyou2021]

Is nm-testing/Mockup-qwen235-eagle3-fp16 nm-testing/Mockup-qwen235-eagle3-fp16-speculators-converted? No nm-testing/Mockup-qwen235-eagle3-fp16 in https://huggingface.co/nm-testing currently.

[eldarkurtic]

As the name suggests, nm-testing models are temp models for testing purposes. We will soon release an official qwen235 speculator under RedHatAI which community should use.