Your current environment
Versions
- vLLM:
0.22.1rc1.dev259+g303916e93 (nightly wheel) — also reproduced on the vendor mistralllm/vllm-ms4 image
- transformers:
5.10.0.dev0 (main)
- mistral_common:
1.11.3
- GPU: NVIDIA RTX PRO 6000 Blackwell (sm_120), single GPU, TP=1
🐛 Describe the bug
Serving Mistral Small 4 (mistralai/Mistral-Small-4-119B-2603-NVFP4) with its official EAGLE draft head (mistralai/Mistral-Small-4-119B-2603-eagle, method: "eagle") crashes EngineCore during profile_run. The target model loads fine and SpeculativeConfig(method='eagle', ...) is created; the crash is in the drafter.
The eagle head resolves to EagleMistralLarge3Model. Its __init__ calls nn.Module.__init__(self) directly — it does not call DeepseekV2Model.__init__ — and only sets a curated subset of attributes. But EagleMistralLarge3Model.forward calls super().forward(...) into DeepseekV2Model.forward, which reads attributes that the bypassed parent __init__ would have set. Those attributes are therefore missing:
self.use_mha — set in DeepseekV2Model.__init__ (computed from qk_nope_head_dim/qk_rope_head_dim) and read in DeepseekV2DecoderLayer.forward. Missing on current main.self.aux_hidden_state_layers — was the same class of failure earlier; it has since been added to EagleMistralLarge3Model.__init__ (= ()), but use_mha was not.
So this is a recurring incomplete-init problem: the eagle subclass manually re-implements a subset of DeepseekV2Model.__init__ and drifts out of sync with what DeepseekV2Model.forward requires.
Serve command
vllm serve mistralai/Mistral-Small-4-119B-2603-NVFP4 \
--quantization compressed-tensors --tensor-parallel-size 1 \
--attention-backend TRITON_MLA --max-model-len 16384 \
--tool-call-parser mistral --enable-auto-tool-choice --reasoning-parser mistral \
--speculative_config '{"model":"mistralai/Mistral-Small-4-119B-2603-eagle","num_speculative_tokens":3,"method":"eagle","max_model_len":"16384"}'Traceback (abridged)
EngineCore ... profile_run -> _dummy_run -> self.drafter.dummy_run(...)
File ".../models/mistral_large_3_eagle.py", line ~91, in forward
output = super().forward(...)
File ".../models/deepseek_v2.py", line ~1378, in forward # DeepseekV2DecoderLayer
if not self.use_mha:
AttributeError: 'EagleMistralLarge3Model' object has no attribute 'use_mha'
...
RuntimeError: Engine core initialization failed.
(On the older mistralllm/vllm-ms4 snapshot the same path raised AttributeError: 'EagleMistralLarge3Model' object has no attribute 'aux_hidden_state_layers' instead.)
Suggested fix
Have EagleMistralLarge3Model.__init__ either call DeepseekV2Model.__init__ (so all attributes its forward depends on are set), or explicitly set every attribute DeepseekV2Model.forward / DeepseekV2DecoderLayer.forward reads — use_mha is the one currently missing. An audit for any further such attributes would prevent the next whack-a-mole.
The target model itself (NVFP4 + TRITON_MLA) loads and serves correctly without the drafter; only the EAGLE path is affected.
Before submitting a new issue...
Your current environment
Versions
0.22.1rc1.dev259+g303916e93(nightly wheel) — also reproduced on the vendormistralllm/vllm-ms4image5.10.0.dev0(main)1.11.3🐛 Describe the bug
Serving Mistral Small 4 (
mistralai/Mistral-Small-4-119B-2603-NVFP4) with its official EAGLE draft head (mistralai/Mistral-Small-4-119B-2603-eagle,method: "eagle") crashesEngineCoreduringprofile_run. The target model loads fine andSpeculativeConfig(method='eagle', ...)is created; the crash is in the drafter.The eagle head resolves to
EagleMistralLarge3Model. Its__init__callsnn.Module.__init__(self)directly — it does not callDeepseekV2Model.__init__— and only sets a curated subset of attributes. ButEagleMistralLarge3Model.forwardcallssuper().forward(...)intoDeepseekV2Model.forward, which reads attributes that the bypassed parent__init__would have set. Those attributes are therefore missing:self.use_mha— set inDeepseekV2Model.__init__(computed fromqk_nope_head_dim/qk_rope_head_dim) and read inDeepseekV2DecoderLayer.forward. Missing on currentmain.self.aux_hidden_state_layers— was the same class of failure earlier; it has since been added toEagleMistralLarge3Model.__init__(= ()), butuse_mhawas not.So this is a recurring incomplete-init problem: the eagle subclass manually re-implements a subset of
DeepseekV2Model.__init__and drifts out of sync with whatDeepseekV2Model.forwardrequires.Serve command
vllm serve mistralai/Mistral-Small-4-119B-2603-NVFP4 \ --quantization compressed-tensors --tensor-parallel-size 1 \ --attention-backend TRITON_MLA --max-model-len 16384 \ --tool-call-parser mistral --enable-auto-tool-choice --reasoning-parser mistral \ --speculative_config '{"model":"mistralai/Mistral-Small-4-119B-2603-eagle","num_speculative_tokens":3,"method":"eagle","max_model_len":"16384"}'Traceback (abridged)
(On the older
mistralllm/vllm-ms4snapshot the same path raisedAttributeError: 'EagleMistralLarge3Model' object has no attribute 'aux_hidden_state_layers'instead.)Suggested fix
Have
EagleMistralLarge3Model.__init__either callDeepseekV2Model.__init__(so all attributes itsforwarddepends on are set), or explicitly set every attributeDeepseekV2Model.forward/DeepseekV2DecoderLayer.forwardreads —use_mhais the one currently missing. An audit for any further such attributes would prevent the next whack-a-mole.The target model itself (NVFP4 +
TRITON_MLA) loads and serves correctly without the drafter; only the EAGLE path is affected.Before submitting a new issue...
EagleMistralLarge3Modelinit gap (not the NVFP4#40260shape error, which did not reproduce on sm_120 here).