[Bugfix][Model Runner V2] Restore multimodal draft capability detection

[TQCB]

Purpose

Separate the target model ability to produce multimodal embeddings from the drafter ability to consume them.

Changes

• Add a SupportsMultiModalEmbeddings protocol and typed capability helper.
• Make SupportsMultiModal inherit the protocol.
• Explicitly implement it on supported Eagle and Inkling drafters.
• Centralize capability detection and fallback warnings in DraftModelSpeculator.load_model().
• Allocate embedding buffers only after the loaded drafter capability is known.
• Remove the Inkling model-type hack.
• Leave signature-only drafters such as Eagle3LlamaForCausalLM unmarked.

Validation

pytest tests/v1/worker/test_gpu_autoregressive_speculator.py -q
11 passed

pre-commit on changed files
passed, including ruff and mypy

Model evals were not run; no model implementation or model math changed.

Duplicate check

No open PR implements this capability contract. #36097 propagates multimodal embeddings but does not provide explicit drafter capability detection.

AI assistance

Prepared with assistance from OpenAI Codex; the human submitter reviewed the changes.

[NickLucche]

can we do supports_multimodal here or similar reflection mechanisms? @DarkLight1337

[NickLucche]

can we spare the dummy forward here (trying to centralize everything into profiling run), and instead rely on self.model class inspection here ?

[TheEpicDolphin]

Thanks for the fix @TQCB. Like Nick suggested, I think it would be better to use supports_multimodal like how it's done here:

vllm/vllm/lora/model_manager.py

Lines 158 to 163 in 8700f86

	self.supports_mm: bool = (
	supports_multimodal(self.model)
	# In case the model only supports LoRA for
	# text modules (e.g. ChatGLM)
	and hasattr(self.model, "get_mm_mapping")
	)

, rather than invoking embed_input_ids.

[TQCB]

better to use supports_multimodal

@TheEpicDolphin This would indeed be better, but supports_multimodal depends on either:

• the draft model inheriting SupportsMultiModal
• or explicitly setting the supports_multimodal attribute which is what is actually checked

Neither of these are the case for most draft models. There is a hacky solution which would be:

signature(self.model.embed_input_ids).bind(
          None,
          multimodal_embeddings=None,
          is_multimodal=None,
      )

but I would prefer having a more explicit contract, rather than relying on this ad hoc check. Ideally there would be an advertized supports_multimodal_embeddings capability declared by draft models for us to use here.

[DarkLight1337]

Ideally there would be an advertized supports_multimodal_embeddings capability declared by draft models for us to use here.

Yeah I prefer this approach as well

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[TQCB]

Hello @NickLucche @TheEpicDolphin @DarkLight1337 , I've updated the PR to use a supports_multimodal_embeddings flag for the drafters. I've updated the following models to support this:

• EagleLlama4ForCausalLM
• EagleMistralForCausalLM
• EagleMistralLarge3ForCausalLM

These are the models I found merged external mm embeddings and don't already inherit SupportsMultiModal. Let me know if there are any others I'm missing.

[TQCB]

Updated to use a protocol instead

[DarkLight1337]

The interface LGTM, but @njhill @ywang96 should take a look at the MRv2 part

[njhill]

Thanks @TQCB! Looks good to me overall, just couple of minor comments.

Could we make the same changes here

vllm/vllm/v1/worker/gpu/spec_decode/multi_module_mtp/speculator.py

Lines 42 to 60 in dd11df0

	self.supports_mm_inputs = MULTIMODAL_REGISTRY.supports_multimodal_inputs(
	self.draft_model_config
	)
	# HACK: the Inkling MTP draft has no MM processor of its own (its draft
	# config is flattened text-only), but it consumes the target's merged
	# embeddings at draft prefill — treat it as MM-capable whenever the
	# target is.
	if (
	not self.supports_mm_inputs
	and self.draft_model_config.hf_config.model_type == "inkling_mtp"
	):
	self.supports_mm_inputs = MULTIMODAL_REGISTRY.supports_multimodal_inputs(
	vllm_config.model_config
	)
	self.inputs_embeds: torch.Tensor | None = None
	if self.supports_mm_inputs:
	self.inputs_embeds = torch.zeros(
	self.max_num_tokens, self.hidden_size, dtype=self.dtype, device=device
	)

, and this can be used to remove the Inkling hack :)

[njhill]

This could be moved to load_model to avoid allocating unnecessarily

In fact would it now make sense to move all of the self.support_mm_inputs logic into load_model?

[TQCB]

Thanks for the comments @njhill . I've moved the support check to the parent, and now all the children just check the flag and allocate the buffers they need based on the result. This feels pretty clean to me, LMK what you think!

[njhill]

Thanks @TQCB LGTM!

cc @TheEpicDolphin @benchislett re the final structure