[MODEL] Adding Support for Qwen3.5 Models

[JJJYmmm]

Purpose

This PR adds model support for the upcoming Qwen3.5 models, including both dense and MoE variants.

🫡 Many thanks to @wulipc and @sighingnow for model verification and review, and to @ywang96 and @Isotr0py from the vLLM team for their collaboration!

Reference HF implementation - huggingface/transformers#43830

[mergify]

Hi @JJJYmmm, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[gemini-code-assist]

Code Review

This pull request adds support for the Qwen3.5 series of models, including dense, MoE, and MTP variants. The changes are comprehensive, touching model configuration, definitions, and registries. The implementation largely follows existing patterns for adding new models. However, I've identified a significant code duplication issue in the __init__ method for the multimodal MoE model (Qwen3_5MoeForConditionalGeneration), which could lead to maintenance problems and bugs. I've suggested a refactoring to address this.

I am having trouble creating individual review comments. Click here to see my feedback.

vllm/model_executor/models/qwen3_5.py (966-1000)

The __init__ method of Qwen3_5MoeForConditionalGeneration is almost a complete copy of Qwen3_5ForConditionalGeneration.__init__. This code duplication is problematic for maintainability, as changes in the base class __init__ will not be reflected here, potentially leading to bugs.

Since Qwen3_5MoeForConditionalGeneration inherits from Qwen3_5ForConditionalGeneration, it should ideally call super().__init__(). However, this is not possible because the base class __init__ hardcodes the language model to the non-MoE version.

To resolve this, consider refactoring to reduce code duplication. One approach is to introduce a shared private _init method or a common base class that accepts the language model class as a parameter. This would make the relationship between the dense and MoE versions clearer and more robust.

[mergify]

Documentation preview: https://vllm--34110.org.readthedocs.build/en/34110/

[ywang96]

Thanks for the great contribution! Looking forward to the model release!

[DarkLight1337]

Need to update test registry

[vadiklyutiy]

How we can run it before official weights released?

[Isotr0py]

How we can run it before official weights released?

We have verified the accuracy on preview checkpoints. Let's trigger the nightly wheel building pipeline, so that user can run the model out-of-box immediately when official weights release.

[vadiklyutiy]

How we can run it before official weights released?

We have verified the accuracy on preview checkpoints. Let's trigger the nightly wheel building pipeline, so that user can run the model out-of-box immediately when official weights release.

I meant how to run with dummy weights to check performance for example

[ywang96]

How we can run it before official weights released?

We have verified the accuracy on preview checkpoints. Let's trigger the nightly wheel building pipeline, so that user can run the model out-of-box immediately when official weights release.

I meant how to run with dummy weights to check performance for example

Since the model definition itself is also part of the release, perhaps you can create a dummy model config yourself and test it with --load-format dummy

[vadiklyutiy]

How we can run it before official weights released?

We have verified the accuracy on preview checkpoints. Let's trigger the nightly wheel building pipeline, so that user can run the model out-of-box immediately when official weights release.

I meant how to run with dummy weights to check performance for example

Since the model definition itself is also part of the release, perhaps you can create a dummy model config yourself and test it with --load-format dummy

Yeah, what I actually asked is a config. Didn't know that model config is also will be release on model launch date. Thx for clarification.