[Model Runner V2] support qwen35 / mamba hybrid model

[izhuhaoran]

Purpose

As titled, this pr support qwen3.5 model for model runner v2.

Test

Qwen3.5-35B-A3B-FP8, mtp with num_speculative_tokens = 2
NVIDIA H20 GPUs, TP=2

bence serve

bench args: --dataset-name spec_bench --spec-bench-output-len 1024 --max-concurrency 32 --num-prompts 512

Traffic request rate: inf
Burstiness factor: 1.0 (Poisson process)
Maximum request concurrency: 32
============ Serving Benchmark Result ============
Successful requests:                     512       
Failed requests:                         0         
Maximum request concurrency:             32        
Benchmark duration (s):                  204.40    
Total input tokens:                      140763    
Total generated tokens:                  503319    
Request throughput (req/s):              2.50      
Output token throughput (tok/s):         2462.37   
Peak output token throughput (tok/s):    1120.00   
Peak concurrent requests:                45.00     
Total token throughput (tok/s):          3151.02   
---------------Time to First Token----------------
Mean TTFT (ms):                          141.86    
Median TTFT (ms):                        108.06    
P99 TTFT (ms):                           292.57    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          12.57     
Median TPOT (ms):                        12.69     
P99 TPOT (ms):                           14.46     
---------------Inter-token Latency----------------
Mean ITL (ms):                           31.56     
Median ITL (ms):                         29.19     
P99 ITL (ms):                            113.78    
----------------End-to-end Latency----------------
Mean E2EL (ms):                          12517.42  
Median E2EL (ms):                        13103.73  
P99 E2EL (ms):                           14919.61  
---------------Speculative Decoding---------------
Acceptance rate (%):                     75.33     
Acceptance length:                       2.51      
Drafts:                                  200764    
Draft tokens:                            401528    
Accepted tokens:                         302477    
Per-position acceptance (%):
  Position 0:                            84.38     
  Position 1:                            66.29     
==================================================

lm_eval

lm_eval --model local-completions --tasks gsm8k --batch_size 128 --model_args model=Qwen/Qwen3.5-35B-A3B-FP8,base_url=http://localhost:8000/v1/completions,max_retries=3

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.7763	±	0.0115
		strict-match	5	exact_match	↑	0.7582	±	0.0118

[gemini-code-assist]

Code Review

This pull request adds support for Qwen3.5/Mamba hybrid models to the Model Runner V2. The changes are extensive, touching configuration, KV cache management, attention utilities, and model state handling to accommodate the hybrid nature of these models. The implementation correctly introduces new logic for Mamba's state caching and integrates it with the existing attention mechanism. One area for improvement is a potential performance bottleneck in the metadata preparation for hybrid models, which I've commented on.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @izhuhaoran.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @izhuhaoran.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @izhuhaoran.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @izhuhaoran.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[WoosukKwon]

Hi @izhuhaoran, sorry for the delay in the review. Can you please rebase your PR given that we had major updates?

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @izhuhaoran.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[izhuhaoran]

@WoosukKwon The current CI failures appear unrelated to this PR. I reproduced the two failing tests on top of main, and they fail there as well. Could you please help manually merge this PR?