[Chore] Port more vendored configs and processors to upstream Transformers by hmellor · Pull Request #60823 · vllm-project/vllm

hmellor · 2026-10-09T09:41:42Z

Overview

Follow-up to #60623. Ports the vLLM code for 6 more models to the upstream Transformers 5.16.1 schemas, and deletes their vendored configs and processors (9 files). Also fixes registry entries that kept models out of CI.

Related PRs:

[Chore] Use upstream Transformers DeepseekV4Config #61392: the DeepSeek-V4 part of this PR, split out.
[Chore] Use upstream Transformers MiniMaxM3VLConfig #61813: the MiniMax-M3 part of this PR, split out.
[Chore] Use upstream Transformers InklingProcessor #61883: the Inkling part of this PR, split out.
[BUGFIX] Fix opencua processor on transformers 5 #47438: a stale draft (last updated 2026-07-16) with a partial OpenCUA fix. This PR supersedes it.

Claims

Deletes the vendored configs for step3p5, muse_glimmer, cosmos3_edge, colpali, colmodernvbert and qianfan_ocr. It also deletes the vendored cosmos3_edge and muse_glimmer processors, and the model code now reads the upstream fields:
- Step3p5: reads layer_types / mlp_layer_types / per-type rope_parameters. Upstream Step3p7TextConfig is registered in _CONFIG_REGISTRY, because the hub remote code fails validation.
- MuseGlimmer and Cosmos3-Edge: they consume upstream processor outputs. Cosmos3-Edge reorders upstream's merge-window patches back to raster order before SigLIP2.
- ColPali: reads PaliGemma from vlm_config.
- Qianfan-OCR: a small adapter maps the upstream vision fields onto InternViT's.
- ColModernVBERT: ModernVBertModel is mapped to ColModernVBertForRetrieval.
Fixes:
- ColModernVBERT: the vendored config dropped the checkpoint's sliding-window rope_theta (160000 → 10000). vLLM now matches the HF reference.
- Step-3.5-Flash MTP: built with an IndexError before; it now builds.
- InternS1-Pro: failed to initialize (model_config was not set).
- OpenCUA: Transformers v5 builds a processor's sub-processors from its __init__ signature, which here only exposed tokenizer, so the processor had no image_processor. Fixed by overriding get_attributes.
- Kimi-K2.5: the spec-decode image token is now read from image_token_id. ModelConfig.is_quantized() now uses the arch config, so it sees quantization configs nested in text_config.
Renames the vendored InternVL{,Image,Video}Processor to InternVLChat*, so they stop shadowing the upstream classes.
Registry entries CI was skipping:
- FireRedASR2, FunASR and InternS1 caps removed; they work on Transformers v5.
- Isaac and InternS1-Pro caps narrowed to the HF comparison.
- Phi-4-reasoning-vision and OpenCUA pinned to the compatible refs/pr/8 and refs/pr/10 (OpenCUA's cap narrowed to the HF comparison).
- Cohere2-MoE switched to the renamed CohereLabs/North-Mini-Code-1.0.
Kept vendored:
- qwen3_asr: upstream silently loads the original nested checkpoints with default sizes.
- Step3p7 and MiniMax-M3 processors: upstream tiles differently, or lacks the opt-in resize modes.

Validation

main vs this branch on CPU, transformers 5.19.0. Each checkpoint's config and meta-device model build were dumped with .agents/skills/transformers-upgrade/scripts/ and diffed.

Checkpoint	Model diff vs `main`
Step-3.5-Flash, Step-3.7-Flash (+ MTP drafts)	0
Cosmos3-Edge	0 params and modules. `num_layers` goes 56 → 28: the attention layers only, as Nemotron-H no longer counts MLP blocks.
Muse-Glimmer-30B (+ DFlash drafter)	0 params. Constant `use_qk_norm` / `use_output_gate` flags dropped.
colpali-v1.3-hf, Qianfan-OCR	0
colmodernvbert-merged	sliding-window `rope_theta` 10000 → 160000 (the fix above). vs HF fp32: max abs diff 0.39 → 8e-5.

Processor outputs, bit-identical to main unless noted:

Cosmos3-Edge: pixels are identical after the raster reorder.
Qianfan, ColPali, ColModernVBERT: identical.
Muse-Glimmer: upstream's torchvision LANCZOS differs from PIL by at most 1 uint8 level on about 0.3% of pixels.

pytest tests/transformers_utils tests/config tests/models/test_registry.py tests/tokenizers_ tests/test_config.py  # failures identical to main (environment: no Ray/CUDA on the test host)
pytest tests/models/multimodal/processing/test_common.py -k "<touched models>"  # 96 passed, 9 skipped

Pending on GPU:

Evals: ColModernVBERT (tests/models/multimodal/pooling/test_colmodernvbert.py) and Muse-Glimmer image/video.
Smoke tests: Step-3.5-Flash with MTP.
tests/models/test_initialization.py for the touched architectures.

Pull Request Checklist

I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).
AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

…rmers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

hmellor · 2026-10-09T12:55:47Z

gsm8k on 8x B200 (transformers 5.19.0). main (2c99ee9) ran on GPUs 0-3 and this branch (cfeac40) on GPUs 4-7, side by side with TP=4.

Model	`vllm/models` package	`main`	branch
`zai-org/GLM-5.3-Flash`	glm5next	0.914	0.908
`deepseek-ai/DeepSeek-V4-Flash`	deepseek_v4	0.942	0.932

Single runs over all 1319 questions, so the differences of 0.6 and 1.0 points (8 and 13 questions) are within run-to-run noise. Invalid responses were 0.000 to 0.001 for both.

DeepSeek-V4-Flash covers this PR's deepseek_v4 port: the config-derived compress_ratios, the hash layers, and the nested {main, compress} rope.

vllm serve <model> --tensor-parallel-size 4 --gpu-memory-utilization 0.9 --max-model-len 8192
python tests/evals/gsm8k/gsm8k_eval.py --port <port>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

hmellor · 2026-10-17T21:49:22Z

/ci run

github-actions · 2026-10-17T21:49:43Z

✅ Triggered Buildkite CI #95623 for commit 4a9082c77263.

johnnynunez-nv · 2026-10-19T00:06:06Z

This also fixes Cosmos3-Edge with --model-impl transformers, which is broken on main because the Transformers backend receives the vendored classes instead of the upstream ones. CPU check without weights (config, processor and tokenizer files of the HF export only), transformers 5.19.0:

	`main` (`36e0ae2`)	this branch (`4a9082c`)
`ModelConfig(model, model_impl="transformers").hf_config`	vendored Nemotron-H-backed `Cosmos3EdgeConfig`; `AutoModel.from_config(...)` on meta raises `AttributeError: 'Cosmos3EdgeTextConfig' object has no attribute 'hidden_act'`	`transformers` `Cosmos3EdgeConfig`, 28 decoder layers, architecture `TransformersMultiModalForCausalLM`
Processor picked by the Transformers backend (`MultiModalProcessingInfo.get_hf_processor`)	vendored `Cosmos3EdgeProcessor`; `_get_num_multimodal_tokens(image_sizes=...)` raises `AttributeError: 'Cosmos3EdgeImageProcessor' object has no attribute 'get_number_of_image_patches'`	`transformers` `Cosmos3EdgeProcessor`; pixels identical to `AutoProcessor`, 193x317 image -> 308 patches / 77 tokens

On GPU, this branch also loads the HF export with LLM(..., model_impl="transformers") (Using Transformers modeling backend) and generates text.

mergify · 2026-10-19T03:38:57Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @hmellor.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

…sors Transformers >= 5.9 stores registered image processors as dicts, which crashes get_possibly_dynamic_module() when resolving the feature extractor by name. Apply the same workaround as MiniMaxVLProcessor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> # Conflicts: # vllm/transformers_utils/config.py # vllm/transformers_utils/configs/__init__.py

hmellor · 2026-10-19T21:14:56Z

/ci run

github-actions · 2026-10-19T21:15:17Z

✅ Triggered Buildkite CI #96000 for commit d6f79e40f011.

Resolve the conflict with vllm-project#60823's Step-3.5 config refactor: keep the attention_cls hook in Step3p5DecoderLayer, and read Step-5's MTP layer types from layer_types_with_mtp. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Andreas Karatzas <akaratza@amd.com>

ZJY0516 · 2026-10-20T07:36:59Z

It breaks ColPali model test, #62627. Please test more and be more careful next time

hmellor · 2026-10-20T14:12:17Z

This was a merge race. The test that failed on main passed in this PR

…ult rope type transformers >= 5.17 builds MuseGlimmerVisionRotaryEmbedding only for rope_type "axial", and maps the checkpoint's "default" to it through the config class's default_rope_type in standardize_rope_params. vLLM's vendored MuseGlimmerVisionConfig replaces transformers' in AutoConfig and lacked the attribute, so on transformers 5.17-5.19 the Transformers backend failed: "supports only axial rope, but requested default" (5.16.1, the floor of v0.32's range, still accepted "default"). With it, get_config's patch_rope_parameters maps the vision sub-config to "axial". Upstream fixes this by deleting the vendored config in 19997aa (vllm-project#60823, after v0.32.0); retire this commit when rebasing past it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

hmellor and others added 2 commits October 9, 2026 10:35

[Chore] Port more vendored configs and processors to upstream Transfo…

255002e

…rmers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Leave OpenCUA to vllm-project#47438

6c3352b

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

mergify Bot added deepseek Related to DeepSeek models multi-modality Related to multi-modality (#4194) minimax inkling DSv4 DSv4.1 Related to DeepSeek-V4.1 models cpu Related to CPU backends speculative-decoding labels Oct 9, 2026

Restore OpenCUA fix, superseding vllm-project#47438

cfeac40

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

This was referenced Oct 9, 2026

[BUGFIX] Fix opencua processor on transformers 5 #47438

Closed

[Bugfix][DeepSeek V4] Name the config field a checkpoint is missing #57818

Closed

[Bugfix] DeepSeek V4: support transformers >= 4.57 normalized compress_ratios (AMD + NVIDIA) #44031

Closed

hmellor marked this pull request as ready for review October 9, 2026 17:03

hmellor requested review from AndreasKaratzas, DarkLight1337, MatthewBonanni, WoosukKwon, benchislett, dllehr-amd, hongxiayang, mgoin, njhill, tjtanaa, tlrmchlsmth, yewentao256, ywang96 and zyongye as code owners October 9, 2026 17:03

mergify Bot removed the needs-rebase label Oct 15, 2026

Drop Inkling changes, moved to a separate PR

f3aca7f

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

hmellor mentioned this pull request Oct 15, 2026

[Chore] Use upstream Transformers InklingProcessor #61883

Merged

6 tasks done

Merge branch 'main' into unvendor-followups

d519f11

Isotr0py approved these changes Oct 16, 2026

View reviewed changes

Comment thread vllm/model_executor/models/step3p5.py

Merge branch 'main' into unvendor-followups

4a9082c

hmellor enabled auto-merge (squash) October 17, 2026 21:49

github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 17, 2026

johnnynunez-nv mentioned this pull request Oct 19, 2026

[Bugfix][Frontend] Constrain Qwen3 XML tool calls written without <tool_call> #62364

Closed

6 tasks done

This was referenced Oct 19, 2026

[Bugfix][Frontend] Constrain Qwen3 XML tool calls written without <tool_call> #62368

Open

brain: Cosmos3-Edge on vLLM 0.32.0 with an NVFP4 KV cache johnnynunez/cascade#397

Merged

mergify Bot added the needs-rebase label Oct 19, 2026

vllm-project deleted a comment from github-actions Bot Oct 19, 2026

Merge remote-tracking branch 'upstream/main' into unvendor-followups

d6f79e4

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> # Conflicts: # vllm/transformers_utils/config.py # vllm/transformers_utils/configs/__init__.py

mergify Bot removed the needs-rebase label Oct 19, 2026

hmellor merged commit 19997aa into vllm-project:main Oct 20, 2026
233 checks passed

hmellor deleted the unvendor-followups branch October 20, 2026 01:20

wenjinhust mentioned this pull request Oct 20, 2026

[CI][Bugfix] ColPali miss hidden_size #62627

Merged

6 tasks

zkawfanx mentioned this pull request Oct 22, 2026

[Start here] Classify the pinned DSv4.1 timeline and prepare chronological backport batches zkawfanx/vllm#18

Open

3 of 5 tasks

Navigation Menu

Repository navigation

[Chore] Port more vendored configs and processors to upstream Transformers - #60823

[Chore] Port more vendored configs and processors to upstream Transformers#60823
hmellor merged 10 commits into
vllm-project:mainfrom
hmellor:unvendor-followups

hmellor commented Oct 9, 2026 •

edited

Loading

Uh oh!

hmellor commented Oct 9, 2026

Uh oh!

Uh oh!

hmellor commented Oct 17, 2026

Uh oh!

github-actions Bot commented Oct 17, 2026

Uh oh!

johnnynunez-nv commented Oct 19, 2026

Uh oh!

mergify Bot commented Oct 19, 2026

Uh oh!

hmellor commented Oct 19, 2026

Uh oh!

github-actions Bot commented Oct 19, 2026

Uh oh!

Uh oh!

ZJY0516 commented Oct 20, 2026

Uh oh!

hmellor commented Oct 20, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

hmellor commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Claims

Validation

Uh oh!

hmellor commented Oct 9, 2026

Uh oh!

Uh oh!

hmellor commented Oct 17, 2026

Uh oh!

github-actions Bot commented Oct 17, 2026

Uh oh!

johnnynunez-nv commented Oct 19, 2026

Uh oh!

mergify Bot commented Oct 19, 2026

Uh oh!

hmellor commented Oct 19, 2026

Uh oh!

github-actions Bot commented Oct 19, 2026

Uh oh!

Uh oh!

ZJY0516 commented Oct 20, 2026

Uh oh!

hmellor commented Oct 20, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

hmellor commented Oct 9, 2026 •

edited

Loading