Skip to content

[GLM-OCR] GLM-OCR with MTP Support#33005

Merged
vllm-bot merged 25 commits intovllm-project:mainfrom
zRzRzRzRzRzRzR:glm
Jan 26, 2026
Merged

[GLM-OCR] GLM-OCR with MTP Support#33005
vllm-bot merged 25 commits intovllm-project:mainfrom
zRzRzRzRzRzRzR:glm

Conversation

@zRzRzRzRzRzRzR
Copy link
Copy Markdown
Contributor

@zRzRzRzRzRzRzR zRzRzRzRzRzRzR commented Jan 24, 2026 •
edited
Loading

A dense model using the GLM-4-0414 architecture with bias, featuring a completely new VIT structure and MTP implementation.

@zRzRzRzRzRzRzR
init
0c34be2 
Signed-off-by: zRzRzRzRzRzRzR <2448370773@qq.com>
@zRzRzRzRzRzRzR
init for norm
4fd050b 
Signed-off-by: zRzRzRzRzRzRzR <2448370773@qq.com>
@zRzRzRzRzRzRzR
rename with date
8eb025a 
Signed-off-by: zRzRzRzRzRzRzR <2448370773@qq.com>
@zRzRzRzRzRzRzR
rename with date
1e88e46 
Signed-off-by: zRzRzRzRzRzRzR <2448370773@qq.com>
@zRzRzRzRzRzRzR
rename with date
5394b96 
Signed-off-by: zRzRzRzRzRzRzR <2448370773@qq.com>
@zRzRzRzRzRzRzR
refactor using GlmOcrVisionTransformer
a743415 
Signed-off-by: zRzRzRzRzRzRzR <2448370773@qq.com>
@zRzRzRzRzRzRzR
rename with date
b980976 
Signed-off-by: zRzRzRzRzRzRzR <2448370773@qq.com>
@zRzRzRzRzRzRzR
add
45d8248 
Signed-off-by: zRzRzRzRzRzRzR <2448370773@qq.com>
@zRzRzRzRzRzRzR
RMSNorm add
80f7c5f 
Signed-off-by: zRzRzRzRzRzRzR <2448370773@qq.com>
@zRzRzRzRzRzRzR
RMSNorm add
e010952 
Signed-off-by: zRzRzRzRzRzRzR <2448370773@qq.com>
@zRzRzRzRzRzRzR
RMSNorm add
f488249 
Signed-off-by: zRzRzRzRzRzRzR <2448370773@qq.com>
@zRzRzRzRzRzRzR
RMSNorm add
56779c8 
Signed-off-by: zRzRzRzRzRzRzR <2448370773@qq.com>
@zRzRzRzRzRzRzR
loader
26e9aaf 
Signed-off-by: zRzRzRzRzRzRzR <2448370773@qq.com>
@zRzRzRzRzRzRzR
loader change
67e5ca0 
Signed-off-by: zRzRzRzRzRzRzR <2448370773@qq.com>
@zRzRzRzRzRzRzR
loader change
ebcf948 
Signed-off-by: zRzRzRzRzRzRzR <2448370773@qq.com>
@zRzRzRzRzRzRzR
change for GlmOcrMultiTokenPredictorLayer
6502630 
Signed-off-by: zRzRzRzRzRzRzR <2448370773@qq.com>
@zRzRzRzRzRzRzR
test
13f8c19 
Signed-off-by: zRzRzRzRzRzRzR <2448370773@qq.com>
@mergify
Copy link
Copy Markdown
Contributor

mergify Bot commented Jan 24, 2026

Documentation preview: https://vllm--33005.org.readthedocs.build/en/33005/

@mergify mergify Bot added documentation Improvements or additions to documentation multi-modality Related to multi-modality (#4194) new-model Requests to new models speculative-decoding v1 labels Jan 24, 2026
gemini-code-assist[bot]
gemini-code-assist Bot reviewed Jan 24, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for the GLM-OCR model and its Multi-Token Prediction (MTP) variant. The changes are comprehensive, including documentation updates, new model implementation files (glm_ocr.py, glm_ocr_mtp.py), example usage, and extensive test configurations. The implementation correctly follows the patterns for adding new models within the vLLM framework, inheriting and adapting from the existing GLM-4 series models where appropriate. The MTP support is also consistent with existing implementations. After a thorough review, I found no critical or high-severity issues. The code appears to be well-structured and correct.

@zRzRzRzRzRzRzR
test
043a327 
Signed-off-by: zRzRzRzRzRzRzR <2448370773@qq.com>
cursor[bot]
cursor Bot reviewed Jan 24, 2026
View reviewed changes
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.

Comment @cursor review or bugbot run to trigger another review on this PR

Comment thread tests/models/registry.py
@zRzRzRzRzRzRzR
mrope
1aae743 
Signed-off-by: zRzRzRzRzRzRzR <2448370773@qq.com>
@zRzRzRzRzRzRzR
fix mrope_positions
03d667c 
Signed-off-by: zRzRzRzRzRzRzR <2448370773@qq.com>
Isotr0py
Isotr0py reviewed Jan 24, 2026
View reviewed changes
Comment thread vllm/model_executor/models/glm4_1v.py Outdated
Comment thread vllm/model_executor/models/glm_ocr.py Outdated
DarkLight1337
DarkLight1337 reviewed Jan 25, 2026
View reviewed changes
Comment thread vllm/model_executor/models/glm_ocr.py
@zRzRzRzRzRzRzR
Merge branch 'vllm-project:main' into glm
5058ab8
@zRzRzRzRzRzRzR
update
ec18447 
Signed-off-by: zRzRzRzRzRzRzR <2448370773@qq.com>
@zRzRzRzRzRzRzR
remove multimodal_config
1e264f7 
Signed-off-by: zRzRzRzRzRzRzR <2448370773@qq.com>
@zRzRzRzRzRzRzR
Copy link
Copy Markdown
Contributor Author

zRzRzRzRzRzRzR commented Jan 26, 2026

update, please check.

Isotr0py
Isotr0py approved these changes Jan 26, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@Isotr0py Isotr0py left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM now

@Isotr0py Isotr0py added the ready ONLY add when PR is ready to merge/full CI is needed label Jan 26, 2026
@Isotr0py
fix CI
c47b27b 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
@Isotr0py Isotr0py enabled auto-merge (squash) January 26, 2026 12:28
@vllm-bot vllm-bot merged commit bb17e8f into vllm-project:main Jan 26, 2026
55 of 57 checks passed
apd10 pushed a commit to apd10/vllm that referenced this pull request Jan 31, 2026
@zRzRzRzRzRzRzR @Isotr0py
[GLM-OCR] GLM-OCR with MTP Support (vllm-project#33005)
85e9d12 
Signed-off-by: zRzRzRzRzRzRzR <2448370773@qq.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cursor cursor[bot] cursor[bot] left review comments

@Isotr0py Isotr0py Isotr0py approved these changes

@DarkLight1337 DarkLight1337 DarkLight1337 approved these changes

@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 is a code owner

@benchislett benchislett Awaiting requested review from benchislett benchislett is a code owner

@luccafong luccafong Awaiting requested review from luccafong luccafong is a code owner

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon is a code owner

@youkaichao youkaichao Awaiting requested review from youkaichao youkaichao is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@mgoin mgoin Awaiting requested review from mgoin mgoin is a code owner

@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth is a code owner

@houseroad houseroad Awaiting requested review from houseroad houseroad is a code owner

@hmellor hmellor Awaiting requested review from hmellor hmellor is a code owner

@yewentao256 yewentao256 Awaiting requested review from yewentao256 yewentao256 is a code owner

@ProExpertProg ProExpertProg Awaiting requested review from ProExpertProg ProExpertProg is a code owner

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation multi-modality Related to multi-modality (#4194) new-model Requests to new models ready ONLY add when PR is ready to merge/full CI is needed speculative-decoding v1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@zRzRzRzRzRzRzR @Isotr0py @DarkLight1337 @vllm-bot