[Model]Support MiniCPM-o 4.5

[tc-mb]

Purpose

Hi team — I'm from the MiniCPM-V / MiniCPM-o model team and have been
maintaining the integration of the V and O series into vLLM. We have been
following vllm-omni since day one, and this PR brings both
MiniCPM-o 2.6 and MiniCPM-o 4.5 into the project.

Apologies for the delay — over the past weeks the team has been focused on
shipping the MiniCPM-o technical report
(arXiv 2604.27393) and the
MiniCPM-V 4.6 release. Hope this PR makes it easier for the community to
serve the MiniCPM-o family on vllm-omni, and we look forward to deeper
collaboration on omni-modal serving going forward.

What's added

Models (vllm_omni/model_executor/models/)

• minicpmo_4_5/: full omni pipeline for MiniCPM-o 4.5
  • minicpmo_4_5_omni.py — top-level conditional generation wrapper
  • minicpmo_4_5_omni_llm.py — thinker (LLM) stage
  • minicpmo_4_5_omni_tts.py — talker (TTS) stage
  • minicpmo_4_5_omni_t2w.py — token-to-waveform stage
• minicpmo_2_6/: full omni pipeline for MiniCPM-o 2.6 (same 4-file
  layout as 4.5).
• Registry entries in model_executor/models/registry.py for all 8 new
  architectures (MiniCPMO{26,45}Omni{,LLM,TTS,T2W}ForConditionalGeneration).

Stage input processors
(vllm_omni/model_executor/stage_input_processors/)

• minicpmo_2_6_omni.py, minicpmo_4_5_omni.py — provide
  llm2tts / tts2t2w adapters wired into the stage YAMLs below.

Default stage configs (vllm_omni/model_executor/stage_configs/)

• minicpmo.yaml — MiniCPM-o 2.6 default
• minicpmo_8x4090.yaml — MiniCPM-o 2.6 on an 8×4090 host
• minicpmo45_2gpu.yaml — MiniCPM-o 4.5, 2-GPU layout
• minicpmo45_3gpu.yaml — MiniCPM-o 4.5, 3-GPU (thinker TP=2)
• minicpmo45_8x4090.yaml — MiniCPM-o 4.5 on an 8×4090 host

Online serving example (examples/online_serving/minicpmo/)

• gradio_demo.py, run_gradio_demo.sh, README.md — single Gradio UI
  that drives both 2.6 and 4.5 endpoints over the OpenAI-compatible API.

API server (vllm_omni/entrypoints/openai/api_server.py)

• Pre-load the model's HF config with trust_remote_code=True (with GPU
  visibility temporarily hidden) so HuggingFace transformers_modules is
  registered in the API server process. This is required for ZMQ pickle
  deserialization of MiniCPM-o stage outputs that reference dynamic
  modules. Failures cleanly fall through, so non-trust_remote_code
  models are unaffected.

Notes

This PR was merged with the latest main (clean fast-forward from
fdb0efea); MiniCPM-o-specific code lives entirely under the paths
listed above and the changes outside those paths are limited to the
registry entry and the API-server pre-load described above.

Test Plan

We validate both models via the OpenAI-compatible server and the Gradio
demo shipped in examples/online_serving/minicpmo/.

1. Launch a backend server

# MiniCPM-o 4.5, 8×4090 layout (thinker TP=4 on GPU0-3, talker+t2w share GPU4)
vllm-omni serve <path-to-MiniCPM-o-4_5> \
    --stage-configs-path vllm_omni/model_executor/stage_configs/minicpmo45_8x4090.yaml \
    --trust-remote-code \
    --host 0.0.0.0 --port 8099

# MiniCPM-o 2.6
vllm-omni serve <path-to-MiniCPM-o-2_6> \
    --stage-configs-path vllm_omni/model_executor/stage_configs/minicpmo.yaml \
    --trust-remote-code \
    --host 0.0.0.0 --port 8091

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 74b5e5fd67

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[hsliuustc0106]

thanks for you contribution, can we split it into 2 PRs?

[tc-mb]

thanks for you contribution, can we split it into 2 PRs?

No problem.

I'll continue merging this PR with minicpm-o4.5, and then open another PR to merge o2.6. Do you think this is appropriate?

Opening two PRs simultaneously might require back-and-forth discussions about the merging syntax. Merging one first will allow me to understand the requirements for merging vllm-omni, saving you the trouble of reviewing twice.

[lishunyang12]

thanks for you contribution, can we split it into 2 PRs?

No problem.

I'll continue merging this PR with minicpm-o4.5, and then open another PR to merge o2.6. Do you think this is appropriate?

Opening two PRs simultaneously might require back-and-forth discussions about the merging syntax. Merging one first will allow me to understand the requirements for merging vllm-omni, saving you the trouble of reviewing twice.

We can focus on MiniCPM-o 4.5 first. You can down scope this pr so that we can fast forward the reviewing process.

[zhumingjue138]

please add UT case if it is necessary

[tc-mb]

please add UT case if it is necessary

ok, added a unit-test suite for the MiniCPM-o 4.5 path in tests/model_executor/models/minicpmo_4_5
PTAL.

[Nightwing-77]

could we have E2E tests covering offline and online inference!?

[hsliuustc0106]

I suggest to add them in nightly-test, please follow the corresponding md

[Gaohan123]

LGTM. Thanks