[Bugfix][MiniCPM-o] Align structured chat content with native omni rendering

[Sy0307]

Purpose

Related to #6815 (test_image_to_text_audio_001[async_chunk]: AssertionError: The audio content is not same as the text). Image captions can be spoken correctly and then followed by extra speech, failing the existing 0.8 audio/text consistency threshold.

Cause and change

Generic string-content rendering adds a newline between media and text. The official MiniCPM-o implementation in chat(omni_mode=True) instead concatenates those parts directly. In the four-image diagnostic comparison, the framework input had 128 embedding rows versus the native reference's 127; removing the extra newline token (198) made the remaining embeddings identical.

Add a bundled structured-content ChatML template adapted from the model checkpoint, preserving explicit text whitespace without inserting separators. Enable it in the MiniCPM online test server and serving recipe. Make the image test explicitly image-first, matching the native image-chat layout. Include CPU regression tests, package data, and template path dependencies in the existing Buildkite ready/merge jobs.

The image order change is part of this fix: image-first with the old string template still failed in prior controls. This PR does not alter the consistency threshold, TTS/thinking kwargs, sampler, or stopping policy.

Test Plan

From the repository root, with model weights and test dependencies available:

python -m pytest -q tests/model_executor/models/minicpmo_4_5/test_chat_template.py -m 'core_model and cpu' --run-level core_model
python -m pytest -sv tests/e2e/online_serving/test_minicpmo_4_5.py::test_text_to_audio_001 tests/e2e/online_serving/test_minicpmo_4_5.py::test_audio_to_text_audio_001 tests/e2e/online_serving/test_minicpmo_4_5.py::test_image_to_text_audio_001 tests/e2e/online_serving/test_minicpmo_4_5.py::test_video_to_text_audio_001 --run-level full_model

The image node above is also the original reproducer on the unpatched checkout; its failure depends on the generated image.

Hardware: single H200. vLLM: 0.28.0; PyTorch: 2.13.0+cu130; Transformers: 5.15.1. vLLM-Omni base: 504e5a6a6e7d3d00105a1c04ba5bd84fd62d5d15.

Test Result

• Standalone patch on the base above: 8 CPU template tests passed; all four original online nodes passed (4 passed, 14 warnings in 517.36s). The original image node passed all five concurrent requests with similarity approximately 1.0.
• Repository-pinned pre-commit checks passed for all seven changed files, including Ruff lint/format, mypy 3.10, Markdown, YAML, Buildkite schema, test marks, SPDX, and whitespace. DCO sign-off verified.
• Standalone H200 expanded validation: 212/212 requests passed, similarity approximately 1.0 throughout (floating-point range 0.9999999999999998–1.0000000000000002). Four fixed images × five requests plus 32 images (seeds 64–95) × one/five concurrent requests; image-first with the same TTS kwargs and existing consistency assertion. The diagnostic runner caps Stage 0 KV cache at 8 GiB; the four original online nodes above use the original deploy memory configuration. Raw generated audio and per-request scores were retained locally.

Scope

Existing deployments must enable both documented template flags. Text-first requests still have speech-quality counterexamples in the earlier expanded tests; this PR does not claim to resolve arbitrary input orders or establish a unique introducing PR. Formal H100/Buildkite results remain separate from the local H200 validation.

AI assistance: Codex assisted with investigation, implementation, tests, self-review, and this description.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/model_integration.md.

Module owners: @tzhouam @gcanlin @y-null

Routing: @tzhouam via module of the changed files, CODEOWNERS; @gcanlin via module of the changed files; @y-null via semantic router, model owner

@Sy0307, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit 375cb3ea08d3 produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[congw729]

LGTM

[tlysanhuo]

Thanks for the precise diagnosis — the 128-vs-127 embedding row comparison is a clean way to pin this.

One question on scope: the PR fixes this via an opt-in bundled template plus updating the e2e test to use it. If the extra separator comes from our generic content rendering path (rather than an upstream/transformers default), wouldn't fixing the default renderer for minicpmo keep every deployment correct without requiring the flag? If it is upstream behavior, the bundled template is the right mechanism — maybe worth a line in the description clarifying which it is.

Context that may be useful: we ran the exact failing node 7× green on a local H100 at the same commit with the old template (report in #6815), so the extra-speech tail doesn't reproduce on every stack — consistent with a borderline similarity sitting right around the 0.8 gate. Removing the extra conditioning token should remove the variance source entirely, but the default path would still carry it.

Minor: transformers.utils.chat_template_utils._compile_jinja_template is a private API — if it moves in a future transformers release the CPU test breaks; is there a public helper worth using instead?

[Sy0307]

Minor: transformers.utils.chat_template_utils._compile_jinja_template is a private API — if it moves in a future transformers release the CPU test breaks; is there a public helper worth using instead?

Need follow-up PR to fix it. Can you help it and make this bugfix more completed? Thx.

[tlysanhuo]

Follow-up sent: #7362 — swaps the private _compile_jinja_template for the public render_jinja_template as discussed. conversations takes a batch and the return is (rendered_conversations, continuation_chunks), so the string comes out via [0][0] (noted in a comment). Verified on transformers 5.14.1: your test file passes 8/8 unchanged, ruff clean.