[Bugfix][MiniCPM-o] Size the duplex HD slice reservation from the frame

[twu3202]

Purpose

Issue 15 of #7636.

_duplex_vision_tokens gives the base frame of every stacked pair three 66-token
blocks (vllm_omni/model_executor/models/minicpmo_4_5/duplex/plugin.py:110), but
how many blocks MiniCPM-o actually makes depends on the frame's size. At the
official max_slice_nums=2 there are only two answers:
MiniCPMVImageProcessor.get_sliced_grid takes
min(ceil(w * h / scale_resolution**2), max_slice_nums) and returns no grid at
all below 2, so the frame is either one tile or three blocks. Anything that fits
in one tile was getting three.

The number is not only a scheduling hint. build_duplex_data_plane_prompt turns
it into [scheduler_token_id] * budget, and preprocess
(minicpmo_4_5_omni.py:392) fits the unit's embeddings into exactly that many
prompt positions:

• More slots than embeddings: the gap is filled with pad-token embeddings in
  front of the unit, and the model reads them. On main a 448x448 pair puts 132
  of them before every unit.
• Fewer slots than embeddings: part of the unit is cut. The warning at
  minicpmo_4_5_omni.py:411 only fires while the whole prompt is still shorter
  than one unit. After that it is the start of the unit that goes, which is where
  the unit marker and the frames sit, and nothing is logged.

Cutting is the worse of the two, so the size is read from the frame's own header,
in the bare-base64 JPEG/PNG Stage0 decodes, and anything that cannot be read keeps
the old count.

The tile size belongs to the checkpoint, not to the code:
MiniCPMVImageProcessor is built with scale_resolution=config.image_size
(minicpmo_4_5_omni_llm.py:3124) and Stage0 loads the checkpoint's own processor
(duplex/stage0.py:686). It is resolved once when the session opens and carried
in the private runtime config next to duplex_scheduler_token_id.

Test Plan

vLLM Version: 0.29.0

vLLM-Omni Commit: fa639b8

1. Stream tests/assets/minicpmo_4_5/response_required_16k.wav with a stacked
   camera pair on every unit into a three-stage server built from fa639b889
   and from this branch, and record the prompt slots and embeddings of each unit.
2. For ten frame sizes in JPEG and PNG, compare what the plugin reserves with
   what MiniCPMOProcessor.process_image on the released checkpoint produces for
   max_slice_nums=[2, 1].
3. Check w * h <= tile against the model's grid search over a wide size range
   at three scale_resolution values.
4. pytest -q tests/model_executor -m 'core_model and cpu', plus tests/engine
   and tests/entrypoints at the same marker, and
   test_duplex_single_session_video_input on both commits.

Test Result

1. Live server

Real weights. To fit a 48 GB card, Stage 0 ran with
max_num_batched_tokens: 4096 and max_model_len: 32768; the rest of
minicpmo_4_5.yaml is as shipped. The clip goes in as 200 ms appends, and each
unit carries the frame plus a 2W x H composite. A logging-only wrapper around
preprocess recorded how many positions each unit was scheduled into and how many
embeddings Stage0 built for it.

frames	commit	slots per unit	embeddings per unit
448x448	fa639b889	277	145	132 pads in front
448x448	this branch	145	145
960x540	both	277	277
960x540, base frame forced to one block	this branch	145	277	131-132 short

The last unit of the clip gets 12 more slots in every row, with or without this
change.

The forced row is a deliberate under-reservation, to check the measurement can
see one. All five units with frames were short, and the server logged the
truncation warning once, for the first of them.

One thing showed up that this change does not cause. With 448x448 frames, this
branch stops mid-answer on this clip in 6 of 6 runs: four audio deltas and no
response.done. fa639b889 finishes all 6. With the pads gone the model's output
changes, and an empty TTS segment now ends just before the answer starts, while
the client's last units are still arriving. That hits a race in the silence
continuation: the continuation armed by that segment waits one chunk period, finds
new units and an open response, gives up, and nothing schedules another unit.
Replaying the same order of outputs through the session runner on fa639b889
stops the same way. Reported in #7729.

2. What the model actually slices

66 tokens per block, stacked pair (current frame + composite):

frame	plugin	checkpoint	before
224x224	132	132	264
320x180	132	132	264
448x448	132	132	264
448x449	264	264	264
640x480	264	264	264
896x672	264	264	264
960x540	264	264	264
1280x720	264	264	264
540x960	264	264	264
1920x1080	264	264	264

Same in JPEG and PNG, 20 cases, no mismatches. The plugin column reads the tile
from the checkpoint's config through the same reader a session uses. The released checkpoint
uses scale_resolution=448, so 448x448 is the largest frame that still fits one
tile.

Worth being clear about the size of this: only frames that fit in one tile
change. 640x480 and everything above still reserves 264, so an ordinary webcam
capture is unaffected.

3. Boundary sweep

66 861 sizes, including 4000x40, 40x4000, 1x200000 and everything near the
boundary, at three scale_resolution values:

scale_resolution	grids at max_slice_nums=2	sizes where w * h <= tile gets it wrong
336	None, (1, 2), (2, 1)	0
448	None, (1, 2), (2, 1)	0
560	None, (1, 2), (2, 1)	0

4. CI scope

	result
tests/model_executor	2316 passed, 1 skipped, 0 failed
tests/engine	1074 passed, 1 skipped, 0 failed
tests/entrypoints	1725 passed, 3 skipped, 1 xfailed, 0 failed

tests/model_executor/models/minicpmo_4_5/duplex/test_scheduler_token_budget.py
alone: 70 passed. It is a new file using the new signature, so it does not run
against main; what it pins is the before column in §2.

test_duplex_single_session_video_input, the video case of the merge-level
Omni · MiniCPM-o 4.5 Duplex Test, passes on fa639b889 and on this branch with
real weights and the Stage 0 settings from §1. Its 448x448 frames get 132 fewer
slots per unit here, and no unit is short.

Not run: the rest of the CUDA jobs the MiniCPM-o group gates, including the
duplex perf job.

Why the tile is read from the config

The obvious alternative is to call get_sliced_grid directly. plugin.py runs
engine-side and nothing it imports reaches minicpmo_4_5_omni_llm, so calling it
per append would drag the whole model module into the engine core. The comparison
is spelled out in a comment instead, against a value that comes from the
checkpoint. The test parametrizes scale_resolution over 336/448/560 to keep
that honest — a hardcoded 448 fails six of those cases.

Note

Neither _duplex_vision_tokens nor duplex_scheduler_token_budget had any
tests. The new file also covers the audio side of the budget, which this does not
change, and which frame of a pair gets sliced, which nothing covered before.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/model_integration.md, docs/design/module/ar_runtime.md.

Module owners: @tzhouam @fake0fan @Gaohan123

Routing: @tzhouam via module of the changed files, CODEOWNERS; @fake0fan via module of the changed files; @Gaohan123 via module of the changed files

@twu3202, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[twu3202]

Self-review done. Two things worth mentioning, both fixed in 24a61bd:

• _model_vision_tile_pixels had no unit coverage: every case drove the budget with a literal tile, so a checkpoint whose scale_resolution could not be read would still have passed the file while production quietly fell back to the sliced reservation. Covered now, round trip through the runtime config included.
• build_duplex_data_plane_prompt parsed a frame header twice per append. Moved back into the branch that uses it.

I also fixed a number in the description: the per-file count read 87, which is the whole duplex/ directory. This file is 70 now, 60 before these tests.

What else I checked: the reservation only moves up when the tile or the header cannot be read, duplex_vision_tile_pixels is server-owned so a client cannot shrink it, and the three CI scopes pass.

[linyueqian]

Reviewed at 24a61bd9 (static read against main and against the processor code; nothing executed). The reservation now matches what Stage 0 emits: Stage 0 hands the decoded frame to process_image([frame], max_slice_nums=2) without resizing, get_sliced_grid at max_slice_nums=2 is binary (ceil(w*h/scale^2) <= 1 gives one tile, anything else gives the 2-cell grid plus source, so 1 or 3 blocks), and the plugin's width * height <= tile_pixels is the same predicate on the same raw size for positive integers, including the exact-tile and one-pixel-over boundaries. EXIF orientation and alpha cannot change the decision because neither side transposes before sizing and the area is rotation invariant. Every failure to read the header falls back to the sliced count, which over-reserves, and Stage 0 raises on the same frame, so there is no reachable under-reserve from decoding. Frames past the first are counted, not decoded. test_a_stacked_pair_reserves_what_the_model_slices_it_into pins the counts against the processor's own grid walk over JPEG and PNG, three tile sizes and the boundary sizes. Two suggestions inline (one about where the tile size is read from, one duplicate decode on first appends); neither blocks. Approving once the general lane reports on this head.

[linyueqian]

[suggestion] The tile area is read from the model config (slice_config.scale_resolution, then image_size) while Stage 0 slices with the checkpoint's processor, which reads preprocessor_config.json. For the released checkpoint all three are 448 so they agree, but a deployment that overrides one and not the other turns this into an under-reservation (a 500x500 base frame with a 560 tile here and 448 in the processor reserves 132 instead of 264 vision tokens, and the worker then drops the unit tail with a warning). Reading the value from the same processor Stage 0 loads, or falling back to the sliced count whenever the two sources disagree, keeps the reservation tied to the code that does the slicing. The tests give both sides the same scale, so they cannot see this.

[linyueqian]

[suggestion] For a first append with a stacked pair and a known tile size this is the second base64 decode and header parse of the same frame in one call: duplex_scheduler_token_budget at the top of the function already computed _duplex_vision_tokens for the same payload. Computing the vision tokens once and reusing them in both branches keeps the append path to one decode.