[Frontend][Rust] Support mm_processor_kwargs in chat completions

[almogtavor]

Purpose

Roadmap item from #44280: mm_processor_kwargs.

The Rust frontend rejected it with mm_processor_kwargs is not supported. on /v1/chat/completions. This accepts it and makes it take effect.

Implementation

The kwargs are merged onto the model's resolved PreProcessorConfig in vllm-chat, right before the image and video preprocessors run, so both modalities honour a per-request override such as min_pixels or max_pixels.

The merge deserializes the request map as a PreProcessorConfig, which type checks known keys, then copies over only the keys the request actually set, leaving the rest of the model default intact. Keys the struct does not name land in its #[serde(flatten)] extra map, which is where model-specific keys from preprocessor_config.json already go, so they still reach the processor rather than being dropped. A bad value (for example "max_pixels": "lots") is a request validation error and returns HTTP 400.

Test

cargo test -p vllm-chat -p vllm-server

All green. New tests in vllm-chat cover: unset kwargs keep the model config, only requested keys are overridden, unknown keys survive through extra, and a wrong value type produces InvalidMmProcessorKwargs classified as a request validation error. A vllm-server test covers the convert layer forwarding the field.

cargo clippy -p vllm-chat -p vllm-server --all-targets is clean and cargo fmt applied.

Runtime check on an H100

Built vllm-rs from this branch against the image's own vLLM commit and served Qwen2-VL-2B with VLLM_USE_RUST_FRONTEND=1, confirming the Rust frontend was the process answering (it exits on startup if it is not). Same 1024x1024 image in every request, only mm_processor_kwargs differs:

request	prompt_tokens
no kwargs (model default)	1391
{"max_pixels": 200704}	278
{"max_pixels": 50176}	86
{"merge_kernel_size": [2, 2]} (unknown key)	1391

{"max_pixels": "lots"} returns HTTP 400 with invalid mm_processor_kwargs: invalid type: string "lots", expected usize and "type": "invalid_request_error".

So the override reaches the image processor and changes the token count, an unknown key is forwarded through extra without breaking the request, and a bad value is a client error rather than a 500.

AI assistance was used for this change.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[BugenZhao]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fbb6f29953

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @almogtavor.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[almogtavor]

@BugenZhao Rebased on main, and all three review points are addressed.

Cache key. This was a real bug. The overrides change the tensors, but the item hash came straight from the raw image or clip hash, so the same image sent twice with different max_pixels would have shared an encoder cache entry. Media hashes now carry a digest of the effective kwargs, canonicalised through a BTreeMap so the digest does not depend on map iteration order. Two tests cover it: different overrides give different keys, and no overrides leave the upstream hash untouched.

Audio. Rejected rather than silently ignored, since the audio preprocessor takes a different config type and would drop the overrides.

Error conversion. Now uses to_report_string() instead of to_string(), per rust/AGENTS.md.

One TODO remains, on the field list in merge_mm_processor_kwargs. PreProcessorConfig derives Deserialize but not Serialize, so a generic JSON merge is not possible and the typed fields have to be listed. Anything not listed still reaches the processor through the struct's extra catch-all, so a new upstream field degrades to pass-through rather than being dropped.

[almogtavor]

@BugenZhao this implements the mm_processor_kwargs item from #44280 for chat completions. Per-request kwargs are merged onto a clone of the model's PreProcessorConfig before processor.preprocess, so unknown keys still flow through extra the way the crate handles model-specific fields.

Verified at runtime rather than only in unit tests: built vllm-rs from this branch against the image's own vLLM commit and served Qwen2-VL, confirming the Rust frontend was the process answering. Same image each request, only the kwargs differ:

no kwargs            1391 prompt_tokens
max_pixels=200704     278
max_pixels=50176       86
unknown key          1391   (forwarded via extra, no error)
max_pixels="lots"    HTTP 400  invalid mm_processor_kwargs: invalid type: string "lots", expected usize

[almogtavor]

@BugenZhao ping on this when you get a chance. all three codex findings are fixed and the threads are resolved, and i just merged main which needed one line because #50368 added a prepare_media call site that now passes None for the kwargs

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @almogtavor.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork