feat(multimodal): let vLLM workers expand Kimi-K3 and GLM-5.3-Flash image anchors

[hello-alexmcc]

Description

Problem

Kimi-K3 and GLM-5.3-Flash engines that advertise worker-side media processing (mm_processor=inprocess / mm_processor=redis) never receive media from the router. It keeps preprocessing on its own side and counts every image request as

smg_mm_processing_total{model="kimi-k3-nvfp4",mode="router",reason="model_not_opted_in"} 26
smg_mm_processing_total{model="glm-5.3-flash",mode="router",reason="model_not_opted_in"}

In strict mode (SMG_MM_PROCESSING=worker) both models fail every image request with multimodal_worker_processing_unsupported_model.

ModelProcessorSpec::worker_expandable defaults to false and neither KimiK3VisionSpec nor Glm53FlashSpec overrides it, so ensure_worker_expandable in the gRPC router's media planning classifies both as not opted in and falls back to router-side processing.

Solution

Opt both specs in for images, the same way MiniMax-M3 was opted in (#2609):

• Kimi-K3, image. The router's anchor is <|media_pad|> (config media_placeholder_token_id). vLLM's kimi_k3 prompt updates build the whole <|media_begin|>image {w}x{h}<|media_content|>…<|media_end|> block from that single token, so a worker can expand it itself.
• GLM-5.3-Flash, image. The router's anchor is <|image|> (config image_token_id), the single token vLLM's GLM processors replace with the image's placeholder run.

GLM-5.3-Flash video is deliberately not opted in, for two independent reasons: vLLM's GLM family targets the three-token <|begin_of_video|><|video|><|end_of_video|> block for video, not a lone token, so a single anchor would never be replaced; and placeholder_token_id_for(Video) in this spec returns the image id (the router builds the <|begin_of_image|>… frame blocks itself), so even the anchor id would be wrong for worker expansion. Enabling video here needs a change to how the router emits video anchors and is out of scope. Kimi-K3 declares images only.

Changes

• crates/multimodal/src/registry/kimi_k3.rs: worker_expandable returns true for Modality::Image, plus a unit test.
• crates/multimodal/src/registry/glm53_flash.rs: same, image only, plus a unit test.
• crates/multimodal/scripts/check_worker_anchor_parity.py: kimi_k3 and glm53_flash entries (config_id anchors on media_placeholder_token_id / image_token_id; no video entry for GLM).

Test Plan

• cargo test -p llm-multimodal kimi_k3 (21 passed) and cargo test -p llm-multimodal glm53_flash (14 passed): the new tests image_anchor_can_be_expanded_by_the_worker and only_the_image_anchor_can_be_expanded_by_the_worker fail on main and pass here.
• cargo test -p llm-multimodal: 465 passed, 0 failed.
• cargo test --workspace: 136 suites, 6000 passed, 0 failed, 43 ignored.
• cargo clippy --workspace --all-targets -- -D warnings clean (run without --all-features: this machine has no OpenCV for the opencv-video feature); cargo +nightly fmt --all clean; pre-commit hooks (codespell, whitespace, ruff, ruff-format) pass.
• The parity script was not run. It needs a vLLM install and no vLLM build is available for this machine (macOS arm64). Expected when the reviewer runs python crates/multimodal/scripts/check_worker_anchor_parity.py --model kimi_k3 --trust-remote-code and --model glm53_flash --trust-remote-code: for each model the lone anchor becomes exactly one placeholder range and the surrounding tokens are unchanged. If it reports that vLLM does not expand a lone anchor for either model, that model's opt-in should be dropped rather than forced.
• Not verified here (no cluster access): one image_url request per model reporting smg_mm_processing_total{model=…,mode="worker",reason="auto_uniform"} with the same usage.prompt_tokens as router mode.

Checklist

• cargo +nightly fmt passes
• cargo clippy --all-targets -- -D warnings passes (without --all-features, see above)
• (Optional) Documentation updated
• (Optional) Please join us on Slack #sig-smg to discuss, review, and merge PRs

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: e06cc009-580d-4e32-bb5d-87c087c628c3

📥 Commits

Reviewing files that changed from the base of the PR and between f45d339 and 770bd94.

📒 Files selected for processing (3)

• crates/multimodal/scripts/check_worker_anchor_parity.py
• crates/multimodal/src/registry/glm53_flash.rs
• crates/multimodal/src/registry/kimi_k3.rs

Included review availability: 9 reviews are currently available. Your included PR review attempts over the past 7 days set your current allowance at 10 reviews per hour.

---

📝 Summary

Summary by CodeRabbit

• New Features
  • Added multimodal processing support for Kimi K3 and GLM-5.3 Flash models.
  • Image inputs are now correctly recognized and expanded for both models.
  • Kimi K3 and GLM-5.3 Flash continue to restrict expanded media handling to images; video and audio inputs are not expanded.
• Tests
  • Added coverage validating modality-specific behavior for images, video, and audio.
  • Added consistency checks to ensure worker processing matches each model’s supported media behavior.

Walkthrough

Changes

Worker Anchor Parity

Layer / File(s)	Summary
Model expansion rules crates/multimodal/src/registry/kimi_k3.rs, crates/multimodal/src/registry/glm53_flash.rs	Kimi K3 and GLM-5.3 Flash now allow worker expansion for image anchors only. Tests cover image, video, and audio modalities.
Anchor parity checks crates/multimodal/scripts/check_worker_anchor_parity.py	The parity checker adds image anchor rules for both models. GLM-5.3 Flash has no video rule because its video target is a three-token block.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Feature

Merge Risk: ⚪ Minimal · up to 770bd

The PR adds image-anchor support for both models while preserving the intended router-side video behavior; no merge-blocking risk remains.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 63.64% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 11 functions across 3 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: enabling worker-side expansion for Kimi-K3 and GLM-5.3-Flash image anchors.
Description check	✅ Passed	The description directly explains the problem, solution, model-specific behavior, tests, and known validation limits for the changeset.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

_{Comment @coderabbitai help to get the list of available commands.}

[claude]

Reviewed the full diff. No issues found — 0 🔴 Important, 0 🟡 Nit, 0 🟣 Pre-existing.

• Both worker_expandable overrides follow the established opt-in pattern (qwen3_vl, llama4, minimax_m3 in #2609) and are covered by unit tests asserting the exact modality set.
• The parity-script anchor rules mirror the Rust specs' derivation (media_placeholder_token_id for kimi_k3 via placeholder_token_id, image_token_id for glm53_flash via placeholder_token_id_for(Image)).
• Checked the mixed-modality path: ensure_worker_expandable requires every modality in the plan to be expandable, so a GLM image+video request falls back to router-side processing as a whole — leaving video opted out is safe, and is the right call given placeholder_token_id_for(Video) returns the image id.
• No e2e lane hardcodes worker_expandable expectations for these two models, so nothing flips.

The one open risk — whether vLLM really expands the lone anchors — is clearly disclosed in the Test Plan with a decision rule (drop the opt-in if the parity script fails), so nothing to flag there beyond a reminder to run check_worker_anchor_parity.py for both models before merging.