refactor(preprocessor)!: let workers declare their per-image token

[furionw]

Summary

Kimi-K3's prompt carries one placeholder per image, and the serving engines disagree on which token belongs there: vLLM builds the native media sequence from <|kimi_image_placeholder|>, while SGLang only repeats <|media_pad|> up to the feature count.

expands_image_pad_token encoded that as a mode flag naming how SGLang's processor works internally. The preprocessor used it to pick between two renderer-supplied strings and rewrite the rendered segments after the fact. The flag was only ever a proxy for the string, and the rewrite only existed because rendering ran before the engine's preference was consulted.

This carries the string instead. image_placeholder_token: Option<String> is read from the processor the worker actually loaded, so it states that engine's real contract rather than an assumption about it, and the formatter renders it directly. substitute_image_pad_token and swap_marker_segments are gone, and so is the notion of pad expansion.

Why this is safe for non-K3 models

SGLang now declares a token for every multimodal model it serves, not just K3. Containment is structural: the token is threaded only into kimi_k3_formatter_for, and no Jinja-templated family has a parameter to receive one. For Qwen3-VL, is_kimi_k3() returns false, execution falls through to PromptFormatter::from_parts, and the field is never read by any code path.

That is stronger than what it replaces. The old rewrite ran for every model on every request and was inert only because both trait accessors had to be Some and Jinja prompts carry no segments.

Workers that declare nothing are unchanged: vLLM keeps rendering the checkpoint default, and skip_serializing_if keeps them byte-identical on the wire.

Follow-up commit

Review found that allow_special: true on a rendered segment only selects tiktoken's special-aware encode method — it is not a membership check. An unregistered token falls through to ordinary BPE and silently shatters, so one image becomes several ids. The preprocessor now checks the declaration once at construction and warns. It sits in new_with_parts because the discovery path builds the formatter and preprocessor separately and never calls new.

Breaking changes

• ModelRuntimeConfig.expands_image_pad_token is replaced by ModelRuntimeConfig.image_placeholder_token.
• Rolling upgrade is a silent downgrade: ModelRuntimeConfig has no deny_unknown_fields, so an old worker publishing expands_image_pad_token at a new frontend deserializes to None and reverts to the default with no error.

Known limitations (pre-existing, not introduced here)

• Only the first-registered worker's declaration takes effect per model — mdcsum does not cover runtime_config, so a mixed vLLM+SGLang pool for one model is first-card-wins. Same shape as the flag it replaces; now recorded in the field doc.
• mm-routing's routing_image_token_id is still resolved from config.json and is deliberately not wired to the declared token. Verified behaviour-preserving: token ids are byte-identical before and after for both engines, and no family in the lightseek registry was ever affected. Wiring it needs a kimi_k3 registry entry and a fix to the mm_hashes gate first.

Test plan

• cargo test -p dynamo-llm --no-default-features --features mm-routing — preprocessor suite 15/15
• cargo clippy -p dynamo-llm --no-default-features --features mm-routing --all-targets -- -D warnings — clean
• cargo check on lib/bindings/python — clean
• pre-commit run on all changed files — clean
• New declared_token_does_not_change_a_jinja_rendered_prompt, the regression guard for the containment claim above
• New SGLang accessor tests: text-only worker, token read from the loaded processor, multiple spellings declined, missing mm_tokens logged distinctly, engine without a tokenizer manager
• Built and tested on macOS with --no-default-features; block-manager is Linux-only. The 9 ZMQ/event-plane lib-test failures in that environment were confirmed pre-existing by reproducing the identical set on the unmodified base. Linux CI is the real gate.

Merge order

Requires ai-dynamo/frontend-crates#155 to merge into feat/Kimi-K3 first. CI on this PR will be red until then — the branch calls kimi_k3_formatter_for with four arguments and the pinned renderer still takes three.

After that lands, cargo update -p dynamo-renderer is needed for both manifests — the root Cargo.toml and lib/bindings/python/Cargo.toml, which carries its own [patch.crates-io] block because the bindings crate is outside the workspace.

🤖 Generated with Claude Code

[devin-ai-integration]

Devin Review found 1 new potential issue.

[devin-ai-integration]

Devin Review found 1 new potential issue.

[datadog-official]

🎯 Code Coverage (details)
• Patch Coverage: 30.00%
• Overall Coverage: 32.10% (-3.79%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: d369e47 | Docs | Datadog PR Page | Give us feedback!}

[furionw]

Superseded by the smaller draft stack:\n\n- #12396 — native Kimi-K3 frontend (base)\n- #12394 — vLLM media-pad expansion (stacks on #12396)\n- #12393 — SGLang ServerArgs override compatibility (stacks on #12396)\n- #12395 — SGLang Kimi-K3 multimodal alignment (stacks on #12393)\n\nThis preserves the agreed lineage while allowing the vLLM and SGLang work to review and land independently.