Skip to content

feat(mm-routing): support Kimi K3 image requests via llm-multimodal 1.9 - #13337

Closed
furionw wants to merge 1 commit into
mainfrom
qiwa/k3-mm-routing
Closed

furionw wants to merge 1 commit into
mainfrom
qiwa/k3-mm-routing

Conversation

@furionw

@furionw furionw commented Aug 16, 2026 •

Copy link
Copy Markdown
Contributor

Why

Kimi K3 multimodal KV routing needs the K3 vision spec added in llm-multimodal 1.9. Dynamo currently pins 1.7, so K3 falls back to text-prefix routing and cannot distinguish requests by image content.

Depends on #13059 for the vLLM 0.27.1 runtime support. Tracks DIS-2678.

What Change

  • Bump llm-multimodal to 1.9 and llm-tokenizer to 1.6, including compatible Tokio, Mio, and Velo pins.
  • Reproduce K3 worker routing tokens: media wrapper, original dimensions, and content-derived pad run.
  • Add frontend-to-vLLM-KV-event hash parity coverage and advertise K3 in the support matrix.

Test Plan

  • Two TP=8 workers; A/B share text and dimensions, differing only in image pixels.
  • Each request has 6,184 prompt tokens; 1,536-token KV blocks yield four full blocks plus a 40-token tail.
  • Cold A → worker 3236652464029061 at 0/5: none of five routing blocks overlapped.
  • Cold B → worker 5439765509436224 at 0/5: independently seed B's image-specific cache.
  • Warm A ×4 → A's worker at 4/5: four full blocks overlap, totaling 6,144 cached tokens.
  • Warm B ×4 → B's worker at 4/5: same cache reuse; the partial tail is not reusable.

Dependency

  • llm-multimodal 1.9 supplies Kimi-K3 support and requires llm-tokenizer 1.6 plus Tokio 1.52.3 or newer.
  • Dynamo and Velo 0.1 pin Tokio 1.48, while Tokio 1.52 requires Mio 1.2 or newer.
  • Tokio 1.52.3, Mio 1.2, and Velo 0.3 are therefore required resolver-compatible bumps, not unrelated upgrades.
  • The remaining lockfile changes are transitive updates from the multimodal, tokenizer, and consolidated Velo dependency graphs.

@github-actions github-actions Bot added feat documentation Improvements or additions to documentation frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` labels Aug 16, 2026
@github-actions

github-actions Bot commented Aug 16, 2026 •

Copy link
Copy Markdown
Contributor

@furionw
furionw force-pushed the qiwa/k3-mm-routing branch from 12644ad to bd93a80 Compare August 16, 2026 20:51
Signed-off-by: furionw <qiwa@nvidia.com>
@furionw
furionw force-pushed the qiwa/k3-mm-routing branch from bd93a80 to cb89c97 Compare August 16, 2026 20:53
@datadog-official

This comment has been minimized.

@furionw furionw changed the title feat(mm-routing): support Kimi K3 via llm-multimodal 1.9 feat(mm-routing): support Kimi K3 image requests via llm-multimodal 1.9 Aug 17, 2026
@furionw

furionw commented Aug 17, 2026

Copy link
Copy Markdown
Contributor Author

duplicate of #13371

@furionw furionw closed this Aug 17, 2026
@furionw
furionw deleted the qiwa/k3-mm-routing branch August 29, 2026 00:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation feat frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant