[HanFa]

Summary

Connect GLM-5.3-Flash image routing to the default Rust frontend through the shared glm5_next planner. Use checkpoint metadata for image token counts and validate the rendered prompt layout before constructing exact routing tokens. Unsupported configuration, accounting failures, and layout failures retain text-prefix routing.

Details

• Pin the optional dynamo-multimodal dependency to 0.2.2 with default features disabled; update both Cargo lockfiles.
• Select the GLM adapter before the legacy flat-config loader so nested processor_config.json checkpoints are supported.
• Delegate image counts and layout validation to the shared planner. Expand image placeholders using the existing repeated-pad path, retaining delimiters and the worker-bound prompt.
• Add Rust coverage for checkpoint-driven geometry, single/multiple images, layout parity, content-sensitive routing tokens, and fallback without partial routing/hash installation.
• Add real Python binding resolver tests using shared lightweight checkpoint fixtures, including nested/flat configs and unsupported or malformed metadata.
• Add tests/mm_router/test_router_rust_mm_glm5_next_e2e.py: nightly H100 E2E that drives the Rust frontend's KV router against a real vLLM or SGLang worker serving inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small pinned at a2d7ca22e78d (a 5 GB BF16 cut of GLM-5.3-Flash that keeps its tokenizer, chat template, processor config and token ids; the shared predownload fetches that revision and the workers load the snapshot from a local path). The engines size KV blocks for the model's hybrid KDA layers themselves, so the module reads the block size the worker registered from the frontend's dynamo_frontend_model_kv_cache_block_size metric and derives the image sizes and every block assertion from it. Per engine it checks worker expansion (router block total vs the engine's reported prompt length, per-image increments following GLM geometry), repeated-image cache overlap plus the engine's prefix-cache hit, different-image isolation at the router and in the engine cache, and two images whose runs straddle block boundaries, where swapping the second image must keep the first image's blocks. Exclude h100 tests from the nightly single-GPU vLLM lane the way the SGLang lane already does.
• tests/conftest.py: @pytest.mark.model(repo_id, revision=...) pins the shared predownload to that revision, so a test that loads a fixed revision finds it cached once HF_HUB_OFFLINE is set. The collected entry reads repo_id@revision, and download_models passes the revision to snapshot_download. Marks without revision= download the default branch as before.

Context

Follow-up to #16947, now merged into main. This PR contains the GLM integration commit, rebased onto upstream main at 4b487d890d, plus three test commits: 3d54251fae adds the E2E module requested in review, 97296c2c6 reworks it after its first GPU run (GB200, in review), which showed that the original checkpoint does not start and that the engines do not use 64-token KV blocks for this model, and da11fe54e addresses the review of that rework (pinned-revision predownload, a stricter two-image overlap check, a typed per-request observation). Relative to the previously reviewed revision (e3d865aba3), the only change to the integration commit is a one-line clippy fix in lib/llm/src/preprocessor/glm5_next_tests.rs: the & in CheckedFile::from_disk(&model_dir.join("config.json")) triggered clippy::needless_borrows_for_generic_args and failed the rust-clippy (.) pre-merge job. The previous run's lychee failure was a pre-existing 404 in the v1.5.0 release notes, fixed on main by #17091 and picked up by the rebase.

Supersedes the temporary stacked review at HanFa#3.

Where should the reviewer start?

Start with lib/llm/src/preprocessor/mm_routing/glm5_next.rs, then the adapter selection and layout-validation call sites. lib/llm/src/preprocessor/glm5_next_tests.rs exercises the preprocessing boundary; lib/bindings/python/tests/test_routing_image_token_id.py exercises the existing Rust resolver exposed to Python. tests/mm_router/test_router_rust_mm_glm5_next_e2e.py is the engine-backed check of the same contract.

Validation

• The branch is four commits on top of upstream main at 4b487d890d: the GLM integration commit (11 files; rebased without conflicts, the only change since the previously reviewed revision is the one-line clippy fix noted above), the E2E test commit (the test module plus a one-line nightly lane filter), the E2E rework (the test module only), and its review follow-up (the test module plus the revision= support in tests/conftest.py).
• On the rebased base: cargo fmt --all --check and git diff --check passed, and cargo metadata --locked --no-deps passed for both the workspace and the Python binding crate (both lockfiles pin dynamo-multimodal 0.2.2).
• On the rebased base: cargo clippy -p dynamo-llm --no-default-features --features mm-routing --all-targets --no-deps -- -D warnings passed (the lint target that failed in the previous pre-merge run).
• On the rebased base: cargo test -p dynamo-llm --no-default-features --features mm-routing --locked --lib preprocessor:: -- --test-threads=4: 405 passed, 2 ignored.
• Rebuilt the Python bindings on the rebased base with maturin develop --uv --features mm-routing --locked (Python 3.12, abi3 wheel), then ran python -m pytest lib/bindings/python/tests/test_routing_image_token_id.py -q: 11 passed.
• E2E module at 97296c2c6: pytest --collect-only collects 6 tests (3 per engine); the nightly vllm-h100-test / sglang-h100-test marker filters select 3 each, and the non-H100 nightly lane filters and the post_merge and pre_merge markers select none. The repository pre-commit hooks pass on it (isort, black, flake8, ruff, codespell, pytest marker report).
• A model of the router and engine caches (full-block hashing, image-content-sensitive keys, at most all but the last prompt token reused) passes every assertion in the module for worker block sizes from 64 to 7920 tokens and text lengths of 1 to 79 tokens. It fails the isolation assertions when image content is left out of the router's keys or the engine's cache keys.
• At da11fe54e: the pre-commit hooks pass, pytest --collect-only still collects 6 tests, and the collected download list carries inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small@a2d7ca22…. Against the Hub, predownloading the default branch and then resolving an older pinned commit (the repo's initial commit) offline fails with LocalEntryNotFoundError; predownloading the pinned revision resolves it offline. In the cache model above, the new two-image check passes all 1185 block-size and text-length cases. A router bug that keys each image by every image in the request, so that a new second image also changes the first image's blocks, fails only the new check, in all 1185 cases; the old bound caught it in 66.
• Review ran 3d54251fae on GB200: tiny-random/glm-5.3-flash did not start (vLLM rejected its sparse-MLA head size, and SGLang failed to load its block-FP8 weights). With a BF16 checkpoint both engines served, but they advertised 4352-token (vLLM) and 256-token (SGLang) KV blocks against the module's assumed 64.
• 97296c2c6 and da11fe54e have not run on a GPU yet: GLM5-Next's DSA layers need the sparse MLA attention that vLLM 0.32 and SGLang 0.5.22 implement only for Hopper and Blackwell, which rules out the 24 GiB Ada runners behind the post-merge lanes and my RTX 5090 (SM120). It carries no profiled_vram_gib marker yet and runs in the sequential GPU stage until profiled on an H100. The checkpoint is BF16, so the FP8 kernel pins are gone and both engines run their default kernels.
• The Rust and binding checks above used macOS without model weights; the E2E rework and its review follow-up were checked on Linux without a Hopper or Blackwell GPU.

The nightly E2E covers vLLM and SGLang worker expansion and KV-event hash parity with the small BF16 checkpoint. TensorRT-LLM coverage and the supported runtime/checkpoint matrix with the full model remain follow-up work; this PR does not claim full backend acceptance.

Related Issues

Part of #16948.

Internal tracking: DIS-3123 (multimodal-aware KV routing model support).

Consumes the shared planner from ai-dynamo/frontend-crates#412, released in dynamo-multimodal 0.2.2.

Summary by CodeRabbit

• New Features
  • Added image-routing support for GLM-5 Next models, using checkpoint settings to determine image token counts and routing information.
• Bug Fixes
  • Image prompts with unsupported layouts or invalid configuration no longer receive exact routing information, helping prevent incorrect routing.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi HanFa! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

[coderabbitai]

No actionable comments were generated in the recent review.

Walkthrough

The change adds a GLM5-Next image-routing backend that counts image tokens and validates prompt layouts. Preprocessing uses validation results when building multimodal routing information. Tests cover configuration resolution, routing output, and invalid inputs.

Changes

GLM5-Next Image Routing

Layer / File(s)	Summary
Routing configuration and backend lib/llm/Cargo.toml, lib/llm/tests/data/glm5-next-routing/*, lib/llm/src/preprocessor/mm_routing/{mod.rs,glm5_next.rs}, lib/bindings/python/tests/test_routing_image_token_id.py	Adds GLM5-Next model and processor settings, enables the dynamo-multimodal dependency, and implements checkpoint-based image-token counting and prompt validation. Tests cover token-ID resolution, token counts, and invalid configurations.
Preprocessor validation and routing behavior lib/llm/src/preprocessor.rs, lib/llm/src/preprocessor/mm_routing/image.rs, lib/llm/src/preprocessor/glm5_next_tests.rs	Adds a prompt-validation hook and calls it before constructing multimodal routing information. Tests cover routing output, validation failures, and image gathering behavior.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Merge Risk: Minimal · up to e3d86

No identified issue blocks merging after normal checks.

Pre-merge checks: 4

Check name	Status	Explanation
Title check		The title clearly identifies the main change: integrating the GLM-5.3-Flash image planner into multimodal routing.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Description check		The description is detailed and substantively complete. It has a Summary and Details, Context, Where to start, and an extensive Validation section with commands and results. It does not follow the rep…

Full details: Description check

Explanation

The description is detailed and substantively complete. It has a Summary and Details, Context, Where to start, and an extensive Validation section with commands and results. It does not follow the repository template headings exactly. It has no 'Overview', 'Key Changes', or 'Issues and Ongoing Work' headings, and it has no 'Closes #XXXX' line. It also leaves the 'I used Dynamo's /pr-checklist skill' checkbox absent. Related issue links are given as 'Part of' and internal tracking. The content maps onto the template sections, and the template's 'Closes' line is advisory per the template comment. Because the description is mostly complete, the check passes, with a note on the template mismatch.

Finishing Touches

Fix failing CI checks

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Tip

🎯 Customize your risk assessments

Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.

✅ Automatic approvals, with your rules

Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.

ℹ️ Recent review info

⚙️ Run configuration

• Source mode: Auto
• Repository: ai-dynamo/dynamo
• Branch: main
• File: .coderabbit.yaml
• Commit: cf620cd
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 735c8124-76ce-41d7-8160-69a667e04328

📥 Commits

Reviewing files that changed from the base of the PR and between cf620cd and e3d865a.

⛔ Files ignored due to path filters: 2

• Cargo.lock is excluded by !**/*.lock
• lib/bindings/python/Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing: 9

• lib/bindings/python/tests/test_routing_image_token_id.py
• lib/llm/Cargo.toml
• lib/llm/src/preprocessor.rs
• lib/llm/src/preprocessor/glm5_next_tests.rs
• lib/llm/src/preprocessor/mm_routing/glm5_next.rs
• lib/llm/src/preprocessor/mm_routing/image.rs
• lib/llm/src/preprocessor/mm_routing/mod.rs
• lib/llm/tests/data/glm5-next-routing/config.json
• lib/llm/tests/data/glm5-next-routing/processor_config.json

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

[HanFa]

/ok to test e15298f

[furionw]

/ok to test e15298f

[devin-ai-integration]

⏹️ Dynamo PR CI cancelled — run 41936721700 (attempt 1) on da11fe54eb

Gate checks: ✅ backend-status-check · ❌ deploy-status-check · ✅ dynamo-status-check

Framework	Deploy
vLLM	⏹️ 1 ✅ 3

Failure details

No jobs failed. 1 job was cancelled: vllm Deploy Test / agg. It does not look like a fail-fast cancel, because nothing else failed. Its logs show worker pods stuck in a SyncError loop for about 25 minutes before the cancel, so either the job timed out or someone cancelled it by hand.

⏹️ vllm Deploy Test / agg: cancelled while worker pods were stuck in SyncError

Job: vllm Deploy Test / agg · Cancelled step: Run Dynamo Deploy Test · Logs: gh run view --job 126961251373 -R ai-dynamo/dynamo --log-failed

[TEST] 2026-11-05T00:48:10 INFO ManagedDeployment:     vllm-agg-router-0-worker-crc7j: SyncError - Error syncing: patch host object: update object: Operation cannot be fulfilled on pods "vllm-agg-router-0-worker-crc7j-x-default-x-ci-41936721700"
[TEST] 2026-11-05T00:48:10 INFO ManagedDeployment:     vllm-agg-router-0-worker-skqg8: SyncError - Error syncing: patch host object: update object: Operation cannot be fulfilled on pods "vllm-agg-router-0-worker-skqg8-x-default-x-ci-41936721700"
##[error]The operation was canceled.

The deploy step ran from 00:23:52 to 00:48:49 UTC. Throughout, the vllm-agg-router worker pods logged SyncError ... Operation cannot be fulfilled on pods on a loop until the step was cancelled. This looks like a K8s pod sync or scheduling stall, so rerunning the failed jobs is likely enough.

For agents

{"pr": 17121, "run_id": 41936721700, "run_attempt": 1, "head_sha": "da11fe54ebd0ffa375baa8061414443700caee03",
 "failures": [{"job": "vllm Deploy Test / agg", "job_id": 126961251373, "failed_step": "Run Dynamo Deploy Test", "signature": "cancelled after ~25 min; worker pods SyncError 'Operation cannot be fulfilled on pods'", "tests": [], "log_cmd": "gh run view --job 126961251373 -R ai-dynamo/dynamo --log-failed"}]}

_{Posted automatically by Devin for run 41936721700. Updated on every full-CI run of this PR.}

[devin-ai-integration]

External PR triage

Blocker: CI is green on the current head, but none of the requested CODEOWNER teams has reviewed since 2026-10-22 (6 days). Only the review bot has approved.

Next action (@ai-dynamo/dynamo-frontend-codeowners @ai-dynamo/dynamo-multimodal-codeowners @ai-dynamo/dynamo-router-codeowners @ai-dynamo/dynamo-runtime-codeowners @ai-dynamo/dynamo-ops-codeowners): Review and approve, or request changes.

Status as of 2026-10-28 for head e15298f7ea. This comment is edited in place on later runs.

Written by Devin

[krishung5]

Overall lgtm!

[krishung5]

Could we add postmerge E2E coverage with the tiny GLM checkpoint for vLLM and SGLang? These tests verify frontend/planner agreement, but not worker expansion or KV-event hash parity. Please check repeated-image cache overlap, different-image isolation, and multiple images across block boundaries.

[HanFa]

I have added tests/mm_router/test_router_rust_mm_glm5_next_e2e.py, please note that they will be run in the nightly e2e since they requires sparse MLA attention which vLLM and sglang implements for only Blackwell and Hopper.

From my agent:

One caveat on "post-merge": GLM5-Next's DSA layers need sparse MLA attention, which vLLM 0.32 (FlashMLA sparse / FA3 / FlashInfer SM90) and SGLang 0.5.22 (NSA) implement only for Hopper and Blackwell, so even the tiny checkpoint cannot run on the 24 GiB Ada runners behind the post-merge lanes. I marked the module nightly + h100 so it runs in vllm-h100-test / sglang-h100-test, and added not h100 to the nightly single-GPU vLLM lane the way the SGLang lane already has it. If you want it post-merge, that needs an H100 post-merge lane; happy to follow up if the CI owners prefer that.

[HanFa]

meanwhile can I have anther ok to test?

[krishung5]

/ok to test 3d54251

[krishung5]

@HanFa thanks for addressing the comments!
The new E2E tests hit two issues when I ran them on GB200:

• The default tiny-random/glm-5.3-flash checkpoint fails during startup: vLLM rejects its sparse-MLA head dimensions, and SGLang encounters a BF16 → FP8 weight-loading error.
• With a compatible BF16 checkpoint, both backends serve requests, but the assertions assume a KV block size of 64. The workers actually advertise 4352 for vLLM and 256 for SGLang.

Could we switch to inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small, pin its revision, and derive the assertions from the worker’s effective block size? The prompts/images also need enough tokens to span multiple blocks so the tests exercise image-prefix reuse. The "resolved image-placeholder token id" assertion should be removed because GLM’s resolver path doesn’t emit that message.

[krishung5]

/ok to test 97296c2

[krishung5]

/ok to test da11fe5

[krishung5]

Validated on GB200 with the pinned small GLM checkpoint and one worker per backend:

• Original PR stack: vLLM 0.32.0 and SGLang 0.5.22 each passed all 3 E2E tests, confirming repeated-image cache overlap, different-image isolation, and multi-image expansion/reuse.
• Local merge with latest main: vLLM still passed 3/3. SGLang 0.5.23 failed the exact-routing assertions as expected: worker cache hits remained, but image-bearing router overlap was zero due to the wide-pad fallback in fix(mm-routing): handle SGLang engine-owned multimodal pad values #17126. Exact routing needs the upstream fix and corresponding Dynamo integration.