Skip to content

[Fix] Re-encode multimodal embeddings after cache mismatch - #36595

Merged
mickqian merged 4 commits into
sgl-project:mainfrom
mickqian:codex/mm-cache-length-fix
Aug 27, 2026
Merged

mickqian merged 4 commits into
sgl-project:mainfrom
mickqian:codex/mm-cache-length-fix

Conversation

@mickqian

@mickqian mickqian commented Aug 27, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

based on #32404

A compact multimodal feature-hash collision can return a cached embedding whose token count does not match the placeholder span in the request. The mismatch currently reaches _adjust_embedding_length and aborts prefill with Insufficient multimodal embedding length.

Modifications

  • Validate cached embedding token counts against the item offsets on all chunked-prefill cache-hit paths.
  • Evict malformed entries and re-encode them instead of passing them to the scheduler.
  • Include (hash, token_count) in the per-batch temporary map so colliding items with different placeholder lengths are not deduplicated.
  • Add regression coverage for full, per-item, and batched cache paths.

Validation

  • pre-commit run --files python/sglang/srt/managers/mm_schedule.py test/registered/chunked_prefill/test_mm_chunked_embedding_unit.py
  • python3 -m compileall -q python/sglang/srt/managers/mm_schedule.py test/registered/chunked_prefill/test_mm_chunked_embedding_unit.py
  • Pytest was unavailable locally because PyTorch/pytest are not installed; NVIDIA CI is still required.

CI States

Latest PR Test (Base): ✅ Run #33052289112
Latest PR Test (Extra): ❌ Run #33052288823
Latest PR Test (AMD ROCm 7.2): ❌ Run #33052289142

@mickqian mickqian added run-ci CI: run the baseline test suite on this PR bypass-maintenance labels Aug 27, 2026
@mickqian

Copy link
Copy Markdown
Collaborator Author

Reproduction validation

Using the cached Qwen3-VL-2B-Instruct model on an isolated H200 devbox, I injected a one-token multimodal embedding on cache hit (diagnostic-only change, not part of this PR):

  • Before this fix: the real serving path logged Got 64 tokens in the text but 1 tokens from multimodal embeddings and terminated the scheduler with Insufficient multimodal embedding length.
  • With this fix: the same injected cache corruption was detected and re-encoded; two consecutive multimodal requests both returned HTTP 200, with no embedding-length error.

The injection was applied only in scratch checkouts; no diagnostic code is included in this PR.

@mickqian
mickqian merged commit 20a491d into sgl-project:main Aug 27, 2026
152 of 185 checks passed
vincentzed added a commit to vincentzed/sglang that referenced this pull request Sep 7, 2026
The wrong-length cache-hit guard landed in sgl-project#36595. Keep the LFM2-VL entry in the
encoder-materializes-features set and the HIP cross-request encode, now behind a
dedicated allowlist that keeps NPU and XPU on the per-request fallback. Add CPU
unit tests for the gate and the batched path.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KVPUFiK2bhzjErEnCMv26w
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bypass-maintenance run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant