Skip to content

feat: unify multimodal feature transport - #30904

Merged
mickqian merged 3 commits into
sgl-project:mainfrom
mickqian:codex/mm-feature-transport-flag
Jul 15, 2026
Merged

mickqian merged 3 commits into
sgl-project:mainfrom
mickqian:codex/mm-feature-transport-flag

Conversation

@mickqian

@mickqian mickqian commented Jul 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add --mm-feature-transport={cpu,cuda_ipc} as the single NVIDIA CUDA IPC policy
  • map --keep-mm-feature-on-device and SGLANG_USE_CUDA_IPC_TRANSPORT to the new policy with deprecation warnings
  • reject CUDA IPC on non-NVIDIA and multi-node configurations
  • log the fixed pool reservation, base GPU, tokenizer-worker count, KV-cache headroom effect, and CPU fallback

Memory policy

CUDA IPC remains opt-in (cpu is the default). On H200 TP8 with Kimi-K2.7-Code, the existing 1 GiB pool is allocated immediately on GPU 0; Kimi's largest raw image feature is about 36.75 MiB, so that pool holds about 27 worst-case images. This fixed reservation reduces KV-cache headroom and should not become an unconditional default.

Dependency

This PR depends on #30902, which makes SGLANG_MM_FEATURE_CACHE_MB a hard aggregate budget across tokenizer workers. It should merge first.

Validation

  • py_compile for modified Python files
  • pre-commit hooks, including formatter and registered-test CI validation
  • focused unit-test collection is blocked locally before collection by an existing huggingface_hub==1.6.0 / @strict incompatibility; CI will run the tests in the pinned environment.

CI States

Latest PR Test (Base): ❌ Run #29379619765
Latest PR Test (Extra): ❌ Run #29379619656

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@mickqian

Copy link
Copy Markdown
Collaborator Author

/tag-and-rerun-ci

@mickqian
mickqian marked this pull request as ready for review July 14, 2026 14:39
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@mickqian
mickqian merged commit 947a14d into sgl-project:main Jul 15, 2026
134 of 147 checks passed
roopaksrivastav pushed a commit to roopaksrivastav/sglang that referenced this pull request Jul 16, 2026
Chronostasys pushed a commit to MindLab-Research/sglang that referenced this pull request Aug 24, 2026
jakki-amd pushed a commit to jakki-amd/sglang that referenced this pull request Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant