Repository navigation
feat(sglang): add image number limitation to prevent OOM (#8540) - #44078
Asthenia0412 wants to merge 1 commit into
Conversation
Add a per-request default image-count bound (default 5) that is enforced
at three entry points to prevent OOM when users submit dozens of 4K
images in a single request (~1 GiB GPU memory per image with the fast
image processor).
Changes:
- BaseMultimodalProcessor: add DEFAULT_IMAGE_NUM_LIMITATION = 5 class
constant that subclasses may override; enforce the bound in
validate_mm_data() as a defensive safeguard
- TokenizerManager._validate_mm_limits: when the user did not explicitly
set --limit-mm-data-per-request, default to {"image": 5} instead of
skipping all limits; user-provided values take precedence and an empty
dict disables all limits
- rust_server/config: mirror the default when building the Rust ingress
launch config so Rust-side requests get the same OOM protection
Operators can tune or disable the bound via --limit-mm-data-per-request,
e.g. --limit-mm-data-per-request '{"image": 10}' or '{}' to disable.
Ref: #8540
Signed-off-by: yancy <yancy@users.noreply.github.com>
|
@mickqian @JustinTong0323 @merrymercy @HaiShaw — Hi folks, #8540 asked for a per-request image-count bound to prevent OOM from 4K images (roughly 1 GiB each). This PR adds a default limit of 5 with override via the existing |
|
Same — needs 'run-ci' label to trigger CI tests. |
|
🤖 CI gating notice: This PR's checks ( |
|
Thanks for your interest in SGLang. This account opened 16 PRs in about eight hours, across unrelated subsystems (attention backends, speculative decoding, diffusion loading, scheduler IPC, KV-cache compression, docs), including two PRs for the same change (#44073 and #44117, both for #28808) and a fix for #43929 that is already on main. We can't review contributions at this volume, so we are closing the ones still open. If you want to pursue one, please pick a single change, verify it on a real run (a repro, a test, or numbers), and open a fresh PR. |
Summary
Fix #8540.
Adds a per-request image-count bound across the multimodal processor
pipeline. Each 4K image costs roughly 1 GiB GPU memory with the fast
image processor, so unbounded multi-image requests can easily OOM the
serving instance.
Changes
Three layers of validation to fail fast before any heavy image preprocessing:
config.py): the default{"image": 5}is propagated to the Rust side so early requests can berejected there too.
tokenizer_manager.py):_validate_mm_limits()now falls back toBaseMultimodalProcessor.DEFAULT_IMAGE_NUM_LIMITATION = 5instead ofskipping validation when no limit is configured.
base_processor.py): a defensive check invalidate_mm_data()that honours the ServerArg override if set, or theclass-level constant otherwise.
Operators can tune the bound or disable it:
Test Plan
/cc @merrymercy
DCO: Signed-off-by: yancy yancy@users.noreply.github.com
CI States
Latest PR Test (Base): ❌ Run #38414303599
Latest PR Test (Extra): ❌ Run #38414304356
Latest PR Test (AMD ROCm 10): ❌ Run #38414304566