Skip to content

[VLM] fix bench_serving sglang backend to support image dataset #21294

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yhyang201 merged 2 commits into from
Mar 29, 2026
Merged

[VLM] fix bench_serving sglang backend to support image dataset #21294

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
671085c
fix
yhyang201 Mar 24, 2026
c6574ef
Merge branch 'main' into fix-benchserving-vlm
yhyang201 Mar 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 12 additions & 9 deletions python/sglang/benchmark/datasets/image.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,15 +148,18 @@ def create_mm_data_row(
# Vision tokens = total tokens - text tokens
vision_prompt_len = prompt_len - text_prompt_len

use_raw_prompt = backend in [
"sglang",
"sglang-oai",
"sglang-oai-chat",
"vllm",
"vllm-chat",
"lmdeploy",
"lmdeploy-chat",
]
supported_backends = ["sglang", "sglang-native", "sglang-oai-chat"]
if backend not in supported_backends:
raise ValueError(
f"Image dataset only supports backends: {supported_backends}, "
f"got '{backend}'."
)

# sglang-oai-chat: server's chat handler applies chat template, so send raw text.
# sglang/sglang-native: /generate does not apply chat template, so send prompt_str
# which contains image placeholder tokens needed by the multimodal processor.
use_raw_prompt = backend == "sglang-oai-chat"

return DatasetRow(
prompt=text_prompt if use_raw_prompt else prompt_str,
prompt_len=prompt_len,
Expand Down
Loading