Skip to content

Fix MMMU VLM eval max_tokens for CoT prompt - #27327

Merged
mickqian merged 1 commit into
sgl-project:mainfrom
yhyang201:fix/mmmu-eval-max-tokens
Jun 5, 2026
Merged

mickqian merged 1 commit into
sgl-project:mainfrom
yhyang201:fix/mmmu-eval-max-tokens

Conversation

@yhyang201

@yhyang201 yhyang201 commented Jun 5, 2026

Copy link
Copy Markdown
Collaborator

Motivation

PR #21841 changed the MMMU eval prompt to use Chain-of-Thought instructions ("Think step by step before answering") but kept max_tokens=30. This is too short for models to complete their reasoning before outputting Answer: X, causing all VLM models to fail the nightly MMMU accuracy test since April 2.

Modifications

Increase max_tokens from 30 to 1024 in test/registered/eval/test_vlms_mmmu_eval.py.

Verification

Tested on H200 with all models in the nightly VLM MMMU eval:

Model Score (max_tokens=30) Score (max_tokens=1024) Threshold Status
Qwen2.5-VL-7B 0.28 0.56 0.33 PASS
MiMo-VL-7B-RL 0.30 0.49 0.28 PASS
deepseek-vl2-small 0.28 0.48 0.32 PASS
Qwen3-VL-30B-A3B 0.27 0.47 0.29 PASS
GLM-4.1V-9B-Thinking 0.29 0.47 0.28 PASS
GLM-4.5V-FP8 0.29 0.47 0.26 PASS
Qwen2-VL-7B 0.29 0.35 0.31 PASS
Janus-Pro-7B 0.30 0.33 0.285 PASS
InternVL2_5-2B 0.29 0.30 0.30 PASS

All models that can launch successfully now pass the accuracy threshold.


CI States

Latest PR Test (Base): ❌ Run #26991456983
Latest PR Test (Extra): ❌ Run #26991456867

PR sgl-project#21841 changed the MMMU eval prompt to use Chain-of-Thought
instructions ("Think step by step before answering") but kept
max_tokens=30, which is too short for models to complete their
reasoning before outputting "Answer: X". This caused all VLM
models to fail the nightly MMMU accuracy test since April 2.

Increase max_tokens from 30 to 1024 to give models enough space
to reason and produce the expected answer format.
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@github-actions github-actions Bot added the Multi-modal multi-modal language model label Jun 5, 2026
@mickqian
mickqian merged commit 46c58b5 into sgl-project:main Jun 5, 2026
87 of 96 checks passed
Chronostasys pushed a commit to MindLab-Research/sglang that referenced this pull request Aug 24, 2026
jakki-amd pushed a commit to jakki-amd/sglang that referenced this pull request Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Multi-modal multi-modal language model

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants