[Fix] DSv4.1: Fix BCG image handling and add multimodal bounded replay coverage by Oasis-Git · Pull Request #44131 · sgl-project/sglang

Oasis-Git · 2026-10-13T21:33:13Z

[codex/claude generated]

Motivation

DeepSeek-V4.1 multimodal prefill can produce incorrect results with breakable CUDA graphs (BCG). The eager model wrapper remaps image hash placeholders to model token IDs, but the captured transformer body kept reading the original IDs. Its Engram image-row mask was also omitted when capture used a dummy batch without multimodal inputs.

On four GB300 GPUs, the same 200 MMMU-Pro questions scored 76.5% with TP4+EP4 and 68.0% with BCG before this fix. BCG scored 78.0% after the fix; BCG with decoder bounded replay scored 78.5%.

Modifications

Refresh the captured body's static input IDs from the eager wrapper's prepared IDs, preserving the scheduler's original cache-key IDs.
Capture the DeepSeek-V4.1 Engram image-token mask independently of dummy-batch multimodal metadata.
Add CPU regressions for prepared image IDs and image-row Engram masking; both reproduce failures before the fix.
Replace MMLU with MMMU-Pro in the existing GB300 TP4+EP4 and DP4+EP4 DSPARK tests, and add TP4+EP4 with decoder SWA bounded replay inside BCG. Keep GSM8K and speculative acceptance checks in all three configurations.
Use sgl-eval 0.1.3 with the V4.1 preset. MMMU-Pro uses 200 examples, a 32768-token output cap, and a 0.75 threshold; GSM8K retains its 0.95 threshold.

Accuracy Tests

Local validation on four GB300 GPUs, using the same 200 MMMU-Pro questions, DSPARK block size 5, 64 client threads, 32 maximum running requests, memory fraction 0.7, and 8192-token prefill chunks:

Configuration	MMMU-Pro accuracy	Truncation rate
TP4+EP4, BCG disabled	76.5% (153/200)	4.0%
TP4+EP4 + BCG, before fix	68.0% (136/200)	6.0%
TP4+EP4 + BCG, after fix	78.0% (156/200)	4.5%
TP4+EP4 + BCG + decoder bounded replay, after fix	78.5% (157/200)	3.5%

All four evaluations completed with zero request errors. Each is one run at temperature 1.0, top-p 0.95, without a fixed sampling seed; these measurements are not a deterministic equivalence proof. Encoder bounded replay was disabled throughout. The first three configurations also disabled decoder bounded replay.

39 related unit tests and 45 subtests passed, including the new regressions and existing prefill-runner tests.
Applicable pre-commit hooks passed.
This validation used the PR base revision 961dd5f plus the fix. GSM8K and DP4 were not rerun locally after the fix; CI validation of the updated head remains pending.

Speed Tests and Profiling

No controlled performance benchmark or profiling was performed.

Checklist

Format code with pre-commit.
Add regression and model E2E coverage.
Validate the image-path fix locally on GB300.
Validate the updated head in CI.

CI States

Latest PR Test (Base): ❌ Run #38827264316
Latest PR Test (Extra): ❌ Run #38827264170
Latest PR Test (AMD ROCm 10): ❌ Run #38827264039

Oasis-Git · 2026-10-13T21:49:17Z

[Codex]: Run targeted GB300 E2E coverage with /rerun-test registered/e2e/models/test_deepseek_v41_flash_gb300.py: GSM8K and MMMU-Pro for the TP4+EP4, DP4+EP4, and bounded replay + breakable prefill graph configurations. Monitor the resulting run and inspect any failure before considering a rerun.

Oasis-Git · 2026-10-13T21:49:21Z

/rerun-test registered/e2e/models/test_deepseek_v41_flash_gb300.py

github-actions · 2026-10-13T21:49:53Z

🚀 4-gpu-gb300 (1 test): ❌ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v41_flash_gb300.py

Oasis-Git · 2026-10-13T23:20:47Z

[Codex]: Targeted GB300 E2E failed for 2f322de, attempt 1.

DP4+EP4 DSPARK GSM8K passed: 0.975 against the 0.95 threshold.
DP4+EP4 MMMU-Pro failed before evaluation with UnsupportedModelPresetError: the installed sgl-eval has no built-in preset for deepseek-ai/DeepSeek-V4.1-Flash. This is a test-configuration issue introduced by this PR, not an accuracy-threshold failure.
The suite stopped after these two tests; TP4+EP4 and bounded replay + BCG coverage did not run.

Job: https://github.com/sgl-project/sglang/actions/runs/38473499499/job/115558474269

No rerun was triggered. The MMMU-Pro evaluation configuration must be corrected before rerunning.

[codex/claude generated] ## Summary Add the exact `deepseek-ai/DeepSeek-V4.1-Flash` model ID to the built-in preset registry. SGLang's MMMU-Pro E2E in sgl-project/sglang#44131 currently fails before evaluation because this ID is unsupported. The preset uses temperature 1.0, top-p 0.95, thinking enabled, and reasoning effort high, matching the explicit settings used in the V4.1 MMMU-Pro evaluation driver. It leaves max_tokens unset so benchmark/server defaults apply; callers can set an explicit output budget. This does not claim a completed accuracy baseline or change existing model presets. Add a documented MMMU-Pro invocation and regression coverage for loading the preset, resolving generation settings, and overriding the output budget and reasoning settings. ## Validation - Preset tests: 31 passed. - Pre-commit checks. - No GPU evaluation performed for this registry change. SGLang CI must install a version containing this preset before its existing preset-based V4.1 E2E can use it; this PR alone does not update SGLang's pinned dependency.

…main

Oasis-Git · 2026-10-14T22:12:53Z

[Codex]: Merged current main and updated the test dependency to the published sgl-eval 0.1.3, which includes the V4.1-Flash preset. Run /rerun-test --changed on ceabcf6 to validate the modified GB300 E2E file with GSM8K and MMMU-Pro across all three configurations.

Oasis-Git · 2026-10-14T22:12:56Z

/rerun-test --changed

github-actions · 2026-10-14T22:13:23Z

🚀 4-gpu-gb300 (1 test): ❌ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v41_flash_gb300.py

Oasis-Git · 2026-10-14T23:25:02Z

[Codex]: Changed-test GB300 CI failed for ceabcf6, attempt 1: the Run test step reached its 60-minute timeout.

Verified checkout ceabcf6 and installed sgl-eval 0.1.3; the missing V4.1 preset issue is resolved.

Configuration	GSM8K	MMMU-Pro
DP4+EP4 DSPARK	0.980, passed	0.780, passed
TP4+EP4 DSPARK	0.975, passed	0.785, passed
TP4+EP4 DSPARK + bounded replay + BCG	0.975, passed	Incomplete: 198/200 at timeout

The incomplete bounded-replay MMMU-Pro run displayed 70.20% on 198 completed responses; this is not a final accuracy result. Even both remaining answers being correct would not reach the 0.75 gate given those completed results. Two requests were still generating tokens when the timeout occurred. This run therefore needs both accuracy investigation and evaluation-budget review, not just a longer timeout. No automatic rerun was triggered.

Job: https://github.com/sgl-project/sglang/actions/runs/38646117531/job/116133434347

Oasis-Git · 2026-10-15T00:38:50Z

[Codex]: Rerun changed-test CI on 1944a21 with /rerun-test --changed. All three GB300 configurations now cap MMMU-Pro output at 32,768 tokens per question; the 0.75 accuracy threshold is unchanged.

Oasis-Git · 2026-10-15T00:38:54Z

/rerun-test --changed

github-actions · 2026-10-15T00:39:27Z

🚀 4-gpu-gb300 (1 test): ❌ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v41_flash_gb300.py

Oasis-Git · 2026-10-15T02:16:13Z

[Codex]: Changed-test CI failed for 1944a21, attempt 1, before any model evaluation.

The token-cap edit incorrectly put the function _run_accuracy_eval in the base-class list of DSV41FlashAccuracyMixin, causing TypeError: metaclass conflict while importing the test module. This is a regression in my test edit; the class should inherit only from GSM8KMixin, with _run_accuracy_eval used inside the MMMU-Pro method.

Checkout 1944a21 and sgl-eval 0.1.3 were verified. No GSM8K/MMMU-Pro scores or truncation rates were produced in this attempt. No automatic rerun was triggered.

Job: https://github.com/sgl-project/sglang/actions/runs/38661276098/job/116182966612

Oasis-Git · 2026-10-15T04:11:24Z

[Codex]: Rerun changed-test CI on 961dd5f after fixing the accuracy mixin inheritance. Local class-construction, test-discovery, and MMMU-Pro argument-forwarding checks passed for all three configurations; the 32,768-token cap and 0.75 accuracy threshold remain unchanged.

Oasis-Git · 2026-10-15T04:11:28Z

/rerun-test --changed

github-actions · 2026-10-15T04:11:51Z

🚀 4-gpu-gb300 (1 test): ❌ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v41_flash_gb300.py

Oasis-Git · 2026-10-15T05:04:09Z

[Codex]: Targeted GB300 E2E failed on commit 961dd5f3418a75eb29b5c289608dcbf7cb8374a3 (run, attempt 1, GPU job). Checkout and installed sgl-eval 0.1.3 are verified in the logs.

Configuration	GSM8K	MMMU-Pro	MMMU-Pro truncation
DP4 + EP4	98%	77.5%	5%
TP4 + EP4	97.5%	76.5%	4.5%
TP4 + EP4 + bounded replay + BCG	97.5%	70%; built-in retry 65%	5.5%; retry 6.5%

The bounded-replay + BCG MMMU-Pro test failed the unchanged 75% threshold. All other tests passed. The 32768-token cap allowed evaluation to finish: six tests completed in 2563 seconds, with no evaluation request errors. This is an accuracy assertion failure, not the previous import error or timeout. The results identify the failing configuration but do not isolate bounded replay from BCG or its other configuration differences. No additional rerun was triggered.

Oasis-Git · 2026-10-15T19:51:45Z

[Codex]: Rerun changed-test CI on 786470d after fixing multimodal token IDs and Engram image masking in prefill graphs. This selects the GB300 GSM8K/MMMU-Pro E2E and the new CPU regression tests. Local fixed-BCG MMMU-Pro scored 78.0%, and fixed BCG with decoder bounded replay scored 78.5%; the 0.75 threshold and 32768-token cap are unchanged.

Oasis-Git · 2026-10-15T19:51:46Z

/rerun-test --changed

github-actions · 2026-10-15T19:52:14Z

🚀 4-gpu-gb300 (1 test): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v41_flash_gb300.py

🚀 ubuntu-latest (1 test): ✅ View workflow run

cd test/ && python3 registered/unit/models/test_deepseek_v41_bcg_multimodal.py

Oasis-Git · 2026-10-15T22:33:13Z

[Codex]: Changed-test CI passed on 786470debece0e684adbd3c074d148430c90c2ec (attempt 1; checkout verified in both logs).

GB300 E2E: all six tests passed in 2262 seconds.
CPU image-path regressions: both tests passed.

Configuration	GSM8K	MMMU-Pro	MMMU-Pro truncation
DP4 + EP4	97.5%	76.0%	4%
TP4 + EP4	97.0%	77.0%	6%
TP4 + EP4 + bounded replay + BCG	97.5%	77.5%	3%

All evaluations reported zero request errors. The 75% MMMU-Pro threshold and 32768-token cap were unchanged. This confirms the requested targeted coverage, not full CI.

Oasis-Git · 2026-10-16T00:04:13Z

[Codex]: Rerun the targeted DeepSeek-V4.1 GB300 E2E test on the merged PR head 1a6b30b, as requested. This covers GSM8K and MMMU-Pro across TP4+EP4, DP4+EP4, bounded replay + BCG, and TRT-LLM bounded replay + BCG.

Oasis-Git · 2026-10-16T00:04:15Z

/rerun-test registered/e2e/models/test_deepseek_v41_flash_gb300.py

github-actions · 2026-10-16T00:04:44Z

🚀 4-gpu-gb300 (1 test): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v41_flash_gb300.py

[DSv4.1] Cover bounded replay BCG with GSM8K and MMMU-Pro

2f322de

github-actions Bot added the deepseek label Oct 13, 2026

Oasis-Git mentioned this pull request Oct 13, 2026

[DSv4.1] Add Flash model generation preset sgl-project/sgl-eval#58

Merged

Oasis-Git added 2 commits October 14, 2026 15:12

Merge remote-tracking branch 'upstream/main' into work/pr44131-merge-…

ceb5e34

…main

[DSv4.1] Use sgl-eval 0.1.3 for the Flash model preset

ceabcf6

Oasis-Git requested review from Fridge003, ispobock and merrymercy as code owners October 14, 2026 22:12

github-actions Bot added the dependencies Pull requests that update a dependency file label Oct 14, 2026

[DSv4.1] Cap MMMU-Pro output at 32768 tokens in GB300 CI

1944a21

[DSv4.1] Fix accuracy mixin inheritance

961dd5f

[DSv4.1] Fix multimodal token IDs and Engram masking in prefill graphs

64c46d0

Oasis-Git requested a review from Ying1123 as a code owner October 15, 2026 19:48

Oasis-Git requested a review from hnyls2002 as a code owner October 15, 2026 19:48

Oasis-Git changed the title ~~[DSv4.1] Cover bounded replay BCG with GSM8K and MMMU-Pro~~ [DSv4.1] Fix BCG image handling and add multimodal bounded replay coverage Oct 15, 2026

github-actions Bot added the Multi-modal multi-modal language model label Oct 15, 2026

[DSv4.1] Remove explanatory comments from BCG image fix

786470d

Oasis-Git added 6 commits October 15, 2026 15:40

Merge main and preserve both GB300 bounded replay test configurations

09ad5e6

[DSv4.1] Simplify prepared token ID copy

9a04c9f

Limit prepared input ID synchronization to DeepSeek V4.1

53d5912

Initialize DeepSeek V4.1 flag in prefill graph runner

bec7847

Simplify prepared input ID argument lookup

c4eae37

Merge branch 'main' into test/dsv41-gb300-mmmu-pro-bounded-replay

1a6b30b

Oasis-Git changed the title ~~[DSv4.1] Fix BCG image handling and add multimodal bounded replay coverage~~ [Fix] DSv4.1: Fix BCG image handling and add multimodal bounded replay coverage Oct 16, 2026

Oasis-Git merged commit 0693457 into sgl-project:main Oct 16, 2026
114 of 131 checks passed

bingxche mentioned this pull request Oct 18, 2026

[CI Monitor] Daily Report - 2026-10-18 bingxche/sglang-ci-bot#230

Open

Conversation

Oasis-Git commented Oct 13, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

CI States

Uh oh!

Oasis-Git commented Oct 13, 2026

Uh oh!

Oasis-Git commented Oct 13, 2026

Uh oh!

github-actions Bot commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Oasis-Git commented Oct 13, 2026

Uh oh!

Oasis-Git commented Oct 14, 2026

Uh oh!

Oasis-Git commented Oct 14, 2026

Uh oh!

github-actions Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Oasis-Git commented Oct 14, 2026

Uh oh!

Oasis-Git commented Oct 15, 2026

Uh oh!

Oasis-Git commented Oct 15, 2026

Uh oh!

github-actions Bot commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Oasis-Git commented Oct 15, 2026

Uh oh!

Oasis-Git commented Oct 15, 2026

Uh oh!

Oasis-Git commented Oct 15, 2026

Uh oh!

github-actions Bot commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Oasis-Git commented Oct 15, 2026

Uh oh!

Oasis-Git commented Oct 15, 2026

Uh oh!

Oasis-Git commented Oct 15, 2026

Uh oh!

github-actions Bot commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Oasis-Git commented Oct 15, 2026

Uh oh!

Oasis-Git commented Oct 16, 2026

Uh oh!

Oasis-Git commented Oct 16, 2026

Uh oh!

github-actions Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Oasis-Git commented Oct 13, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Oct 13, 2026 •

edited

Loading

github-actions Bot commented Oct 14, 2026 •

edited

Loading

github-actions Bot commented Oct 15, 2026 •

edited

Loading

github-actions Bot commented Oct 15, 2026 •

edited

Loading

github-actions Bot commented Oct 15, 2026 •

edited

Loading

github-actions Bot commented Oct 16, 2026 •

edited

Loading