Skip to content

[Fix] Preserve model runner contracts in prefill CUDA graphs - #35452

Merged
Oasis-Git merged 3 commits into
sgl-project:mainfrom
aurickq:aurick/prefill-cg-runner-contracts
Sep 20, 2026
Merged

Oasis-Git merged 3 commits into
sgl-project:mainfrom
aurickq:aurick/prefill-cg-runner-contracts

Conversation

@aurickq

@aurickq aurickq commented Aug 19, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

PrefillCudaGraphRunner constructs new ForwardBatch objects for capture and replay. Those objects currently bypass ModelRunner.prepare_dummy_forward_batch, even though model runners use that hook to prepare graph-specific batch state. The runner also rebuilds LogitsProcessorOutput while trimming graph padding but drops customized_info.

Modifications

  • Apply prepare_dummy_forward_batch to both capture-time and replay-time ForwardBatch objects.
  • Preserve customized_info when trimming full-prefill graph outputs.
  • Add focused tests for both contracts.

Accuracy Tests

No model math or kernels are changed.

  • 7 focused unit tests passed, including 2 parameterized subtests.

Speed Tests and Profiling

Not applicable. The hooks run during graph setup/replay preparation, and the output change only preserves existing metadata.

Checklist

  • Format code with the repository pre-commit configuration.
  • Add unit tests.
  • No documentation update is required.
  • No speed benchmark is required for this contract-preservation fix.
  • Follow the SGLang code style guidance.

CI States

Latest PR Test (Base): ❌ Run #35505146165
Latest PR Test (Extra): ❌ Run #35505146680
Latest PR Test (AMD ROCm 10): ❌ Run #35505145958

@aurickq

aurickq commented Aug 19, 2026

Copy link
Copy Markdown
Contributor Author

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Aug 19, 2026
@ispobock ispobock added bypass-fastfail run-ci-extra CI: also run the extra suite (requires run-ci) labels Aug 19, 2026
@ispobock

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

@Oasis-Git Oasis-Git left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@aurickq
aurickq force-pushed the aurick/prefill-cg-runner-contracts branch from fde967f to d8cc915 Compare August 24, 2026 19:49
@Oasis-Git

Copy link
Copy Markdown
Collaborator

@Oasis-Git

Copy link
Copy Markdown
Collaborator

@Oasis-Git

Copy link
Copy Markdown
Collaborator

/rerun-group cpu

@github-actions

github-actions Bot commented Aug 25, 2026 •

Copy link
Copy Markdown
Contributor

Results for /rerun-group cpu:

🚀 ubuntu-latest (34 tests): ❌ View workflow run

cd test/ && python3 registered/cpu/arm64/test_moe.py
cd test/ && python3 registered/cpu/quant/test_autoround.py
cd test/ && python3 registered/cpu/test_activation.py
cd test/ && python3 registered/cpu/test_binding.py
cd test/ && python3 registered/cpu/test_bmm.py
cd test/ && python3 registered/cpu/test_causal_conv1d.py
cd test/ && python3 registered/cpu/test_comm.py
cd test/ && python3 registered/cpu/test_cpu_graph.py
cd test/ && python3 registered/cpu/test_decode.py
cd test/ && python3 registered/cpu/test_extend.py
cd test/ && python3 registered/cpu/test_flash_attn.py
cd test/ && python3 registered/cpu/test_gemm.py
cd test/ && python3 registered/cpu/test_intel_amx_attention_backend_a.py
cd test/ && python3 registered/cpu/test_intel_amx_attention_backend_b.py
cd test/ && python3 registered/cpu/test_intel_amx_attention_backend_c.py
cd test/ && python3 registered/cpu/test_mamba.py
cd test/ && python3 registered/cpu/test_mla.py
cd test/ && python3 registered/cpu/test_moe.py
cd test/ && python3 registered/cpu/test_norm.py
cd test/ && python3 registered/cpu/test_qkv_proj_with_rope.py
cd test/ && python3 registered/cpu/test_qwen3.py
cd test/ && python3 registered/cpu/test_rank_consensus_checker.py
cd test/ && python3 registered/cpu/test_request_decompression.py
cd test/ && python3 registered/cpu/test_request_headers.py
cd test/ && python3 registered/cpu/test_rope.py
cd test/ && python3 registered/cpu/test_server_args_backend.py
cd test/ && python3 registered/cpu/test_shared_expert.py
cd test/ && python3 registered/cpu/test_spec_eagle_cpu.py
cd test/ && python3 registered/cpu/test_spec_eagle_parity_cpu.py
cd test/ && python3 registered/cpu/test_spec_eagle_topk_cpu.py
cd test/ && python3 registered/cpu/test_spec_kernels.py
cd test/ && python3 registered/cpu/test_store_cache.py
cd test/ && python3 registered/cpu/test_subblock_sparse_attention.py
cd test/ && python3 registered/cpu/test_topk.py

@Oasis-Git Oasis-Git removed bypass-fastfail run-ci-extra CI: also run the extra suite (requires run-ci) labels Aug 25, 2026
@Oasis-Git

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

1 similar comment
@Oasis-Git

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

@ispobock

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

@aurickq
aurickq force-pushed the aurick/prefill-cg-runner-contracts branch from 48aa615 to 0bea6e0 Compare September 18, 2026 22:11
@Oasis-Git

Copy link
Copy Markdown
Collaborator

@Oasis-Git
Oasis-Git merged commit d229952 into sgl-project:main Sep 20, 2026
197 of 230 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants