[CI] fix the teardown output of disaggregation test by hnyls2002 · Pull Request #19193 · sgl-project/sglang

hnyls2002 · 2026-02-23T20:22:08Z

Fix popen_with_error_check to output LB logs directly to the terminal instead of piping and dumping on kill.
Suppress SIGKILL (-9) during teardown while still reporting real crashes.
Remove argument allow_exit

Before this PR:

After this PR:

gemini-code-assist · 2026-02-23T20:22:21Z

Summary of Changes

Hello @hnyls2002, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refines the popen_with_error_check utility, primarily to enhance the debugging experience and robustness of CI tests, particularly for disaggregation tests. By directing subprocess output to the terminal and gracefully handling SIGKILL signals, it aims to provide clearer insights into test failures and reduce spurious errors related to process termination.

Highlights

Direct Log Output: Modified popen_with_error_check to output subprocess logs directly to the terminal instead of capturing them, improving visibility during test execution.
SIGKILL Suppression: Implemented logic to suppress SIGKILL (exit code -9) errors during process teardown, preventing false failures in CI while still reporting genuine crashes.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

python/sglang/test/test_utils.py
- Updated subprocess.Popen to allow direct output to stdout and stderr by setting them to None.
- Replaced process.communicate() with process.wait() for simpler process waiting.
- Added a check to ignore exit code -9 (SIGKILL) to prevent exceptions during forced termination.
- Simplified the exception message for non-zero exit codes, removing stdout and stderr details.
- Configured the monitoring thread as a daemon to ensure it doesn't prevent program exit.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

hnyls2002 · 2026-02-23T20:22:46Z

/rerun-stage stage-c-test-8-gpu-h20

gemini-code-assist

Code Review

This pull request improves the popen_with_error_check utility for CI tests. The changes redirect subprocess output to the terminal for better visibility, correctly handle process termination with process.wait(), suppress exceptions for expected SIGKILL signals, and make the monitoring thread a daemon. These are all positive changes. However, I've identified a pre-existing logic bug in the exception-raising condition that this PR doesn't address. I've suggested a fix to ensure the allow_exit parameter behaves as intended.

python/sglang/test/test_utils.py

github-actions · 2026-02-23T20:23:34Z

✅ Triggered stage-c-test-8-gpu-h20 to run independently (skipping dependencies).

github-actions · 2026-02-23T20:23:56Z

🔗 View workflow run

hnyls2002 · 2026-02-23T20:28:34Z

/gemini review

github-actions · 2026-02-23T20:29:11Z

❌ Stage stage-c-test-8-gpu-h20 (again) doesn't support isolated runs yet.

NVIDIA stages:

stage-a-test-1
stage-a-cpu-only
stage-b-test-small-1-gpu
stage-b-test-large-1-gpu
stage-b-test-large-2-gpu
stage-b-test-4-gpu-b200
stage-c-test-4-gpu-h100
stage-c-test-8-gpu-h200
stage-c-test-8-gpu-h20
stage-c-test-4-gpu-b200
stage-c-test-4-gpu-gb200
stage-c-test-deepep-4-gpu
stage-c-test-deepep-8-gpu-h200
multimodal-gen-test-1-gpu
multimodal-gen-test-2-gpu

AMD stages:

sgl-kernel-unit-test-amd
sgl-kernel-unit-test-2-gpu-amd
stage-a-test-1-amd
stage-b-test-small-1-gpu-amd
stage-b-test-small-1-gpu-amd-mi35x
stage-b-test-large-1-gpu-amd
stage-b-test-large-2-gpu-amd
multimodal-gen-test-1-gpu-amd
multimodal-gen-test-2-gpu-amd
stage-c-test-large-8-gpu-amd
stage-c-test-large-8-gpu-amd-mi35x

Other stages will be added soon. For now, use /rerun-failed-ci for those stages.

github-actions · 2026-02-23T20:29:52Z

✅ Triggered stage-c-test-8-gpu-h20 to run independently (skipping dependencies).

github-actions · 2026-02-23T20:29:58Z

🔗 View workflow run

gemini-code-assist

Code Review

The changes improve the reliability of CI tests by preventing hangs during teardown and providing real-time logs. The use of a daemon thread for monitoring and the switch to process.wait() are positive improvements. However, the new logic now silently allows the monitored process to exit with code 0, which was previously treated as an error. This might hide cases where a server exits prematurely but cleanly.

python/sglang/test/test_utils.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

…o xverse_moe * 'xverse_moe' of https://github.com/xiaobaicxy/sglang: (275 commits) fix: add missing blank line after docstring in serving_transcription.py (sgl-project#19206) Whisper model support & `/v1/audio/transcriptions` endpoint & benchmark (sgl-project#16983) fix: patch docker image fixes (sgl-project#19100) [PD-Disagg] Unify prefill info data transition flow, all with `PrefillServerInfo` (sgl-project#19195) [CI] Tiny enhance the dp attention load blance benchmark (sgl-project#19194) add new ci user (sgl-project#19133) [CI] fix the teardown output of disaggregation test (sgl-project#19193) [PD-Disagg] Support query dp rank from bootstrap server. (sgl-project#19168) [Kernel Slimming] Migrate AWQ marlin repack kernel to JIT (sgl-project#18949) [Diffusion] Match rotary_embedding module name style (sgl-project#19179) [Refactor] Split rotary_embedding.py into a modular package (sgl-project#19144) [NPU] bump sgl-kernel-npu to 2026.02.01.post2 (sgl-project#19178) Use single mma warp group for short q_len in FA to optimize decoding performance (sgl-project#18985) Reorganize topk logic to clean up code and expose logical experts (sgl-project#16945) [ROCm] Use unreg path for custom all-reduce during CUDA graph capture (sgl-project#19162) [diffusion] feat: detect Flux2 custom VAE path from component_paths (sgl-project#19170) [AMD] ENV flags tuning and cleanup (sgl-project#19176) Fix bench_one_batch_server by moving the print statements (sgl-project#19175) Update rocm7.2 Dockerfile to install amdsmi for QuickReduce Initialization (sgl-project#19091) Revert "Refactor graph input buffers (sgl-project#18991)" (sgl-project#19173) ...

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

fix

79aac6a

gemini-code-assist bot reviewed Feb 23, 2026

View reviewed changes

python/sglang/test/test_utils.py Outdated Show resolved Hide resolved

fix allow_exit

3f1b044

gemini-code-assist bot reviewed Feb 23, 2026

View reviewed changes

python/sglang/test/test_utils.py Outdated Show resolved Hide resolved

python/sglang/test/test_utils.py Outdated Show resolved Hide resolved

hnyls2002 and others added 2 commits February 23, 2026 12:33

Update python/sglang/test/test_utils.py

c7eff7d

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

fix

d8b6ecf

hnyls2002 merged commit 2aa3fe3 into main Feb 23, 2026
53 of 61 checks passed

hnyls2002 deleted the lsyin/fix-teardown-of-pdlb branch February 23, 2026 20:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI] fix the teardown output of disaggregation test#19193

[CI] fix the teardown output of disaggregation test#19193
hnyls2002 merged 4 commits intomainfrom
lsyin/fix-teardown-of-pdlb

hnyls2002 commented Feb 23, 2026 •

edited

Loading

Uh oh!

gemini-code-assist bot commented Feb 23, 2026

Uh oh!

hnyls2002 commented Feb 23, 2026 •

edited

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

github-actions bot commented Feb 23, 2026

Uh oh!

github-actions bot commented Feb 23, 2026

Uh oh!

hnyls2002 commented Feb 23, 2026

Uh oh!

github-actions bot commented Feb 23, 2026

Uh oh!

github-actions bot commented Feb 23, 2026

Uh oh!

github-actions bot commented Feb 23, 2026

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

hnyls2002 commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot commented Feb 23, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

hnyls2002 commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

github-actions bot commented Feb 23, 2026

Uh oh!

github-actions bot commented Feb 23, 2026

Uh oh!

hnyls2002 commented Feb 23, 2026

Uh oh!

github-actions bot commented Feb 23, 2026

Uh oh!

github-actions bot commented Feb 23, 2026

Uh oh!

github-actions bot commented Feb 23, 2026

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

hnyls2002 commented Feb 23, 2026 •

edited

Loading

hnyls2002 commented Feb 23, 2026 •

edited

Loading