[AMD] Run MI355X disaggregation Nightly Test with runtime checkout code mechanism by yctseng0211 · Pull Request #30386 · sgl-project/sglang

yctseng0211 · 2026-07-07T10:49:47Z

Summary

Run the MI355X disagg nightly against the workflow checkout instead of image-baked sglang / sglang-router packages.
Stage the checkout on shared NFS, mount it read-only into Slurm containers, and reinstall checkout sglang for prefill/decode/bench.
Build and reinstall checkout sglang-router in the bench container before launching the router.
Add SGLANG_USE_CHECKOUT_RUNTIME=0 opt-out to use image-baked packages.
Clean per-run MI355X scratch state and avoid staging .git/config.
Support recipe-driven max_total_tokens; cap DSV4 Flash MI355X 1k/1k configs at 8551168.

Tests

Accuracy test: passed.

CI States

Latest PR Test (Base): ✅ Run #28873717185
Latest PR Test (Extra): ❌ Run #28873716286

gemini-code-assist

Code Review

This pull request introduces the SGLANG_USE_CHECKOUT_RUNTIME option to scripts/ci/slurm/launch_mi355x.sh, allowing the Slurm compute-node containers to reinstall and run the sglang package directly from the workflow's git checkout instead of using the image's baked-in version. The review feedback suggests optimizing the workspace staging by excluding the .git directory from the tarball to save disk space and transfer time. Additionally, it recommends using conditional parameter expansion when modifying PYTHONPATH to avoid trailing colons, which can unintentionally append the current working directory to Python's search path.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

gemini-code-assist · 2026-07-07T10:51:18Z

+    tar --exclude='__pycache__' --exclude='*.pyc' \
+        -C "$GITHUB_WORKSPACE" -cf - . | tar -C "$CHECKOUT_STAGE" -xf -


Staging the entire $GITHUB_WORKSPACE directory over shared NFS can be slow and consume significant disk space if the .git directory is large. Excluding the .git directory from the tar command is a safe optimization since the checkout SHA is already retrieved via git rev-parse and passed as an environment variable.

Suggested change

tar --exclude='__pycache__' --exclude='*.pyc' \

-C "$GITHUB_WORKSPACE" -cf - . | tar -C "$CHECKOUT_STAGE" -xf -

tar --exclude='__pycache__' --exclude='*.pyc' --exclude='.git' \

-C "$GITHUB_WORKSPACE" -cf - . | tar -C "$CHECKOUT_STAGE" -xf -

gemini-code-assist · 2026-07-07T10:51:18Z

+python3 -m pip install --no-deps --no-build-isolation -e "$RUNTIME_CHECKOUT/python"
+
+export RUNTIME_CHECKOUT
+export PYTHONPATH="$RUNTIME_CHECKOUT/python:${PYTHONPATH:-}"


Using :${PYTHONPATH:-} when PYTHONPATH is empty or unset results in a trailing colon (e.g., /path/to/dir:). In Python, an empty entry in PYTHONPATH (such as a trailing colon) is implicitly treated as the current working directory (.), which can lead to unexpected import behavior or security risks. Using the ${PYTHONPATH:+:$PYTHONPATH} parameter expansion cleanly appends the existing PYTHONPATH only if it is non-empty.

Suggested change

export PYTHONPATH="$RUNTIME_CHECKOUT/python:${PYTHONPATH:-}"

export PYTHONPATH="$RUNTIME_CHECKOUT/python${PYTHONPATH:+:$PYTHONPATH}"

gemini-code-assist · 2026-07-07T10:51:19Z

+source "\$CIDIR/model_flags.sh"
+bash "\$CIDIR/install_checkout_sglang.sh"
+if [[ "\${SGLANG_USE_CHECKOUT_RUNTIME:-1}" != "0" ]]; then
+  export PYTHONPATH=/tmp/sglang-checkout-runtime/python:\${PYTHONPATH:-}


Using :\${PYTHONPATH:-} when PYTHONPATH is empty or unset results in a trailing colon, which implicitly adds the current working directory to Python's search path. Use the \${PYTHONPATH:+:\$PYTHONPATH} parameter expansion to safely append the existing PYTHONPATH only if it is non-empty.

Suggested change

export PYTHONPATH=/tmp/sglang-checkout-runtime/python:\${PYTHONPATH:-}

export PYTHONPATH="/tmp/sglang-checkout-runtime/python\${PYTHONPATH:+:\$PYTHONPATH}"

gemini-code-assist · 2026-07-07T10:51:19Z

+source "\$CIDIR/model_flags.sh"
+bash "\$CIDIR/install_checkout_sglang.sh"
+if [[ "\${SGLANG_USE_CHECKOUT_RUNTIME:-1}" != "0" ]]; then
+  export PYTHONPATH=/tmp/sglang-checkout-runtime/python:\${PYTHONPATH:-}


Using :\${PYTHONPATH:-} when PYTHONPATH is empty or unset results in a trailing colon, which implicitly adds the current working directory to Python's search path. Use the \${PYTHONPATH:+:\$PYTHONPATH} parameter expansion to safely append the existing PYTHONPATH only if it is non-empty.

Suggested change

export PYTHONPATH=/tmp/sglang-checkout-runtime/python:\${PYTHONPATH:-}

export PYTHONPATH="/tmp/sglang-checkout-runtime/python\${PYTHONPATH:+:\$PYTHONPATH}"

gemini-code-assist · 2026-07-07T10:51:19Z

+    if [ "\${SGLANG_USE_CHECKOUT_RUNTIME:-1}" != "0" ]; then
+      export PYTHONPATH=/tmp/sglang-checkout-runtime/python:\${PYTHONPATH:-}
+    else
+      export PYTHONPATH=/sgl-workspace/sglang/python:\${PYTHONPATH:-}
+    fi


Using :\${PYTHONPATH:-} when PYTHONPATH is empty or unset results in a trailing colon, which implicitly adds the current working directory to Python's search path. Use the \${PYTHONPATH:+:\$PYTHONPATH} parameter expansion to safely append the existing PYTHONPATH only if it is non-empty.

Suggested change

if [ "\${SGLANG_USE_CHECKOUT_RUNTIME:-1}" != "0" ]; then

export PYTHONPATH=/tmp/sglang-checkout-runtime/python:\${PYTHONPATH:-}

else

export PYTHONPATH=/sgl-workspace/sglang/python:\${PYTHONPATH:-}

fi

if [ "\${SGLANG_USE_CHECKOUT_RUNTIME:-1}" != "0" ]; then

export PYTHONPATH="/tmp/sglang-checkout-runtime/python\${PYTHONPATH:+:\$PYTHONPATH}"

else

export PYTHONPATH="/sgl-workspace/sglang/python\${PYTHONPATH:+:\$PYTHONPATH}"

fi

bingxche · 2026-07-07T11:21:41Z

Dispatched the full MI355X disaggregation workflow on amd-disagg-checkout (configs left empty, so all MI355X disagg matrix entries should run): https://github.com/sgl-project/sglang/actions/runs/28862299872

yctseng0211 · 2026-07-07T13:56:12Z

https://github.com/sgl-project/sglang/actions/runs/28871753121

Co-authored-by: YC Yen-Ching Tseng <yctseng@amd.com>

yctseng0211 · 2026-07-07T14:25:35Z

https://github.com/sgl-project/sglang/actions/runs/28873810961

…de mechanism (sgl-project#30386) Co-authored-by: bingxche <bingxche@amd.com>

yctseng0211 requested review from Fridge003, HaiShaw, Kangyan-Zhou, bingxche, ispobock and merrymercy as code owners July 7, 2026 10:49

github-actions Bot added the amd label Jul 7, 2026

gemini-code-assist Bot reviewed Jul 7, 2026

View reviewed changes

bingxche and others added 4 commits July 7, 2026 09:23

ci: run MI355X disagg with checkout sglang

54c2e31

ci: clean MI355X disagg scratch after run

371ef5e

[AMD] Cap DSV4 Flash max_total_num_tokens (#30313)

1e0890e

Co-authored-by: YC Yen-Ching Tseng <yctseng@amd.com>

ci: run MI355X disagg router from checkout

0aac976

yctseng0211 force-pushed the amd-disagg-checkout branch from ce746e2 to 0aac976 Compare July 7, 2026 14:23

michaelzhang-ai mentioned this pull request Jul 7, 2026

[DO NOT MERGE][AMD][DI][CI] Kimi-K2.6 disagg diagnostic recipes for non-MTP GSM8K drop #30433

Closed

3 tasks

bingxche approved these changes Jul 8, 2026

View reviewed changes

bingxche merged commit 9bf122a into main Jul 8, 2026
120 of 127 checks passed

bingxche deleted the amd-disagg-checkout branch July 8, 2026 01:27

yctseng0211 mentioned this pull request Jul 8, 2026

[AMD] WIP - Amd/kimik26 disagg decodemeta #30543

Closed

5 tasks

Chronostasys pushed a commit to MindLab-Research/sglang that referenced this pull request Aug 24, 2026

[AMD] Run MI355X disaggregation Nightly Test with runtime checkout co…

15a7138

…de mechanism (sgl-project#30386) Co-authored-by: bingxche <bingxche@amd.com>

Navigation Menu

[AMD] Run MI355X disaggregation Nightly Test with runtime checkout code mechanism - #30386

[AMD] Run MI355X disaggregation Nightly Test with runtime checkout code mechanism#30386
bingxche merged 4 commits into
mainfrom
amd-disagg-checkout

yctseng0211 commented Jul 7, 2026 •

edited by bingxche

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Jul 7, 2026

Uh oh!

gemini-code-assist Bot Jul 7, 2026

Uh oh!

gemini-code-assist Bot Jul 7, 2026

Uh oh!

gemini-code-assist Bot Jul 7, 2026

Uh oh!

gemini-code-assist Bot Jul 7, 2026

Uh oh!

bingxche commented Jul 7, 2026

Uh oh!

yctseng0211 commented Jul 7, 2026

Uh oh!

yctseng0211 commented Jul 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		tar --exclude='__pycache__' --exclude='*.pyc' \
		-C "$GITHUB_WORKSPACE" -cf - . \| tar -C "$CHECKOUT_STAGE" -xf -

	export PYTHONPATH="$RUNTIME_CHECKOUT/python:${PYTHONPATH:-}"
	export PYTHONPATH="$RUNTIME_CHECKOUT/python${PYTHONPATH:+:$PYTHONPATH}"

	export PYTHONPATH=/tmp/sglang-checkout-runtime/python:\${PYTHONPATH:-}
	export PYTHONPATH="/tmp/sglang-checkout-runtime/python\${PYTHONPATH:+:\$PYTHONPATH}"

Conversation

yctseng0211 commented Jul 7, 2026 • edited by bingxche Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Tests

CI States

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

bingxche commented Jul 7, 2026

Uh oh!

yctseng0211 commented Jul 7, 2026

Uh oh!

yctseng0211 commented Jul 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yctseng0211 commented Jul 7, 2026 •

edited by bingxche

Loading