Skip to content

[AMD][DI][CI] nightly mi355x disagg: stage packed logs under RUNNER_TEMP, not /tmp - #43945

Merged
bingxche merged 1 commit into
sgl-project:mainfrom
Lzy17:ci/nightly-pack-logs-runner-temp
Oct 14, 2026
Merged

bingxche merged 1 commit into
sgl-project:mainfrom
Lzy17:ci/nightly-pack-logs-runner-temp

Conversation

@Lzy17

@Lzy17 Lzy17 commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

What

In the "Pack logs" step of nightly-amd-mi355x-disagg.yml, create the staging directory under $RUNNER_TEMP instead of the host's /tmp.

On pit2 the runner host (pit2-vm-amd-xl-02) has a 31 GB root disk shared with other users. On 2026-10-12 it was 100% full, so mktemp -d failed and the job was marked failed after the test had already passed (dsv4pro-fp4-1k1k-1p1d-dp8ep8, GSM8K 0.949). $RUNNER_TEMP is inside the runner's work directory on the shared home, so it does not depend on that disk. It falls back to /tmp if unset.

Validation

Not run end to end; this is a one-line change to a shell step. The failing command and the disk state were confirmed on the runner host.


CI States

Latest PR Test (Base): ✅ Run #38328926344
Latest PR Test (Extra): ❌ Run #38328926122
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

@michaelzhang-ai michaelzhang-ai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The 2n and 4n jobs share these steps through the *mi355x-steps anchor, so this one edit covers both.

Side benefit: the failing legs on 10-11 and 10-12 also print cp: preserving permissions for '/tmp/tmp.../server_exit_...': Operation not supported, which comes from cp -a going from the NFS log dir to local /tmp. Staging on the same filesystem as the logs should get rid of that.

Optional: the staging dir could be dropped entirely with tar czf "$LOG_BUNDLE" -C "$LOG_DIR" "${LOG_FILES[@]}". Fine to keep it if it's there to avoid tar's "file changed as we read it" error.

The full root disk on the runner host still needs to be cleaned up separately, since other jobs on that host will keep hitting it.

@michaelzhang-ai

michaelzhang-ai commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Verification run: https://github.com/sgl-project/sglang/actions/runs/38354542913
The run combines #43944, #43945, #43977 and #43985 on top of current main.

Status: passed.

"Pack logs" succeeded in all 5 legs ("Packed 14 log file(s)" each). The cp: preserving permissions ... Operation not supported warning from the old /tmp staging no longer appears.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants