Repository navigation
[AMD][DI][CI] nightly mi355x disagg: stage packed logs under RUNNER_TEMP, not /tmp - #43945
Conversation
michaelzhang-ai
left a comment
There was a problem hiding this comment.
LGTM. The 2n and 4n jobs share these steps through the *mi355x-steps anchor, so this one edit covers both.
Side benefit: the failing legs on 10-11 and 10-12 also print cp: preserving permissions for '/tmp/tmp.../server_exit_...': Operation not supported, which comes from cp -a going from the NFS log dir to local /tmp. Staging on the same filesystem as the logs should get rid of that.
Optional: the staging dir could be dropped entirely with tar czf "$LOG_BUNDLE" -C "$LOG_DIR" "${LOG_FILES[@]}". Fine to keep it if it's there to avoid tar's "file changed as we read it" error.
The full root disk on the runner host still needs to be cleaned up separately, since other jobs on that host will keep hitting it.
|
Verification run: https://github.com/sgl-project/sglang/actions/runs/38354542913 Status: passed. "Pack logs" succeeded in all 5 legs ("Packed 14 log file(s)" each). The |
What
In the "Pack logs" step of
nightly-amd-mi355x-disagg.yml, create the staging directory under$RUNNER_TEMPinstead of the host's/tmp.On pit2 the runner host (
pit2-vm-amd-xl-02) has a 31 GB root disk shared with other users. On 2026-10-12 it was 100% full, somktemp -dfailed and the job was marked failed after the test had already passed (dsv4pro-fp4-1k1k-1p1d-dp8ep8, GSM8K 0.949).$RUNNER_TEMPis inside the runner's work directory on the shared home, so it does not depend on that disk. It falls back to/tmpif unset.Validation
Not run end to end; this is a one-line change to a shell step. The failing command and the disk state were confirmed on the runner host.
CI States
Latest PR Test (Base): ✅ Run #38328926344
Latest PR Test (Extra): ❌ Run #38328926122
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.