[stmatengss]

Description

Improve nightly CI reliability by making unavailable hardware coverage explicit, bounding Docker build resource use, and exposing actionable TestPyPI upload errors. RFC #4751 records the observed failures, evidence, recovery plan, and longer-term decisions.

• Make MI350X integration opt-in through NIGHTLY_ROCM_INTEGRATION=true or the run_rocm_integration dispatch input. Record the choice once, report missing coverage, and accept only an intentional hardware skip. Enabled hardware tests and ROCm wheel builds/smoke tests remain required.
• Use the configured Buildx builder with bounded BuildKit and CMake/PyTorch concurrency, cache-only output, memory/disk diagnostics, and a 180-minute timeout. The full Dockerfile is still built.
• Check TestPyPI credentials early and use noninteractive, verbose Twine uploads. Add regression tests for workflow decisions and failure propagation.

TestPyPI storage cleanup, upload-limit changes, and ongoing retention remain owner-side actions described in the RFC. The native test compilation regression was already fixed by #4729. The RFC also documents overlap with #4450 and #2951 and the relationship to the broader roadmap in #3556.

Suggested Merge Priority

• P0 — Urgent
• P1 — High
• P2 — Normal
• P3 — Low

Module

• Transfer Engine (mooncake-transfer-engine)
• Mooncake Store (mooncake-store)
• Mooncake Conductor (mooncake-conductor)
• Reshard (mooncake-reshard)
• Mooncake EP (mooncake-ep)
• Mooncake PG (mooncake-pg)
• Integration (mooncake-integration)
• P2P Store (mooncake-p2p-store)
• Python Wheel (mooncake-wheel)
• Common (mooncake-common)
• Mooncake RL (mooncake-rl)
• CI/CD
• Docs
• Other

Type of Change

• Bug fix
• New feature
• Refactor
• Breaking change
• Documentation update
• Performance improvement
• Other

How Has This Been Tested?

Test commands:

python -m unittest discover -s scripts/ci/tests -v
python -m pytest -q mooncake-wheel/tests/test_testpypi_wheel_gate.py
actionlint -shellcheck='' .github/workflows/nightly.yml
pre-commit run --files .github/workflows/nightly.yml docker/mooncake.Dockerfile scripts/ci/tests/test_nightly_ci.py
git diff --check

Test results:

• Unit tests pass: 33 CI script tests, including 11 new nightly tests; 6 existing release-gate tests.
• Integration tests pass: the hosted Docker build succeeded; the complete nightly run is still pending. MI350X hardware execution and TestPyPI publishing have not been validated.
• Manual testing done: actionlint 1.7.12 passed; all 36 nightly shell steps passed bash -n; changed-file pre-commit passed.

The real hosted Docker build passed in about 32 minutes at b1da7c6cfd669fd17055d764f7824562e032224d. The complete nightly run is still pending and was dispatched with skip_publish=true.

The local Docker regression test uses a stub to check arguments, monitor cleanup, and exit-status preservation. No packages have been uploaded or deleted during this validation.

Checklist

• I have performed a self-review of my own code (human review pending; agent diff review completed)
• I have formatted my code using ./scripts/code_format.sh (not applicable: no C/C++ changes)
• I have run pre-commit on the files changed in this PR and all hooks pass
• I have updated the documentation (workflow comments and configuration instructions above)
• I have added tests to prove my changes are effective
• For changes >500 LOC: I have filed an RFC issue ([RFC]: Restore reliable nightly CI and TestPyPI publishing #4751; also records this smaller recovery change)

AI Assistance Disclosure

• No AI tools were used
• AI tools were used (specify below)

OpenAI Codex investigated workflow logs, implemented the changes and regression tests, ran local validation, and prepared the RFC. The human submitter remains responsible for reviewing every changed line and understanding and defending the change end-to-end.

[Copilot]

🔵 Needs a closer look

Full nightly and publishing validation remain incomplete, and reduced mandatory hardware coverage needs maintainer sign-off.

No open issues 🎉

What changed in this PR

Improves nightly CI reliability and publishing diagnostics as a focused recovery step under RFC #4751.

Changes:

• Makes MI350X hardware tests opt-in and reports missing coverage while preserving required ROCm wheel checks.
• Bounds Docker build concurrency and adds resource diagnostics.
• Adds early TestPyPI credential checks, verbose uploads, and regression tests.

File	Description
scripts/​ci/​tests/​test_nightly_ci.py	Tests gate decisions, credentials, and Docker failure propagation.
docker/​mooncake.Dockerfile	Limits CMake and PyTorch build concurrency.
.github/​workflows/​nightly.yml	Adds hardware selection, bounded builds, and publishing diagnostics.

_{Review effort: Balanced}

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[he-yufeng]

Read the full diff against the failure modes from last night's run. The three mechanisms all map cleanly:

1. The ROCm opt-in (repo variable or dispatch input, default off, warning into the step summary) is the right shape for the unassigned-runner failure: the leg simply does not exist until the hardware is provisioned, instead of sitting in the queue until the next nightly's concurrency group cancels the run. The gate-side accounting closes the loop — ALLOW_ROCM_SKIP is derived from the same version-stamp output that gates the job, so a skipped leg is excluded exactly when it was not requested, and a requested-but-failed leg still counts (the or chain keeps .value.result != "skipped" true for failures). I verified the jq logic against the four combinations.

2. The Docker job changes target the zero-failed-steps death from that run: serializing BuildKit workers (max-parallelism = 1), bounding both build systems (BUILD_JOBS → MAX_JOBS + cmake --parallel), and --output=type=cacheonly to skip the image-store copy. I checked the whole job on this head: nothing after the build step runs, pushes, or inspects the image, so cacheonly is safe — the job is purely a build validation, which matches the comment.

3. The TestPyPI credential pre-check plus twine --verbose turns Warehouse's storage-limit response body into a readable log line instead of a bare failure, which is what Nightly build/test failure - 2026-10-18 #4749-class triage needs.

scripts/ci/tests/test_nightly_ci.py exercising the gate's real YAML through subprocess is exactly the right pattern for workflow logic — the gate decisions are now testable without burning a nightly. One residual worth knowing (not a blocker): with the ROCm variable set but the runner still unprovisioned, the leg queues forever and the gate never settles; the variable's documentation says to set it only after provisioning, which covers it.