[teamleaderleo]

A transient failure in the managed fleet compiler cache currently fails the nightly Release build after the initial CAS probe. This change retries once with Xcode's local cache only when the failed output identifies a compilation-cache or CAS error; unrelated compile failures remain red.

Leo said, “it feels like you guys always forget about incremental builds and the external SSDs.” The retry keeps the warm local/DerivedData path available when the shared cache is temporarily unavailable.

The regression test checks the workflow's runtime retry contract (PIPESTATUS, CAS-specific matching, local retry, and propagation of unrelated failures). Follow-up issue: #30105. The settings-alignment follow-up from the review is tracked in #30106.

Validation:

• python3 tests/test_ci_fleet_compilation_cache.py (7 tests)
• python3 scripts/cmux-next/prepush-gate.py

Review receipt: independent router review of #30061 at beb297ffb5f5a77c7c7f539476c3c9f85df42c84 identified this as a P1 follow-up.

Agent session: agent session
🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

---

Summary by cubic

Makes the nightly Release build resilient to a transient managed fleet compilation cache outage: when the build fails with a runtime CAS diagnostic, it retries once with Xcode's local cache. Unrelated compile failures stay red.

• The retry triggers only on compiler runtime cache failures (for example, "CAS error encountered during compilation", "CAS error: No such file or directory", "remote service unavailable"), not on source errors that merely mention CAS or cache.
• Adds a stricter cmux_next_runtime_cas_failed predicate and regression tests covering the retry contract (PIPESTATUS, local retry, unrelated failure propagation).

^{Written for commit 588c01a. Summary will update on new commits.}

[cubic-dev-ai]

2 issues found across 12 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tests/test_ci_fleet_compilation_cache.py">

<violation number="1" location="tests/test_ci_fleet_compilation_cache.py:103">
P2: This retry test only checks workflow source text, so it does not protect the runtime contract described by the PR. Execute the extracted shell with fake builds that cover a CAS failure, an unrelated failure, and the retry’s local-cache arguments; otherwise those regressions can ship while this test remains green.</violation>
</file>

<file name=".github/workflows/nightly.yml">

<violation number="1" location=".github/workflows/nightly.yml:2536">
P2: This matcher is broader than a cache-failure check and can retry unrelated compiler failures whenever the log contains routine cache diagnostics or an unrelated `CAS` occurrence. Match cache-specific error lines such as `CAS error:` or cache/remote/plugin messages containing `error` or `failed`, then keep unrelated failures on the single failing path.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic</sub>

[cubic-dev-ai]

P2: This retry test only checks workflow source text, so it does not protect the runtime contract described by the PR. Execute the extracted shell with fake builds that cover a CAS failure, an unrelated failure, and the retry’s local-cache arguments; otherwise those regressions can ship while this test remains green.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/test_ci_fleet_compilation_cache.py, line 103:

<comment>This retry test only checks workflow source text, so it does not protect the runtime contract described by the PR. Execute the extracted shell with fake builds that cover a CAS failure, an unrelated failure, and the retry’s local-cache arguments; otherwise those regressions can ship while this test remains green.</comment>

<file context>
@@ -0,0 +1,115 @@
+        self.assertIn("CODE_SIGNING_ALLOWED=NO", args)
+        self.assertEqual(args[-1], "build")
+
+    def test_nightly_release_retries_local_cache_after_runtime_cas_failure(self):
+        workflow = (ROOT / ".github/workflows/nightly.yml").read_text()
+        step = workflow.split("      - name: Build nightly app (Release)\n", 1)[1].split("      - name:", 1)[0]
</file context>

[cubic-dev-ai]

P2: This matcher is broader than a cache-failure check and can retry unrelated compiler failures whenever the log contains routine cache diagnostics or an unrelated CAS occurrence. Match cache-specific error lines such as CAS error: or cache/remote/plugin messages containing error or failed, then keep unrelated failures on the single failing path.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/nightly.yml, line 2536:

<comment>This matcher is broader than a cache-failure check and can retry unrelated compiler failures whenever the log contains routine cache diagnostics or an unrelated `CAS` occurrence. Match cache-specific error lines such as `CAS error:` or cache/remote/plugin messages containing `error` or `failed`, then keep unrelated failures on the single failing path.</comment>

<file context>
@@ -2498,22 +2498,47 @@ jobs:
+          run_release_build 0 2>&1 | tee "$RUNNER_TEMP/nightly-release-build.log"
+          release_status=${PIPESTATUS[0]}
+          set -e
+          if [[ "$release_status" -ne 0 ]] && grep -Eiq 'compilation cache|remote service|CAS|cache plugin' "$RUNNER_TEMP/nightly-release-build.log"; then
+            echo "::warning::managed fleet compilation cache failed; retrying the Release build with the local cache"
+            run_release_build 1
</file context>

Suggested change

	if [[ "$release_status" -ne 0 ]] && grep -Eiq 'compilation cache|remote service|CAS|cache plugin' "$RUNNER_TEMP/nightly-release-build.log"; then
	if [[ "$release_status" -ne 0 ]] && grep -Eiq 'CAS error:|compilation cache[^[:cntrl:]]*(error|fail)|remote service[^[:cntrl:]]*(error|fail)|cache plugin[^[:cntrl:]]*(error|fail)' "$RUNNER_TEMP/nightly-release-build.log"; then

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review 588c01a: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr30107 -f ref=lane/nightly-cache-fallback -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); base red: the same failure on the base 08c0011; does not block

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4130052256501 (28aa638015)

[blacksmith-sh]

Found 9 test failures on Blacksmith runners:

Failures

Test	View Logs
/tmp/tmp79d3rq_y/summary-missing.json (live):/summary-missing.json (live):	View Logs
/tmp/tmpdyc9cnk3/summary-fail.json (live):/summary-fail.json (live):	View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/ [new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows	View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames	View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/ [tabs] debug.tab_switch timed 0 of 40 switches	View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames	View Logs
convention violations found (ERROR lines above)./ convention violations found (ERROR lines above).	View Logs
test_cmux_next_follows_the_committed_target_graph (main.FocusedTestMap.test_cmux_ne xt_follows_the_committed_target_graph)/ test_cmux_next_follows_the_committed_target_graph (main.FocusedTestMap.test_cmux_ne xt_follows_the_committed_target_graph)	View Logs
test_every_part_of_bun_run_check_runs_in_the_warn_branch (main.WebviewsWarnSplitCov ersCheck.test_every_part_of_bun_run_check_runs_in_the_warn_branch)/ test_every_part_of_bun_run_check_runs_in_the_warn_branch (main.WebviewsWarnSplitCov ersCheck.test_every_part_of_bun_run_check_runs_in_the_warn_branch)	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[teamleaderleo]

Review (codex, router-review): SHIP at 154bac7

Reviewer: codex gpt-6-astra high, fast tier.

Follow-ups: 2. Each lands as an issue in this repo that links this receipt; the lane that opened the PR fixes it forward.

SHIP at 154bac7

Visual verdict: no images; Leo tries the build

1. [P1] [follow-up] .github/workflows/nightly.yml:2525: The local retry leaves settings=() and expands "${settings[@]}" under set -u. On macOS’s stock Bash 3.2, this raises an unbound-variable error before launching the retry. Use the repository’s existing guarded expansion, ${settings[@]+"${settings[@]}"}, and exercise the failure path under Bash 3.2.

2. [P2] [follow-up] .github/workflows/nightly.yml:2536: The matcher searches the entire log for CAS, but the diagnostic wrapper always echoes COMPILATION_CACHE_CAS_PATH when fleet settings are active. Consequently, unrelated failures also trigger recovery. Using the actual wrapper and a stub compiler, an initial error: no such module Missing triggered two attempts and returned success when the second succeeded. Match cache-specific error diagnostics, excluding echoed settings, and replace the string-presence assertions with executable failure cases.

Validation: all 44 Python tests across the three relevant suites passed, as did the fleet shell test. Native Xcode and Bash 3.2 execution were unavailable.

Follow-ups:

• Fix find focus and align search overlay #1: Make empty-array expansion safe for Bash 3.2.
• Disambiguate Sentry environment between production and dev #2: Restrict retry eligibility to actual cache errors and test unrelated failures at runtime.

[teamleaderleo]

Review (subagent claude-opus-5-5 medium): SHIP at c4daae1

Read the full diff against feat-cmux-next and ran the tests locally at this head: tests/test_ci_fleet_compilation_cache.py (8 OK), tests/test_ci_fleet_compilation_cache.sh (PASS), scripts/cmux-next/tests/scheme-compile-derived-data.test.sh (ok).

• Bash 3.2 / set -u: the per-build array expands as ${settings[@]+"${settings[@]}"}, so astra's P1 at 154bac7 is fixed. settings=("${fleet_cache_settings[@]}") is safe because fleet-compilation-cache.sh always prints the five prefix-mapping/remark settings, even with the cache disabled, so that array is never empty.
• Matcher: cmux_next_runtime_cas_failed keeps only [path: ](fatal )?error: lines before xcodebuild exit status:. An echoed COMPILATION_CACHE_CAS_PATH=... line can't match, which fixes astra's P2. It still matches CAS error / cannot create CAS / failed to load from CAS / replay cached diagnostics / remote service unavailable / cache plugin failed, so Sol's ci(nightly): harden local cache fallback #30130 BLOCK point is resolved. Source errors that mention CAS or cache don't match. Tests cover both directions, including an end-to-end stub of the extracted step (retry gives 2 builds and exits 0; a code error gives 1 build and exits 65).
• Bounded: there is one retry. The retry runs without the fleet settings and keeps the local COMPILATION_CACHE_ENABLE_CACHING=YES. If the retry fails, set -e stops the step. PIPESTATUS[0] is read inside set +e, so the tee doesn't mask the status.
• A nightly with no fleet CAS or with cold_cache behaves as before: one build, and the original exit status is kept.

No blocking findings.

Follow-ups:

• [follow-up] The PR deletes ci(nightly): share compiler-cache settings between warmer, PR Release and nightly Release #30128's test_warmer_and_pr_release_use_the_same_cache_settings, so nothing now asserts that the cache warmer (nightly.yml "Refresh universal nightly compilation cache") and ci-macos "Build app (Release)" still pass "${fleet_cache_settings[@]}". Restore it next to the new tests.
• [follow-up] The local-cache retry also drops the prefix-mapping and COMPILATION_CACHE_ENABLE_DIAGNOSTIC_REMARKS settings, because it drops the whole array and not only the CAS/plugin/remote entries. The build metrics receipt will show no cache remarks for a fallback build. Keeping the non-CAS entries on the retry would preserve them.

[teamleaderleo]

Review (codex, router-review): SHIP at 588c01a

Follow-up: #30106 preserves prefix-mapping and diagnostic settings during local fallback. The CAS error: matcher is covered by the runtime fixture; unrelated compiler errors remain non-retryable. Targeted Python and shell suites pass.