[teamleaderleo]

The nightly Release fallback now retries only when compiler diagnostics identify a real CAS or compilation-cache failure. Successful cache setup output and unrelated compiler errors no longer trigger a second build. The local retry also omits an empty settings array safely under macOS Bash 3.2 with set -u.

Leo said, “it feels like you guys always forget about incremental builds and the external SSDs.” This follow-up keeps the warm local cache available while avoiding a false green from retrying an ordinary compile failure.

This branch is stacked on #30107 (ci(nightly): retry local cache after fleet CAS failure) and should land after that PR. Runtime regression coverage exercises both a CAS failure that retries and an unrelated compiler error that remains red.

Validation:

• python3 -m unittest tests/test_ci_fleet_compilation_cache.py (8 tests)
• git diff --check

Agent session: agent session
🤖 Generated with Claude Code

[teamleaderleo]

Review (codex gpt-6.1-sol xhigh, router-review): BLOCK at 89ce9b3

BLOCK at 89ce9b3

Visual verdict: no images; Leo tries the build

1. [P1][blocking] scripts/cmux-next/lib/compilation-cache.sh:90 no longer recognizes failures such as error: remote service unavailable or error: cache plugin failed. These contain neither remote cache nor compilation cache and no CAS phrase, so the nightly exits after the first failed fleet build instead of retrying locally. The previous guard explicitly handled remote service and cache plugin; retain equivalent failure-scoped matches.

[teamleaderleo]

Review (subagent matcher_delta_review): SHIP at b900bb9.\n\nFollow-up delta retains remote service and cache plugin failure diagnostics while keeping the error-line filter. Targeted matcher and nightly retry tests pass.

[cubic-dev-ai]

3 issues found across 4 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="scripts/cmux-next/lib/compilation-cache.sh">

<violation number="1" location="scripts/cmux-next/lib/compilation-cache.sh:90">
P2: This matcher misses failures whose verb precedes the service name, such as `failed to connect to remote service` or `failed to send request to cache plugin`, so recoverable fleet-cache failures remain red instead of retrying locally. Match the failure forms on either side of `remote service` and `cache plugin`.</violation>

<violation number="2" location="scripts/cmux-next/lib/compilation-cache.sh:90">
P2: This predicate still treats arbitrary source-provided diagnostic text as a runtime cache failure: `#error("remote service unavailable")` triggers the local retry. Distinguish cache failures using a diagnostic/category that cannot be supplied by source code, or remove these ambiguous generic phrases from this predicate.</violation>
</file>

<file name="tests/test_ci_fleet_compilation_cache.py">

<violation number="1" location="tests/test_ci_fleet_compilation_cache.py:152">
P2: This regression test verifies a second build but not that the retry uses the local cache. Record each stub invocation's arguments and assert the first includes fleet settings while the retry omits them.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic</sub>

[cubic-dev-ai]

P2: This matcher misses failures whose verb precedes the service name, such as failed to connect to remote service or failed to send request to cache plugin, so recoverable fleet-cache failures remain red instead of retrying locally. Match the failure forms on either side of remote service and cache plugin.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/cmux-next/lib/compilation-cache.sh, line 90:

<comment>This matcher misses failures whose verb precedes the service name, such as `failed to connect to remote service` or `failed to send request to cache plugin`, so recoverable fleet-cache failures remain red instead of retrying locally. Match the failure forms on either side of `remote service` and `cache plugin`.</comment>

<file context>
@@ -78,6 +78,18 @@ cmux_next_cas_failed() {
+  [[ -f "$1" ]] || return 1
+  local diag='^([^[:space:]]+: )?(fatal )?error: ' errors
+  errors="$(awk '/^xcodebuild exit status:/ { exit } { print }' "$1" | grep -E "$diag" || true)"
+  grep -Eqi 'CAS error encountered during|cannot (create|open) (the )?CAS|failed to (load .* from CAS|open .*CAS|replay cached diagnostics)|cannot create cache key for compilation|((compilation|remote) cache).*(error|failed|unavailable)|((remote service|cache plugin).*(error|failed|unavailable))' <<<"$errors"
+}
+
</file context>

Suggested change

	grep -Eqi 'CAS error encountered during|cannot (create|open) (the )?CAS|failed to (load .* from CAS|open .*CAS|replay cached diagnostics)|cannot create cache key for compilation|((compilation|remote) cache).*(error|failed|unavailable)|((remote service|cache plugin).*(error|failed|unavailable))' <<<"$errors"
	grep -Eqi 'CAS error encountered during|cannot (create|open) (the )?CAS|failed to (load .* from CAS|open .*CAS|replay cached diagnostics)|cannot create cache key for compilation|((compilation|remote) cache).*(error|failed|unavailable)|((remote service|cache plugin).*(error|failed|unavailable))|(failed|could not) .* (remote service|cache plugin)' <<<"$errors"

[cubic-dev-ai]

P2: This regression test verifies a second build but not that the retry uses the local cache. Record each stub invocation's arguments and assert the first includes fleet settings while the retry omits them.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/test_ci_fleet_compilation_cache.py, line 152:

<comment>This regression test verifies a second build but not that the retry uses the local cache. Record each stub invocation's arguments and assert the first includes fleet settings while the retry omits them.</comment>

<file context>
@@ -100,12 +104,64 @@ def test_nightly_xcode_command_receives_shared_settings(self):
+        ):
+            result, count = self._run_nightly_release_stub(diagnostic)
+            self.assertEqual(result.returncode, 0, result.stderr)
+            self.assertEqual(count.read_text().strip(), "2")
+
+        result, count = self._run_nightly_release_stub("error: use of unresolved identifier 'cas'")
</file context>

[cubic-dev-ai]

P2: This predicate still treats arbitrary source-provided diagnostic text as a runtime cache failure: #error("remote service unavailable") triggers the local retry. Distinguish cache failures using a diagnostic/category that cannot be supplied by source code, or remove these ambiguous generic phrases from this predicate.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/cmux-next/lib/compilation-cache.sh, line 90:

<comment>This predicate still treats arbitrary source-provided diagnostic text as a runtime cache failure: `#error("remote service unavailable")` triggers the local retry. Distinguish cache failures using a diagnostic/category that cannot be supplied by source code, or remove these ambiguous generic phrases from this predicate.</comment>

<file context>
@@ -78,6 +78,18 @@ cmux_next_cas_failed() {
+  [[ -f "$1" ]] || return 1
+  local diag='^([^[:space:]]+: )?(fatal )?error: ' errors
+  errors="$(awk '/^xcodebuild exit status:/ { exit } { print }' "$1" | grep -E "$diag" || true)"
+  grep -Eqi 'CAS error encountered during|cannot (create|open) (the )?CAS|failed to (load .* from CAS|open .*CAS|replay cached diagnostics)|cannot create cache key for compilation|((compilation|remote) cache).*(error|failed|unavailable)|((remote service|cache plugin).*(error|failed|unavailable))' <<<"$errors"
+}
+
</file context>

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[teamleaderleo]

Folded into canonical #30107 at 588c01a; no separate merge is needed. Please close this duplicate PR.

[teamleaderleo]

Folded into canonical #30107 (merged 2379dd9); follow-up carried by #30201 at 45d4404. Please close this duplicate.