Repository navigation
ci(nightly): preserve settings on local cache fallback - #30201
Conversation
|
Review (subagent gpt-6.1-sol): SHIP at 7f91848 The fallback excludes only CAS/plugin/remote settings and preserves prefix mappings and diagnostic remarks. Array expansion is safe; runtime and shell tests pass with no findings. |
GitHub Actions
Inactive
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
4 issues found across 5 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="tests/test_ci_fleet_compilation_cache.sh">
<violation number="1" location="tests/test_ci_fleet_compilation_cache.sh:33">
P3: This adds a second run of `test_ci_fleet_compilation_cache.py`. `ci-guards.yml` already runs the shell guard and then the Python test in the same step, and the test is registered in `test-execution.toml`. Drop this line and keep the single invocation in one place.</violation>
</file>
<file name="scripts/cmux-next/lib/compilation-cache.sh">
<violation number="1" location="scripts/cmux-next/lib/compilation-cache.sh:90">
P2: The `CAS` alternatives lack word boundaries, and `-i` makes them match any identifier starting with "cas". A source error such as `error: failed to open 'CASKeyStore.swift'` therefore triggers the local-cache retry, which the PR says must not happen. Add `\b` around `CAS` in those alternatives.</violation>
</file>
<file name=".github/workflows/nightly.yml">
<violation number="1" location=".github/workflows/nightly.yml:2554">
P3: This comment is inaccurate: the nightly step uses the stricter `cmux_next_runtime_cas_failed`, while the other Release paths use `cmux_next_cas_failed`. Reword it to say the nightly fallback deliberately uses the stricter runtime predicate.</violation>
<violation number="2" location=".github/workflows/nightly.yml:2591">
P2: This retries on a local-CAS diagnostic even when no fleet settings were emitted, so the second Release build uses the same settings and repeats the work. Gate the retry on the fleet and local settings actually differing.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| [[ -f "$1" ]] || return 1 | ||
| local diag='^([^[:space:]]+: )?(fatal )?error: ' errors | ||
| errors="$(awk '/^xcodebuild exit status:/ { exit } { print }' "$1" | grep -E "$diag" || true)" | ||
| grep -Eqi 'CAS error(:| encountered during)|cannot (create|open) (the )?CAS|failed to (load .* from CAS|open .*CAS|replay cached diagnostics)|cannot create cache key for compilation|((compilation|remote) cache).*(error|failed|unavailable)|((remote service|cache plugin).*(error|failed|unavailable))' <<<"$errors" |
There was a problem hiding this comment.
P2: The CAS alternatives lack word boundaries, and -i makes them match any identifier starting with "cas". A source error such as error: failed to open 'CASKeyStore.swift' therefore triggers the local-cache retry, which the PR says must not happen. Add \b around CAS in those alternatives.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/cmux-next/lib/compilation-cache.sh, line 90:
<comment>The `CAS` alternatives lack word boundaries, and `-i` makes them match any identifier starting with "cas". A source error such as `error: failed to open 'CASKeyStore.swift'` therefore triggers the local-cache retry, which the PR says must not happen. Add `\b` around `CAS` in those alternatives.</comment>
<file context>
@@ -78,6 +78,18 @@ cmux_next_cas_failed() {
+ [[ -f "$1" ]] || return 1
+ local diag='^([^[:space:]]+: )?(fatal )?error: ' errors
+ errors="$(awk '/^xcodebuild exit status:/ { exit } { print }' "$1" | grep -E "$diag" || true)"
+ grep -Eqi 'CAS error(:| encountered during)|cannot (create|open) (the )?CAS|failed to (load .* from CAS|open .*CAS|replay cached diagnostics)|cannot create cache key for compilation|((compilation|remote) cache).*(error|failed|unavailable)|((remote service|cache plugin).*(error|failed|unavailable))' <<<"$errors"
+}
+
</file context>
| grep -Eqi 'CAS error(:| encountered during)|cannot (create|open) (the )?CAS|failed to (load .* from CAS|open .*CAS|replay cached diagnostics)|cannot create cache key for compilation|((compilation|remote) cache).*(error|failed|unavailable)|((remote service|cache plugin).*(error|failed|unavailable))' <<<"$errors" | |
| grep -Eqi '\bCAS error(:| encountered during)|cannot (create|open) (the )?\bCAS\b|failed to (load .* from \bCAS\b|open .*\bCAS\b|replay cached diagnostics)|cannot create cache key for compilation|((compilation|remote) cache).*(error|failed|unavailable)|((remote service|cache plugin).*(error|failed|unavailable))' <<<"$errors" |
| run_release_build 0 2>&1 | tee "$RUNNER_TEMP/nightly-release-build.log" | ||
| release_status=${PIPESTATUS[0]} | ||
| set -e | ||
| if [[ "$release_status" -ne 0 ]] && cmux_next_runtime_cas_failed "$RUNNER_TEMP/nightly-release-build.log"; then |
There was a problem hiding this comment.
P2: This retries on a local-CAS diagnostic even when no fleet settings were emitted, so the second Release build uses the same settings and repeats the work. Gate the retry on the fleet and local settings actually differing.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/nightly.yml, line 2591:
<comment>This retries on a local-CAS diagnostic even when no fleet settings were emitted, so the second Release build uses the same settings and repeats the work. Gate the retry on the fleet and local settings actually differing.</comment>
<file context>
@@ -2543,26 +2543,57 @@ jobs:
+ run_release_build 0 2>&1 | tee "$RUNNER_TEMP/nightly-release-build.log"
+ release_status=${PIPESTATUS[0]}
+ set -e
+ if [[ "$release_status" -ne 0 ]] && cmux_next_runtime_cas_failed "$RUNNER_TEMP/nightly-release-build.log"; then
+ echo "::warning::managed fleet compilation cache failed; retrying the Release build with the local cache"
+ run_release_build 1
</file context>
| if [[ "$release_status" -ne 0 ]] && cmux_next_runtime_cas_failed "$RUNNER_TEMP/nightly-release-build.log"; then | |
| if [[ "$release_status" -ne 0 ]] && [[ "${fleet_cache_settings[*]}" != "${local_cache_settings[*]}" ]] && cmux_next_runtime_cas_failed "$RUNNER_TEMP/nightly-release-build.log"; then |
| grep -Fq 'glaeda-compile-telemetry.json' "$WORKFLOW" || fail 'compile admission must publish the host telemetry sidecar' | ||
|
|
||
| grep -Fq 'bash "$SCRIPT_DIR/fleet-compilation-cache.sh"' "$COMPILE" || fail 'admission must use the shared settings' | ||
| python3 "$ROOT_DIR/tests/test_ci_fleet_compilation_cache.py" |
There was a problem hiding this comment.
P3: This adds a second run of test_ci_fleet_compilation_cache.py. ci-guards.yml already runs the shell guard and then the Python test in the same step, and the test is registered in test-execution.toml. Drop this line and keep the single invocation in one place.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/test_ci_fleet_compilation_cache.sh, line 33:
<comment>This adds a second run of `test_ci_fleet_compilation_cache.py`. `ci-guards.yml` already runs the shell guard and then the Python test in the same step, and the test is registered in `test-execution.toml`. Drop this line and keep the single invocation in one place.</comment>
<file context>
@@ -30,4 +30,5 @@ grep -Fq 'fleet_cache_setting+=("COMPILATION_CACHE_CAS_PATH=$fleet_cas_root/cas"
grep -Fq 'glaeda-compile-telemetry.json' "$WORKFLOW" || fail 'compile admission must publish the host telemetry sidecar'
grep -Fq 'bash "$SCRIPT_DIR/fleet-compilation-cache.sh"' "$COMPILE" || fail 'admission must use the shared settings'
+python3 "$ROOT_DIR/tests/test_ci_fleet_compilation_cache.py"
echo 'PASS: fleet compiler-cache settings and host telemetry are wired'
</file context>
| # Keep retry classification identical to the other Release compile | ||
| # paths: only compiler diagnostics that identify the CAS/cache as | ||
| # failing may fall back to a local cache. |
There was a problem hiding this comment.
P3: This comment is inaccurate: the nightly step uses the stricter cmux_next_runtime_cas_failed, while the other Release paths use cmux_next_cas_failed. Reword it to say the nightly fallback deliberately uses the stricter runtime predicate.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/nightly.yml, line 2554:
<comment>This comment is inaccurate: the nightly step uses the stricter `cmux_next_runtime_cas_failed`, while the other Release paths use `cmux_next_cas_failed`. Reword it to say the nightly fallback deliberately uses the stricter runtime predicate.</comment>
<file context>
@@ -2543,26 +2543,57 @@ jobs:
- COMPILER_INDEX_STORE_ENABLE=NO \
- CODE_SIGNING_ALLOWED=NO ASSETCATALOG_COMPILER_APPICON_NAME="$CHANNEL_ICON_NAME" \
- "${notary_test_flags[@]}"
+ # Keep retry classification identical to the other Release compile
+ # paths: only compiler diagnostics that identify the CAS/cache as
+ # failing may fall back to a local cache.
</file context>
| # Keep retry classification identical to the other Release compile | |
| # paths: only compiler diagnostics that identify the CAS/cache as | |
| # failing may fall back to a local cache. | |
| # Nightly uses the stricter runtime CAS predicate (lib/compilation-cache.sh | |
| # cmux_next_runtime_cas_failed), so source errors that mention CAS or cache do not | |
| # trigger a local retry. Only compiler diagnostics that identify the cache as failing qualify. |
7f91848 to
45d4404
Compare
Bugbot needs on-demand usage enabledBugbot uses usage-based billing for this team and requires on-demand usage to be enabled. A team admin can enable on-demand usage in the Cursor dashboard. |
GitHub Actions
Inactive
|
Review (codex, router-review): SHIP at 45d4404 Reviewer: codex gpt-6-astra high, fast tier. SHIP at 45d4404 Visual verdict: no images; Leo tries the build No findings. The retry preserves all shared mapping and diagnostic settings while removing the three fleet backend settings. Validation: all 9 Python tests and the shell regression passed. Actual Xcode execution was not available in this box. Follow-ups: none |
GitHub Actions
Inactive
![cmux-bot[bot]](https://avatars.githubusercontent.com/u/171392238?s=60&v=4){kind=small}
|
cmux status (one comment, edited in place; each line is updated by the job that owns it) Review App of this head (app.zip, kept 1 day)
red: cmux-next checks (god files, concurrency, crash safety, l10n); base red: the same failure on the base 7e1e768; does not block Shipped in:
|
A newer verdict blocks 09e307c (no independent SHIP at the head).
|
Found 8 test failures on Blacksmith runners: Failures
|
![Fix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/fix-with-codesmith-light-3x.png)
|
Review (Codex local_settings_review): SHIP at 09e307c\n\nRe-reviewed the exact merge head. The fallback preserves prefix-mapping and diagnostic settings while excluding only CAS path, plugin, and remote service settings. Exact-head cache tests pass (9/9), including runtime retry behavior and shell argument capture; syntax and diff checks are clean. |
The nightly Release fallback now keeps the shared prefix-mapping and diagnostic settings when it drops only the fleet CAS/plugin/remote settings. This preserves incremental cache keys and hit telemetry on the local retry while still removing the unavailable fleet backend.
Leo said, “it feels like you guys always forget about incremental builds and the external SSDs.” This follows the exact-head review of #30107 and resolves the remaining part of #30106 after #30128 aligned the warmer and PR Release settings.
The executable regression runs both the fleet-backed attempt and the local retry, asserting the mapping/diagnostic settings remain while CAS/plugin/remote settings disappear from the retry.
Validation:
python3 -m unittest tests/test_ci_fleet_compilation_cache.py(9 tests)bash tests/test_ci_fleet_compilation_cache.shpython3 scripts/cmux-next/prepush-gate.pyCloses #30106.
Agent session: agent session
🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need.Summary by cubic
The nightly Release build now retries once with the local cache when the managed fleet CAS fails at runtime, instead of failing the job outright. The retry filters out only the fleet CAS/plugin/remote settings, keeping the shared prefix-mapping and diagnostic settings so incremental cache keys and hit telemetry stay intact.
CAS error,remote service unavailable,cache plugin failed); source errors that merely mention "CAS" or "cache" still fail without a second build.Written for commit 09e307c. Summary will update on new commits.