Skip to content

ci: make release sccache failures non-fatal - #1079

Merged
i386 merged 1 commit into
Mesh-LLM:mainfrom
i386:codex/release-cache-fail-open
Jul 26, 2026
Merged

ci: make release sccache failures non-fatal#1079
i386 merged 1 commit into
Mesh-LLM:mainfrom
i386:codex/release-cache-fail-open

Conversation

@i386

@i386 i386 commented Jul 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • put a job-local disk cache in front of the GitHub Actions sccache backend
  • treat cache write and server I/O failures as non-fatal compiler fallbacks
  • preserve the existing disk-only startup fallback and enforce the contract in xtask
  • synchronize the CI topology documentation and inventory

Investigation

Release run https://github.com/Mesh-LLM/mesh-llm/actions/runs/30063516285 failed for two cache-related reasons:

  • four Linux GPU jobs received an Actions cache service-unavailable HTML response while sccache was starting during compiler probing
  • the Windows ROCm native-runtime job aborted when sccache could not persist a temporary cache file

Both failures blocked release artifacts even though the compiler inputs and build logic were otherwise valid.

Validation

  • actionlint -config-file .github/actionlint.yaml
  • git diff --check
  • just check-release
  • just with-lld cargo test -p xtask workflow_checks
  • just with-lld cargo fmt --all -- --check
  • just with-lld cargo check -p xtask
  • just with-lld cargo clippy -p xtask --all-targets -- -D warnings

A release workflow dispatch was not started: the repository CI policy requires explicit authorization for that expensive live operation.

Summary by CodeRabbit

  • CI Improvements

    • Improved build-cache reliability with a disk-first cache and best-effort remote caching.
    • Cache connection and write failures now degrade gracefully without interrupting builds.
    • Builds fall back to local compiler execution when the remote cache service is unavailable.
  • Documentation

    • Updated CI and runner guidance to clarify cache behavior, fallback handling, and configuration requirements.

@github-actions
github-actions Bot requested a review from ndizazzo July 26, 2026 01:55
@coderabbitai

coderabbitai Bot commented Jul 26, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The PR configures sccache with a disk-first, best-effort GitHub Actions cache chain, adds disk-only fallback behavior, strengthens release contract validation, and updates CI documentation and inventory.

Changes

sccache cache contract

Layer / File(s) Summary
Runtime cache configuration
.github/actions/configure-sccache-gha/action.yml, .github/workflows/release.yml
Configures disk and GitHub Actions caching, ignored server I/O and write errors, and disk-only fallback when remote caching is unavailable.
Release contract validation
tools/xtask/src/workflow_checks.rs
Validates required sccache settings and adds failures for missing write fallback or disk-first cache-chain configuration.
Documentation alignment
.agents/skills/manage-ci/references/current-inventory.md, ci/ci.md
Documents cache tiers, failure handling, local fallback, remote probe behavior, and the baked-binary requirement.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ReleaseWorkflow
  participant ConfigureSccacheGHA
  participant Sccache
  participant GithubActionsCache
  ReleaseWorkflow->>ConfigureSccacheGHA: configure sccache environment
  ConfigureSccacheGHA->>Sccache: start disk,gha cache chain
  Sccache->>GithubActionsCache: probe best-effort remote cache
  ConfigureSccacheGHA->>Sccache: restart with disk-only chain on probe failure
Loading

Possibly related PRs

Suggested reviewers: ndizazzo, michaelneale

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title matches the main change: release CI now treats sccache/GHA cache failures as non-fatal.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
tools/xtask/src/workflow_checks.rs (1)

439-459: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Validate exact configuration values, not substrings.

ensure_contains can accept malformed workflow values such as SCCACHE_MULTILEVEL_CHAIN: disk,gha-disabled or SCCACHE_MULTILEVEL_WRITE_ERROR_POLICY: ignore-disabled. Since this checker gates the release fallback contract, use exact-line/scoped validation (or parsed YAML/source checks) and add wrong-value regression tests.

Also applies to: 487-502

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tools/xtask/src/workflow_checks.rs` around lines 439 - 459, The release
workflow checks in the validation function around the required sccache entries
currently allow malformed values through substring matching. Replace the
ensure_contains checks for SCCACHE_DIR, SCCACHE_IGNORE_SERVER_IO_ERROR,
SCCACHE_MULTILEVEL_CHAIN, and SCCACHE_MULTILEVEL_WRITE_ERROR_POLICY with
exact-line or scoped YAML/source validation, and add regression tests proving
suffixed or otherwise incorrect values are rejected.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tools/xtask/src/workflow_checks.rs`:
- Around line 439-459: The release workflow checks in the validation function
around the required sccache entries currently allow malformed values through
substring matching. Replace the ensure_contains checks for SCCACHE_DIR,
SCCACHE_IGNORE_SERVER_IO_ERROR, SCCACHE_MULTILEVEL_CHAIN, and
SCCACHE_MULTILEVEL_WRITE_ERROR_POLICY with exact-line or scoped YAML/source
validation, and add regression tests proving suffixed or otherwise incorrect
values are rejected.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: d847d9fd-0b85-4259-a9c1-ecf19b86d64a

📥 Commits

Reviewing files that changed from the base of the PR and between cdd793b and ae34f48.

📒 Files selected for processing (5)
  • .agents/skills/manage-ci/references/current-inventory.md
  • .github/actions/configure-sccache-gha/action.yml
  • .github/workflows/release.yml
  • ci/ci.md
  • tools/xtask/src/workflow_checks.rs

@i386
i386 merged commit 465cbc1 into Mesh-LLM:main Jul 26, 2026
23 checks passed
michaelneale added a commit that referenced this pull request Jul 27, 2026
* origin/main:
  Fix Metal small-batch matmul parity for GLM verification (#1078)
  Handle K-only transposed KV page import and export (#1084)
  Refresh llama.cpp upstream patch queue (#1085)
  chore: improve embedded native-runtime compatibility guidance (#1043)
  fix(console-ui): chat transcript snapping during live status updates (#1083)
  ci: bump Linux CUDA slim container to gha-convention base runner image
  fix: record activation cache prefix identities (#1041)
  fix: read-only model download caches (#1042)
  ci: disable sccache for Windows ROCm native runtime build (#1087)
  ci: fix v0.74 release GPU builds (sccache disk-only + force_hosted_runners) (#1086)
  Make release sccache failures non-fatal (#1079)
  Keep client-only nodes out of model election (#1074)

# Conflicts:
#	crates/mesh-llm-host-runtime/src/runtime/auto_join.rs
#	crates/mesh-llm-host-runtime/src/runtime/tests/auto_join.rs
#	third_party/llama.cpp/patches/0004-Add-lanes-external-media-and-chat-grammar-support.patch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant