ci: fix v0.74 release GPU builds (sccache disk-only + force_hosted_runners) - #1086
Conversation
Long CUDA/ROCm/Vulkan release builds run past the lifetime of the ephemeral GitHub Actions cache token. When an sccache server starts after the token expires it probes the gha tier of the multilevel chain, that storage read fails permanently (HTTP 400), and the build aborts before any fail-open policy applies. Honor a job-level SCCACHE_GHA_ENABLED=false in configure-sccache-gha by running those lanes disk-only and clearing the cache URL/token so no residual gha configuration re-enables the remote tier. Fast lanes keep the best-effort disk,gha behavior.
📝 WalkthroughWalkthroughThe GitHub Actions changes add an explicit disk-only sccache mode and a manual release-workflow option that routes selected lanes to GitHub-hosted runners. Changessccache disk-only opt-out
Hosted runner routing
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
This pull request is currently a draft. Reviews will not take place until the PR is marked as ready for review. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/actions/configure-sccache-gha/action.yml:
- Around line 26-28: In the SCCACHE_GHA_ENABLED=false branch, extend the
existing environment-variable reset alongside ACTIONS_CACHE_URL,
ACTIONS_RESULTS_URL, and ACTIONS_RUNTIME_TOKEN to also clear
SCCACHE_GHA_CACHE_URL and SCCACHE_GHA_RUNTIME_TOKEN, preventing inherited GHA
configuration from reactivating the remote backend.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 5d9f31b9-d0ec-4990-9000-04459eea8497
📒 Files selected for processing (1)
.github/actions/configure-sccache-gha/action.yml
| core.exportVariable('ACTIONS_CACHE_URL', ''); | ||
| core.exportVariable('ACTIONS_RESULTS_URL', ''); | ||
| core.exportVariable('ACTIONS_RUNTIME_TOKEN', ''); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== candidate files =="
git ls-files | grep -E '^(\.github/actions/configure-sccache-gha/action\.ya?ml|\.agents/skills/manage-ci/SKILL\.md|^ci/ci\.md)$' || true
echo
echo "== action.yml context =="
if [ -f .github/actions/configure-sccache-gha/action.yml ]; then
nl -ba .github/actions/configure-sccache-gha/action.yml | sed -n '1,140p'
fi
echo
echo "== manage-ci docs =="
if [ -f .agents/skills/manage-ci/SKILL.md ]; then
nl -ba .agents/skills/manage-ci/SKILL.md | sed -n '1,220p'
fi
echo
echo "== relevant search =="
rg -n "sccache|SCCACHE|sccache_gha|ACTIONS_CACHE_URL|SCCACHE_GHA" .github .agents ci 2>/dev/null || trueRepository: Mesh-LLM/mesh-llm
Length of output: 336
🌐 Web query:
Mozilla sccache configuration SCCACHE_GHA_CACHE_URL SCCACHE_GHA_RUNTIME_TOKEN
💡 Result:
To configure sccache to use the GitHub Actions (GHA) cache, you must set specific environment variables that provide access to the GitHub Actions cache service [1][2]. Configuration Details: - SCCACHE_GHA_CACHE_URL: Specifies the GitHub Actions cache API URL [1][3]. This variable is used interchangeably or alongside ACTIONS_RESULTS_URL (which has replaced older legacy endpoints) [1][4]. - SCCACHE_GHA_RUNTIME_TOKEN: Provides the GitHub Actions access token required to authenticate with the cache service [1][5]. This works in conjunction with or as an override to the ACTIONS_RUNTIME_TOKEN environment variable [1][6]. Key Setup Requirements: - Enable the backend: You must set SCCACHE_GHA_ENABLED to on [2]. - RW Mode: You can optionally set SCCACHE_GHA_RW_MODE to READ_ONLY if you wish to restrict the cache backend from writing data [1][7]. - Cache Keys: Use SCCACHE_GHA_CACHE_TO to specify the cache key for writing and SCCACHE_GHA_CACHE_FROM to provide a comma-separated list of cache key prefixes for reading [1][6]. Note on Environment Variables: The variables prefixed with SCCACHE_GHA_ take precedence over the standard ACTIONS_ prefixed variables [6][8]. When running in a GitHub Actions workflow, these tokens and URLs are typically dynamically exported using the GitHub Actions runtime environment [2][4]. Ensure you are using current documentation, as some older guides may reference deprecated endpoint variables [4].
Citations:
- 1: https://github.com/mozilla/sccache/blob/main/docs/Configuration.md
- 2: https://github.com/mozilla/sccache/blob/main/docs/GHA.md
- 3: https://docs.rs/crate/sccache/latest/source/docs/Configuration.md
- 4: Time sensitive: GitHub Actions cache service (Update Guidance) mozilla/sccache#2351
- 5: https://android.googlesource.com/toolchain/sccache/+/refs/heads/main/docs/Configuration.md
- 6: https://docs.rs/crate/sccache/0.3.3
- 7: https://github.com/mozilla/sccache/blob/main/docs/Configuration.md?plain=1
- 8: https://crates.io/crates/sccache/0.3.3
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== tool checks =="
(command -v sed || true)
(command -v ast-grep || true)
echo
echo "== action.yml context =="
sed -n '1,140p' .github/actions/configure-sccache-gha/action.yml
echo
echo "== manage-ci inventory/docs snippets =="
echo "--- SKILL.md relevant lines ---"
grep -n -i -E "ci/ci\.md|configure-sccache-gha|sccache|ACTIONS_CACHE_URL|ACTIONS_RUNTIME_TOKEN|SCCACHE|inventory|CI" .agents/skills/manage-ci/SKILL.md || true
echo "--- ci/ci.md relevant lines ---"
grep -n -i -E "configure-sccache-gha|sccache|ACTIONS_CACHE_URL|ACTIONS_RUNTIME_TOKEN|SCCACHE|github" ci/ci.md || true
echo
echo "== repo-wide SCCACHE/GHA references =="
grep -Rn -i -E "SCCACHE|configure-sccache-gha|ACTIONS_CACHE_URL|ACTIONS_RUNTIME_TOKEN|ACTIONS_RESULTS_URL|SCCACHE_GHA|MULTILEVEL_CHAIN" .github .agents ci 2>/dev/null || trueRepository: Mesh-LLM/mesh-llm
Length of output: 24728
Clear sccache’s canonical GHA variables during opt-out.
This branch resets the ACTIONS_* aliases, but sccache also accepts SCCACHE_GHA_CACHE_URL and SCCACHE_GHA_RUNTIME_TOKEN, which can take precedence over the cleared values. Clear them in the SCCACHE_GHA_ENABLED=false path so inherited gha configuration cannot reactivate the remote backend if the chain changes later.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/actions/configure-sccache-gha/action.yml around lines 26 - 28, In
the SCCACHE_GHA_ENABLED=false branch, extend the existing environment-variable
reset alongside ACTIONS_CACHE_URL, ACTIONS_RESULTS_URL, and
ACTIONS_RUNTIME_TOKEN to also clear SCCACHE_GHA_CACHE_URL and
SCCACHE_GHA_RUNTIME_TOKEN, preventing inherited GHA configuration from
reactivating the remote backend.
Adds an opt-in workflow_dispatch input that routes the self-hosted release lanes (x86_64 CUDA, ARM64 smoke) to GitHub-hosted runners for a single run, without changing the repo-wide USE_SELF_HOSTED variable. This lets a release run on hosted runners (which provide /__e/node24) when the self-hosted ARC pods lack node24, while leaving CI and PR-build self-hosted routing intact. Defaults to false, so existing behavior is unchanged unless explicitly set.
michaelneale
changed the title
* origin/main: Fix Metal small-batch matmul parity for GLM verification (#1078) Handle K-only transposed KV page import and export (#1084) Refresh llama.cpp upstream patch queue (#1085) chore: improve embedded native-runtime compatibility guidance (#1043) fix(console-ui): chat transcript snapping during live status updates (#1083) ci: bump Linux CUDA slim container to gha-convention base runner image fix: record activation cache prefix identities (#1041) fix: read-only model download caches (#1042) ci: disable sccache for Windows ROCm native runtime build (#1087) ci: fix v0.74 release GPU builds (sccache disk-only + force_hosted_runners) (#1086) Make release sccache failures non-fatal (#1079) Keep client-only nodes out of model election (#1074) # Conflicts: # crates/mesh-llm-host-runtime/src/runtime/auto_join.rs # crates/mesh-llm-host-runtime/src/runtime/tests/auto_join.rs # third_party/llama.cpp/patches/0004-Add-lanes-external-media-and-chat-grammar-support.patch
1 participant
What this fixes
Restores the ability to cut a
v0.74.0release with GPU bundles. RC attempts (rc1–rc3) failed on the CUDA/ROCm/Vulkan lanes for two independent, non-code reasons:ghatier at startup, that read fails permanently (HTTP 400), and the build aborts./__e/node24, so container jobs died instantly at checkout.Both are addressed here, scoped so nothing else in CI changes.
1. sccache disk-only for GPU lanes
configure-sccache-ghaunconditionally forcedSCCACHE_MULTILEVEL_CHAIN=disk,ghaandSCCACHE_GHA_ENABLED=true, overriding theSCCACHE_GHA_ENABLED=falsethe CUDA/ROCm/Vulkan jobs already declare. Keepingghain the chain leaves a startup storage probe that hard-fails once the token expires — which is why #1079's fail-open policies didn't help (they don't cover the startup read probe; confirmed against sccache 0.16 internals).This PR makes the action honor a job-level
SCCACHE_GHA_ENABLED=false: those lanes run disk-only, and the cache URL/token are cleared so no residualghaconfig re-enables the remote tier. Fast lanes keep the best-effortdisk,ghabehavior unchanged.2.
force_hosted_runnersrelease input (node24, scoped to one run)Adds an opt-in
workflow_dispatchinput,force_hosted_runners(defaultfalse). When set, it routes the self-hosted release lanes (x86_64 CUDA, ARM64 smoke) to GitHub-hostedubuntu-24.04runners for that run only, without touching the repo-wideUSE_SELF_HOSTEDvariable.On hosted runners
/__e/node24is mounted from the host runner (verified in thedocker createline of the last successful GPU release, v0.72.0-rc6 Jun 29), independent of the container image — so the node24 failure disappears with no digest bump or image republish. This leaves CI and PR-build self-hosted routing (and @ndizazzo's #1075 direction) untouched.To cut rc4: dispatch release with
force_hosted_runners=true.Validation
actionlint -config-file .github/actionlint.yaml— passgit diff --check— cleancargo run -p xtask -- repo-consistency release-targets— passRollback
Summary by CodeRabbit
New Features
SCCACHE_GHA_ENABLED=false(disk-only mode).force_hosted_runnersinput to route selected release runs to GitHub-hosted runners for that execution.Bug Fixes