Skip to content

Fix release containers and restore sccache GHA caching - #1067

Merged
i386 merged 2 commits into
mainfrom
agent/fix-release-container-contracts
Jul 24, 2026
Merged

Fix release containers and restore sccache GHA caching#1067
i386 merged 2 commits into
mainfrom
agent/fix-release-container-contracts

Conversation

@i386

@i386 i386 commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • trust the checkout in every prebuilt Linux release container, including both Vulkan jobs
  • connect the baked sccache binary to GitHub Actions cache without downloading a second binary
  • start each job fail-closed with SCCACHE_GHA_ENABLED=false, then export the ephemeral cache URL/token and probe the remote backend
  • stop and restart sccache with job-local disk storage when the GHA backend cannot initialize
  • discover all containerized release jobs in the release consistency checker and enforce the safe-directory, cache configuration, pinned credential exporter, and local fallback contracts

Root cause

Release run 30058024843 exposed two regressions from the runner-image migration:

  1. The prebuilt images contain sccache, but the release workflow inherited SCCACHE_GHA_ENABLED=true without the Actions cache URL/token normally exported by a setup action. Cargo therefore failed before invoking rustc.
  2. The Vulkan jobs were moved into containers without trusting $GITHUB_WORKSPACE, so release-version.sh failed when git ls-files rejected the checkout as dubious ownership.

The publish job requires every platform lane, so these failures prevented the tag, GitHub release, packaging dispatch, and crates.io publication.

Cache behavior

The repository-local configure-sccache-gha action uses pinned actions/github-script to expose the cache credentials already present in an Actions step to the baked sccache. It probes the remote backend by starting the server. If initialization fails, it stops that server, disables the GHA backend, and requires a successful job-local restart before compilation continues.

Validation

  • actionlint -config-file .github/actionlint.yaml
  • composite action YAML and embedded JavaScript syntax validation
  • git diff --check
  • cargo fmt --all --check
  • cargo test -p xtask (9 passed)
  • cargo check -p xtask
  • cargo clippy -p xtask --all-targets -- -D warnings
  • cargo run -p xtask -- repo-consistency release-targets

A full canary=true release dispatch should be run after merge and before retrying v0.74.0; this PR does not dispatch a release.

Summary by CodeRabbit

  • Chores
    • Improved GPU release workflow setup by explicitly disabling GitHub Actions sccache integration where needed and adding a standardized step to configure and start the baked sccache backend.
    • Strengthened release workflow validation to ensure consistent checkout “safe directory” handling and required sccache configuration/steps are present for container jobs.
  • New Features
    • Added a composite action to configure baked sccache with a GitHub Actions cache probe and automatic job-local fallback.
  • Tests
    • Added unit tests covering the release-container workflow contract checks.
  • Documentation
    • Updated CI documentation to reflect the new sccache probe/restart and fallback behavior.

@coderabbitai

coderabbitai Bot commented Jul 24, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Changes

The release workflow now configures baked sccache across GPU container jobs with a GitHub Actions cache probe and local fallback. Documentation records the contract, while xtask checks validate the action and every discovered release container job.

Release sccache backend contracts

Layer / File(s) Summary
Sccache backend configuration
.github/actions/configure-sccache-gha/action.yml, ci/ci.md, .agents/skills/manage-ci/references/current-inventory.md
The composite action exports cache credentials, probes the GHA backend, and restarts sccache with job-local storage when remote startup fails; documentation records this behavior.
Release workflow integration
.github/workflows/release.yml
GPU native-runtime and release-bundle container jobs disable the GHA backend initially and invoke the local configuration action after runner verification.
Workflow contract validation
tools/xtask/src/workflow_checks.rs
Checks validate the action implementation and require each discovered container job to include checkout trust, the local SCCACHE setting, and the configuration step.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ReleaseJob
  participant ConfigureAction
  participant Sccache
  ReleaseJob->>ConfigureAction: invoke local sccache configuration
  ConfigureAction->>Sccache: probe GitHub Actions backend
  Sccache-->>ConfigureAction: startup result
  ConfigureAction->>Sccache: restart with local disk cache on failure
Loading

Suggested reviewers: ndizazzo

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title is concise and matches the main changes: release container fixes plus sccache GHA backend handling.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch agent/fix-release-container-contracts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

This pull request is currently a draft. Reviews will not take place until the PR is marked as ready for review.

@i386
i386 marked this pull request as ready for review July 24, 2026 02:36
@github-actions
github-actions Bot requested a review from ndizazzo July 24, 2026 02:36

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tools/xtask/src/workflow_checks.rs`:
- Around line 462-475: Update release_container_job_names to recognize job-level
container declarations whose lines begin with "    container:", including
shorthand values such as container: node:18, while preserving existing job
filtering. In tools/xtask/src/workflow_checks.rs lines 634-672, add a valid
shorthand-container fixture covering the SCCACHE and safe-directory contract
checks.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: a9134667-4389-4454-936a-b8d8bd2dc12b

📥 Commits

Reviewing files that changed from the base of the PR and between ca5b6ea and 69abfad.

📒 Files selected for processing (2)
  • .github/workflows/release.yml
  • tools/xtask/src/workflow_checks.rs

Comment on lines +462 to +475
fn release_container_job_names(release_workflow: &str) -> Vec<&str> {
release_workflow
.lines()
.filter_map(|line| {
let job_name = line.strip_prefix(" ")?.strip_suffix(':')?;
if job_name.is_empty() || job_name.starts_with(' ') || job_name.contains(' ') {
return None;
}
let job = workflow_job_section(release_workflow, job_name)?;
job.lines()
.any(|job_line| job_line == " container:")
.then_some(job_name)
})
.collect()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

rg -n -C2 'container:|release_container_job_names|VALID_CONTAINER_WORKFLOW' \
  tools/xtask/src/workflow_checks.rs .github/workflows

Repository: Mesh-LLM/mesh-llm

Length of output: 13197


🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

echo "== workflow_checks outline =="
ast-grep outline tools/xtask/src/workflow_checks.rs --view expanded | sed -n '1,220p' || true

echo "== relevant checks implementation =="
sed -n '390,475p' tools/xtask/src/workflow_checks.rs

echo "== workflow parser/job section helpers =="
rg -n -C3 '(fn workflow_job_section|struct .*Yaml|serde|toml|yaml|Job|jobs:|container:)' tools/xtask/src/workflow_checks.rs

echo "== package deps for yaml parser =="
rg -n 'yaml|toml|serde|serde_yaml|kdl|ron' tools/Cargo.toml tools/xtask/Cargo.toml Cargo.toml

Repository: Mesh-LLM/mesh-llm

Length of output: 5432


🌐 Web query:

GitHub Actions jobs.<job_id>.container shorthand syntax container: node:18 allowed

💡 Result:

Yes, the shorthand syntax container: node:18 is valid and allowed in GitHub Actions [1][2][3]. When you only need to specify a container image, you can omit the image keyword and define the image directly as the value of the container key [2][3]. This is functionally equivalent to providing a configuration object with the image defined, such as: container: image: node:18 This shorthand is documented in official GitHub Actions resources as a valid way to simplify workflow configurations [2][3].

Citations:


Recognize shorthand container declarations.

The exact job_line == " container:" match skips supported declarations such as container: node:18, so shorthand container jobs miss the SCCACHE and safe-directory contract checks.

  • tools/xtask/src/workflow_checks.rs#L462-L475: detect any job-level line beginning with container:.
  • tools/xtask/src/workflow_checks.rs#L634-L672: add a valid shorthand-container fixture.
Proposed fix
-                .any(|job_line| job_line == "    container:")
+                .any(|job_line| job_line.strip_prefix("    container:").is_some())
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
fn release_container_job_names(release_workflow: &str) -> Vec<&str> {
release_workflow
.lines()
.filter_map(|line| {
let job_name = line.strip_prefix(" ")?.strip_suffix(':')?;
if job_name.is_empty() || job_name.starts_with(' ') || job_name.contains(' ') {
return None;
}
let job = workflow_job_section(release_workflow, job_name)?;
job.lines()
.any(|job_line| job_line == " container:")
.then_some(job_name)
})
.collect()
fn release_container_job_names(release_workflow: &str) -> Vec<&str> {
release_workflow
.lines()
.filter_map(|line| {
let job_name = line.strip_prefix(" ")?.strip_suffix(':')?;
if job_name.is_empty() || job_name.starts_with(' ') || job_name.contains(' ') {
return None;
}
let job = workflow_job_section(release_workflow, job_name)?;
job.lines()
.any(|job_line| job_line.strip_prefix(" container:").is_some())
.then_some(job_name)
})
.collect()
📍 Affects 1 file
  • tools/xtask/src/workflow_checks.rs#L462-L475 (this comment)
  • tools/xtask/src/workflow_checks.rs#L634-L672
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tools/xtask/src/workflow_checks.rs` around lines 462 - 475, Update
release_container_job_names to recognize job-level container declarations whose
lines begin with "    container:", including shorthand values such as container:
node:18, while preserving existing job filtering. In
tools/xtask/src/workflow_checks.rs lines 634-672, add a valid
shorthand-container fixture covering the SCCACHE and safe-directory contract
checks.

@i386 i386 changed the title Fix release container setup Fix release containers and restore sccache GHA caching Jul 24, 2026
@i386
i386 merged commit 47eced4 into main Jul 24, 2026
22 checks passed
@i386
i386 deleted the agent/fix-release-container-contracts branch July 24, 2026 03:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants