fix: speed up dependency-heavy container rebuilds

[ndizazzo]

Summary

• add stable architecture/toolchain-scoped BuildKit caches for apt, npm, pnpm, rustup, Cargo, Cargo target output, downloads, and sccache
• enable required sccache for the full llama.cpp compilation still present in the client, Fly, and WAN-lab container paths
• make sccache downloads atomic, checksum-verified, and self-healing
• replace git am --no-verify with a Bookworm-compatible hook override
• document the forced-recompile benchmark and add regression coverage

Measured impact

The same patched llama.cpp CPU/static source tree was compiled from clean build directories on Linux ARM64:

Sample	Compile time
Faster no-sccache control	121.2s
Cache population, 273 misses	125.8s
Warm sccache, 273 hits	3.55s

The warm compiler-cache path is 97.1% faster than the control, with zero cache errors.

Release-path audit

• just check-release passes the release-target consistency gate.
• Native runtime release jobs compile through prepare-native-runtime-input and package-native-runtime.sh.
• Static ABI release jobs directly run prepare-llama.sh and build-llama.sh.
• Native SDK release jobs consume the typed static ABI input and package through prepare-native-sdk-input.
• The Bookworm Git fix is therefore exercised by canonical Linux release compilation.
• The three Dockerfiles changed here are auxiliary container paths; canonical published package/image assembly remains in mesh-packaging.

Validation

• just check-release
• 115 focused cache and release-contract tests
• exact pinned llama preparation: 47/47 patches applied, patched SHA e43d604fb427e769ce3c09ebb462743e456d0c99
• ShellCheck, shell syntax, and actionlint
• BuildKit checks for all three Dockerfiles on amd64 and arm64
• forced compile benchmark with 273/273 warm sccache hits

Summary by CodeRabbit

• Performance

  • Improved container build times with architecture-aware caching for dependencies, toolchains, and build artifacts.
  • Added compiler-cache support for faster repeated llama and Rust builds.

• Documentation

  • Added ARM64 container build-cache benchmarks and performance comparisons.

• Reliability

  • Added verified, cache-aware installation for the compiler cache tool.
  • Improved patch application compatibility across Git versions.

• Tests

  • Added coverage for cache validation, recovery, reuse, and Git hook handling.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: cc7334ec-5af4-449d-a580-0e418e9df2ef

📥 Commits

Reviewing files that changed from the base of the PR and between b26b54d and 979d711.

📒 Files selected for processing (1)

• scripts/install-sccache.sh

🚧 Files skipped from review as they are similar to previous changes (1)

• scripts/install-sccache.sh

---

📝 Walkthrough

Walkthrough

Changes

The Dockerfiles add architecture-specific BuildKit caches for package managers, Rust, Cargo, build outputs, and sccache. A new installer validates and installs sccache archives. Llama patch preparation disables Git hooks through configuration. Tests and cache benchmarks document the behavior.

Build cache and reproducibility updates

Layer / File(s)	Summary
Validated sccache installation scripts/install-sccache.sh, scripts/tests/test_install_sccache.py	The installer validates architecture and version inputs, verifies cached or downloaded archives, installs sccache, and tests cache replacement, reuse, and cleanup.
Docker cache wiring docker/Dockerfile.client, docker/skippy-wan-lab/Dockerfile, fly/Dockerfile, docs/BUILD_CACHE_BENCHMARKS.md	The Dockerfiles configure architecture-specific package, toolchain, Cargo, build, and sccache caches. The benchmark document records cache timings and mount scoping.
Git hook handling for llama patches scripts/prepare-llama.sh, scripts/tests/test_prepare_llama.py	Llama patch application disables Git hooks through core.hooksPath. Tests verify that configured failing hooks are not executed.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant DockerBuild
  participant InstallSccache
  participant ReleaseArchive
  participant LlamaBuild
  DockerBuild->>InstallSccache: provide TARGETARCH and SCCACHE_VERSION
  InstallSccache->>ReleaseArchive: download and verify sccache archive
  InstallSccache->>DockerBuild: install sccache binary
  DockerBuild->>LlamaBuild: enable sccache with architecture-specific cache
  LlamaBuild->>DockerBuild: produce cached build outputs

Loading

Possibly related PRs

• Mesh-LLM/mesh-llm#1067: Related sccache configuration in release workflows and contract checks.
• Mesh-LLM/mesh-llm#1123: Related sccache cache-write policy and behavior.

Suggested reviewers: i386

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the primary change: speeding up dependency-heavy container rebuilds through caching improvements.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch agent/buildkit-cache-mounts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/install-sccache.sh`:
- Around line 30-42: Update both curl invocations in the download condition to
include explicit --connect-timeout and --max-time limits alongside --retry 3,
ensuring stalled connections and overall transfers terminate within bounded
durations while preserving the existing cleanup and failure behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 2430a352-776e-462c-ae7e-b21c1c80e8c5

📥 Commits

Reviewing files that changed from the base of the PR and between 3ff69b9 and b26b54d.

📒 Files selected for processing (8)

• docker/Dockerfile.client
• docker/skippy-wan-lab/Dockerfile
• docs/BUILD_CACHE_BENCHMARKS.md
• fly/Dockerfile
• scripts/install-sccache.sh
• scripts/prepare-llama.sh
• scripts/tests/test_install_sccache.py
• scripts/tests/test_prepare_llama.py

[i386]

still my beating heart