Skip to content

ci: disable sccache for Windows ROCm native runtime build - #1087

Merged
michaelneale merged 1 commit into
mainfrom
ci/windows-rocm-native-disable-sccache
Jul 26, 2026
Merged

ci: disable sccache for Windows ROCm native runtime build#1087
michaelneale merged 1 commit into
mainfrom
ci/windows-rocm-native-disable-sccache

Conversation

@michaelneale

@michaelneale michaelneale commented Jul 26, 2026

Copy link
Copy Markdown
Collaborator

What this fixes

The final blocker for a complete v0.74.0 GPU release. rc4 got every Linux CUDA/ROCm/Vulkan and Windows CUDA/Vulkan lane green (node24 + sccache-token-expiry both fixed in #1086), but one lane failed deterministically:

Build native runtime Windows x86_64 ROCm

sccache: error: failed to persist temporary file: The system cannot find the path specified. (os error 3)
ninja: build stopped: subcommand failed. exit code 127

Root cause

sccache + hipcc is broken on Windows for this build. HIP compiles one host action plus one device action per --offload-arch (9 arches here) and bundles them; combined with the long arch-derived build directory (build-stage-abi-dynamic-rocm-gfx90a_..._gfx1201), object paths exceed Windows MAX_PATH, and sccache fails the final atomic temp-file persist (os error 3).

Evidence this is ROCm-native-specific, not a general path/sccache/shell problem:

Lane Build dir sccache Result
Windows CUDA native short on
Windows Vulkan native short on
Windows ROCm release bundle build-stage-abi-rocm (short) on
Windows ROCm native runtime build-stage-abi-dynamic-rocm-<9 arches> (long) on

sccache's HIP support is known-fragile (PyTorch disables ccache/sccache for HIP for similar reasons).

Fix

Set LLAMA_STAGE_USE_SCCACHE=0 for the Windows ROCm native-runtime matrix row only. scripts/build-llama.sh already honors this by omitting the HIP compiler launcher, so the build proceeds without the caching layer that can't handle these paths. CUDA and Vulkan keep sccache unchanged (defaulted to 1 via matrix.use_sccache || '1').

Validation

  • actionlint -config-file .github/actionlint.yaml — pass
  • git diff --check — clean
  • cargo run -p xtask -- repo-consistency release-targets — pass

Real end-to-end proof is the next RC (rc5) once merged.

Rollback

  • Single revert; no other lanes affected. No repo variables changed, no images published.

Summary by CodeRabbit

  • Bug Fixes
    • Improved Windows GPU runtime release builds by applying the appropriate compiler cache settings for different GPU backends.
    • Disabled compiler caching for ROCm builds to improve build reliability.

sccache + hipcc is broken on Windows: HIP compiles one host action plus one
device action per --offload-arch and bundles them, and combined with the long
arch-derived build directory (build-stage-abi-dynamic-rocm-gfx90a_..._gfx1201)
the object paths exceed MAX_PATH. sccache then fails to persist its temp file
(os error 3) and the build aborts.

This only hits the Windows ROCm native-runtime row: Windows CUDA/Vulkan native
runtimes and the Windows ROCm release bundle (shorter build dir) all pass with
the same sccache config. Set LLAMA_STAGE_USE_SCCACHE=0 for that row only, which
build-llama.sh already honors by omitting the HIP compiler launcher. CUDA and
Vulkan keep sccache.
@coderabbitai

coderabbitai Bot commented Jul 26, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: ad041114-5222-46cc-b6c6-4c751499e92d

📥 Commits

Reviewing files that changed from the base of the PR and between 8d49450 and 331e727.

📒 Files selected for processing (1)
  • .github/workflows/release.yml

📝 Walkthrough

Walkthrough

The Windows GPU release workflow now disables sccache for the ROCm matrix entry while defaulting sccache to enabled for other native runtime packaging rows.

Changes

Windows GPU release configuration

Layer / File(s) Summary
Matrix-driven sccache selection
.github/workflows/release.yml
The ROCm matrix row sets use_sccache to '0', while the packaging step maps the matrix value to LLAMA_STAGE_USE_SCCACHE with '1' as the default.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Possibly related PRs

Suggested reviewers: ndizazzo, i386

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main CI change: disabling sccache for the Windows ROCm native runtime build.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ci/windows-rocm-native-disable-sccache

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@michaelneale
michaelneale merged commit bcd42f7 into main Jul 26, 2026
23 checks passed
@michaelneale
michaelneale deleted the ci/windows-rocm-native-disable-sccache branch July 26, 2026 19:26
michaelneale added a commit that referenced this pull request Jul 27, 2026
* origin/main:
  Fix Metal small-batch matmul parity for GLM verification (#1078)
  Handle K-only transposed KV page import and export (#1084)
  Refresh llama.cpp upstream patch queue (#1085)
  chore: improve embedded native-runtime compatibility guidance (#1043)
  fix(console-ui): chat transcript snapping during live status updates (#1083)
  ci: bump Linux CUDA slim container to gha-convention base runner image
  fix: record activation cache prefix identities (#1041)
  fix: read-only model download caches (#1042)
  ci: disable sccache for Windows ROCm native runtime build (#1087)
  ci: fix v0.74 release GPU builds (sccache disk-only + force_hosted_runners) (#1086)
  Make release sccache failures non-fatal (#1079)
  Keep client-only nodes out of model election (#1074)

# Conflicts:
#	crates/mesh-llm-host-runtime/src/runtime/auto_join.rs
#	crates/mesh-llm-host-runtime/src/runtime/tests/auto_join.rs
#	third_party/llama.cpp/patches/0004-Add-lanes-external-media-and-chat-grammar-support.patch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant