Add sccache to manylinux PyTorch build image#3369
Merged
subodh-dubey-amd merged 4 commits intoFeb 12, 2026
Merged
Conversation
marbre
reviewed
Feb 11, 2026
Comment on lines
+13
to
+25
| # Map architecture to sccache release naming convention | ||
| case "${ARCH}" in | ||
| x86_64) | ||
| SCCACHE_ARCH="x86_64-unknown-linux-musl" | ||
| ;; | ||
| aarch64) | ||
| SCCACHE_ARCH="aarch64-unknown-linux-musl" | ||
| ;; | ||
| *) | ||
| echo "Unsupported architecture: ${ARCH}" | ||
| exit 1 | ||
| ;; | ||
| esac |
Member
There was a problem hiding this comment.
We don't support aarch64 thus you can limit to x86_64.
Member
There was a problem hiding this comment.
Why not just yum install sccache?
Contributor
Author
There was a problem hiding this comment.
I checked: sccache isn’t in EPEL 8/9 (manylinux_2_28 is RHEL 8–based). Only ccache is there. Fedora has rust-sccache but that’s not in EPEL, so we’re installing the official GitHub release to get a current sccache (e.g. 0.14.0). See https://repology.org/project/sccache/versions — there’s no EPEL 8/9 entry for sccache.
ScottTodd
approved these changes
Feb 11, 2026
subodh-dubey-amd
deleted the
users/subodh-dubey-amd/sccache-dockerfiles
branch
1 task
lumachad
added a commit
that referenced
this pull request
Feb 12, 2026
…90baef3e8dde4805858 Incorporate changes made in #3369.
Contributor
|
Image bump in #3306 should cover this PR as well. |
marbre
pushed a commit
that referenced
this pull request
Feb 12, 2026
1 task
subodh-dubey-amd
added a commit
that referenced
this pull request
Feb 20, 2026
## Motivation Add sccache support to PyTorch wheel builds for S3-backed distributed caching. Script placed in `build_tools/` per [reviewer feedback](#3171 (comment)), modeled after `build_tools/setup_ccache.py`. Part of sccache PR sequence: [#3369](#3369) → [#3389](#3389) → **this** → workflow wiring. ## Technical Details - **New: `build_tools/setup_sccache_rocm.py`** — generic sccache ROCm helper (CLI + importable): - `find_sccache()` — locate binary; hard fail if missing - `setup_rocm_sccache()` — wrap clang/clang++ with sccache stubs (Linux only) - `restore_rocm_compilers()` — undo wrapping - **Modified: `external-builds/pytorch/build_prod_wheels.py`**: - `--use-ccache` / `--use-sccache` mutually exclusive args - Both hard-fail with `RuntimeError` if the requested cache tool is not found ([per review](#3171 (comment))) — no silent fallback - Added explicit ccache availability check (previously would fail with an unclear subprocess error) - sccache: wrap compilers → set CMAKE launchers → `try`/`finally` around build for guaranteed compiler restore + stats - Moved ccache stats reporting into `finally` block for consistent reporting on both success and failure ## Test Result No workflow changes — sccache wired but not yet invoked by CI (next PR adds `cache_type` input + AWS config). ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.
This was referenced Feb 20, 2026
subodh-dubey-amd
added a commit
that referenced
this pull request
Mar 5, 2026
## Summary Adds `sccache` with S3 remote storage to all four PyTorch wheel build workflows, significantly reducing build times through distributed compiler caching. **PR sequence:** #3369 → #3306 → #3389 → #3482 → **this** → #3189 ([based on Reviewer's Feedback](#3171 (comment))) ## How It Works | | Linux | Windows | |---|---|---| | **Host C/C++** | CMake compiler launchers | CMake compiler launchers | | **HIP device code** | Wraps ROCm `clang`/`clang++` with sccache | Not supported | | **Cleanup** | Restores original compilers via try/finally | N/A | Cache is stored in the `therock-<workflow>-pytorch-sccache` S3 bucket, keyed by `<os>/<arch>/` prefix. ## S3 Cache Configuration Each workflow uses a dedicated S3 bucket and IAM role, keyed by `<os>/<arch>/` prefix: | Workflow | S3 Bucket | IAM Role | |----------|-----------|----------| | Linux CI | `therock-ci-pytorch-sccache` | `therock-ci` | | Windows CI | `therock-ci-pytorch-sccache` | `therock-ci` | | Linux Release | `therock-{release_type}-pytorch-sccache` | `therock-{release_type}` | | Windows Release | `therock-{release_type}-pytorch-sccache` | `therock-{release_type}` | Where `release_type` is one of: `dev`, `nightly`, `prerelease`. ## Impact | Platform | Cold → Warm | Improvement | |----------|------------|-------------| | Linux | ~70m → ~37m | **~49%** | | Windows | ~42m → ~26m | **~38%** | Windows is lower — sccache cannot wrap HIP device compilation on Windows, only host C/C++ via CMAKE launchers. ## Tests ### Linux: - [Linux (Cache Population)](https://github.com/ROCm/TheRock/actions/runs/22226347964/job/64293924748) - 70 mins - [Linux (Cache Hit)](https://github.com/ROCm/TheRock/actions/runs/22231743387/job/64312966557) - 37 mins ### Windows: - [Windows (Cache Population)](https://github.com/ROCm/TheRock/actions/runs/22219252671/job/64280583887) - 42 mins - [Windows (Cache Hit)](https://github.com/ROCm/TheRock/actions/runs/22223608689/job/64284721704) - 26 mins ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.' > Forks: S3 caching is only active for ROCm-owned runs. Fork users can set cache_type to ccache or none, or leave the default — sccache will work locally without S3 access. ---------
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Add sccache to the manylinux x86_64 Docker image used for building portable Linux PyTorch wheels so later PRs can use sccache (e.g. S3-backed) for faster rebuilds without changing workflows or build scripts in this PR.
Technical Details
dockerfiles/install_sccache.sh– installs sccache from mozilla/sccache GitHub releases (v0.14.0; x86_64 and aarch64).dockerfiles/build_manylinux_x86_64.Dockerfile– adds an SCCache install step after the existing CCache block (WORKDIR, COPY, RUN).Test Plan
docker buildx build --file dockerfiles/build_manylinux_x86_64.Dockerfile dockerfiles/.sccache --versionand confirm 0.14.0.Test Result
Submission Checklist