Skip to content

ci: reuse unaffected build stages on pull requests - #7919

Closed
gulsumgudukbay wants to merge 1 commit into
mainfrom
users/gulsumgudukbay/ci-pr-stage-reuse
Closed

gulsumgudukbay wants to merge 1 commit into
mainfrom
users/gulsumgudukbay/ci-pr-stage-reuse

Conversation

@gulsumgudukbay

@gulsumgudukbay gulsumgudukbay commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

ISSUE ID : #3399

Split out of #7912 at @ScottTodd's request so the JAX Bazel cache and this change can be reviewed and rolled out separately. Draft: this is a repository-wide behaviour change that wants deliberate testing, and #7346 may be the better tool for the JAX/PyTorch case since it can reuse ROCm packages too.

Motivation

  • Every pull request rebuilds ROCm from source even when its diff cannot change ROCm.
  • The stage-reuse analysis already runs on every pull request: it maps the diff through BUILD_TOPOLOGY.toml, finds the unaffected stages, verifies their artifacts exist in a healthy main baseline, prints all of it to the step summary, and then rebuilds them anyway, because pull_request passes dry-run.
  • The beneficiaries are pull requests whose diff maps to a narrow source set, e.g. a change under one rocm-libraries/projects/<project>/, which today still rebuilds the full Linux and Windows graph.
  • Pull requests touching .github/, build_tools/, docs/ or scripts/ are not beneficiaries: stage_impact.full_ci_prefixes treats those as conservative full-CI triggers, so they keep rebuilding everything.

Technical Details

  • multi_arch_ci.yml passes reuse-stage for pull_request and keeps every other trigger on dry-run. Manual workflow_dispatch still wins.
  • Pushes deliberately keep rebuilding: a push that reused stages would stop producing the artifacts a later pull request reuses.
  • Flipping the mode alone would be close to a no-op. STAGE_REUSE_CURRENT_SHA was the checkout commit, which on a pull request is a merge commit that is not on the baseline branch, so validate_commit_compatibility returned unknown and rejected every candidate baseline.
  • setup_multi_arch.yml now establishes ancestry from the pull request base commit, which is on main and is by construction an ancestor of the merge commit, so the gate stays as conservative as it is for pushes.
  • Both gates are otherwise unchanged; reuse still fails safe toward a full build when no recent, healthy, commit-compatible baseline exists.

Test Plan

  • pytest build_tools/github_actions/tests for the new test and the existing stage-reuse and baseline-selection suites.
  • pre-commit run --all-files, including actionlint.
  • This pull request cannot exercise the change end to end: its own diff is entirely .github/, build_tools/ and docs/, which are conservative full-CI triggers. Demonstrating reuse needs a separate branch touching submodule source.

Test Result

  • Unit tests: 909 passed. One pre-existing failure unrelated to this change (post_stage_upload_test.py::TestCreateCcacheLogArchive, also fails on main in the same container). pre-commit: all hooks pass.
  • Both workflow edits take effect, per this PR's own setup job:
[STAGE-REUSE] mode=reuse-stage
STAGE_REUSE_CURRENT_SHA: 171a4748c9cf57d843a25eed683f9b871bfdd776   # a main commit, not the merge commit
  • Reuse then correctly declined to engage, because this PR's own paths are conservative full-CI triggers:
[STAGE-REUSE] conservative full rebuild: no stages eligible for reuse.
  reason: 'build_tools/github_actions/tests/stage_reuse_pr_defaults_test.py' matches a conservative full-CI trigger
  reason: 'docs/development/stage_reuse.md' matches a conservative full-CI trigger
- baseline run checked: _none_
  • Still unproven: baseline selection and the commit-compatibility gate never ran, so the base-commit fix has no end-to-end evidence yet. That needs a branch whose diff maps to a narrow source set.

Submission Checklist

@therock-pr-bot

therock-pr-bot Bot commented Sep 3, 2026 •

Copy link
Copy Markdown

✅ All Checks Passed — Ready for Review

Check Status Details
📝 PR Description ✅ Pass —
⛔ Forbidden Files ✅ Pass —
🧪 Unit Test ✅ Pass PR does not contain code files — Unit Test auto-passed
🔎 pre-commit ✅ Pass —
🚫 Draft PR 🔜 To Be Enabled —
🚩 Feature Flag 🔜 To Be Enabled —
📊 Code Coverage 🔜 To Be Enabled —
🤖 therock-pr-bot ✅ Pass —

🎉 All checks passed! This PR is ready for review.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

@therock-pr-bot

therock-pr-bot Bot commented Sep 3, 2026

Copy link
Copy Markdown

🎉 All checks passed! This PR is ready for review.

Multi-Arch CI computed stage-reuse decisions on every pull request and then
discarded them: dry-run reports which stages the diff leaves unaffected while
still rebuilding all of them. Pass reuse-stage for pull_request events so those
stages are copied from a baseline run instead, and keep every other trigger on
dry-run so pushes still produce the artifacts a pull request reuses.

A pull request also checks out a merge commit that is not on the baseline
branch, so validate_commit_compatibility could not place it in the branch
history and rejected every candidate baseline as unknown. Establish ancestry
from the pull request base commit instead.
@gulsumgudukbay
gulsumgudukbay force-pushed the users/gulsumgudukbay/ci-pr-stage-reuse branch from 0bc308b to cb175f0 Compare October 2, 2026 17:19
@github-project-automation github-project-automation Bot moved this from TODO to Done in TheRock Triage Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant