Skip to content

feat: Adding Artifact reuse and stage skip functionality to multi-arch CI - #6264

Merged
ammallya merged 26 commits into
mainfrom
users/ammallya/reuse-artifacts
Jul 21, 2026
Merged

ammallya merged 26 commits into
mainfrom
users/ammallya/reuse-artifacts

Conversation

@ammallya

@ammallya ammallya commented Jun 30, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Issue ID Issue 3399

TheROCK already has the building blocks for stage-aware prebuilt-artifact reuse:

  • stage_impact.py computes which build stages a changed-file set affects
  • baseline_runs.py selects a healthy, commit-compatible baseline workflow run and verifies it contains required artifacts
  • the prebuilt_stages skip mechanism is plumbed through the multi-arch workflows — each stage already guards on !contains(inputs.prebuilt_stages, 'stage-name')

This PR adds skip capability with a report-only default so we can watch the decisions across real runs before any stage is actually skipped, and flip a single switch (STAGE_REUSE_MODE=skip-stage) to turn skipping on.

Technical Details

New module build_tools/github_actions/stage_reuse_decision.py that reuses the existing tooling (no new impact or baseline logic) and applies two gates before a stage is reported reusable:

  1. Impact gate (stage_impact.analyze_stage_impact) — the change does not affect the stage → it is a candidate for reuse.
  2. Availability gate (baseline_runs.select_baseline_run) — a single healthy, commit-compatible baseline run actually contains the artifacts that stage would produce. This runs even in dry-run, so the report reflects what could really be skipped, not just what is logically unaffected.

Two-way STAGE_REUSE_MODE switch:

  • dry-run (default) — runs both gates and prints, per stage, [STAGE-REUSE] stage 'X' ... WOULD be skipped (dry-run) to the console and the step summary (including the baseline run id it would copy from), but returns no stages, so prebuilt_stages is unchanged and every stage still builds.
  • skip-stage — merges only the confirmed-available stages into the per-stage decisions so the orchestrator copies their artifacts and skips the build.

Safety:

  • The manual prebuilt_stages / baseline_run_id workflow_dispatch inputs are never touched — they keep working unchanged in every mode.
  • Conservative fallbacks reuse nothing: full-CI triggers (e.g. build_tools/, workflow, or topology changes), runs without a changed-file list (schedule / dispatch), no baseline found, or any baseline-lookup error.
  • Reporting and baseline lookup failures never fail CI configuration.

Test Plan

  • New unit tests in build_tools/github_actions/tests/stage_reuse_decision_test.py using an injected fake topology and an injected baseline selector:
    • mode parsing + default
    • dry-run verifies artifacts present; unaffected-but-artifacts-missing → rebuild; partial-family availability → rebuild
    • no baseline found → rebuild; baseline lookup error is safe
    • enforce applies only available stages
    • no-diff is conservative; step summary shows baseline + availability
  • Existing stage_impact_test.py, baseline_runs_test.py, and configure_multi_arch_ci_test.py continue to pass

Test Result

 python -m pytest build_tools/github_actions/tests/stage_reuse_decision_test.py -v
======================================================================================================================== test session starts =========================================================================================================================
platform linux -- Python 3.12.3, pytest-8.4.2, pluggy-1.6.0 -- /home/ergurses/TheRock/.venv/bin/python
cachedir: .pytest_cache
rootdir: /home/ergurses/TheRock/build_tools
configfile: pyproject.toml
plugins: cov-7.1.0, check-2.8.0
collected 23 items                                                                                                                                                                                                                                                   

build_tools/github_actions/tests/stage_reuse_decision_test.py::ModeParsingTest::test_default_is_dry_run PASSED                                                                                                                                                 [  4%]
build_tools/github_actions/tests/stage_reuse_decision_test.py::ModeParsingTest::test_explicit_modes PASSED                                                                                                                                                     [  8%]
build_tools/github_actions/tests/stage_reuse_decision_test.py::AvailabilityGateTest::test_baseline_lookup_error_is_safe PASSED                                                                                                                                 [ 13%]
build_tools/github_actions/tests/stage_reuse_decision_test.py::AvailabilityGateTest::test_dry_run_unaffected_but_artifacts_missing_rebuilds PASSED                                                                                                             [ 17%]
build_tools/github_actions/tests/stage_reuse_decision_test.py::AvailabilityGateTest::test_dry_run_verifies_artifacts_present PASSED                                                                                                                            [ 21%]
build_tools/github_actions/tests/stage_reuse_decision_test.py::AvailabilityGateTest::test_enforce_applies_only_available_stages PASSED                                                                                                                         [ 26%]
build_tools/github_actions/tests/stage_reuse_decision_test.py::AvailabilityGateTest::test_no_baseline_found_rebuilds_candidates PASSED                                                                                                                         [ 30%]
build_tools/github_actions/tests/stage_reuse_decision_test.py::AvailabilityGateTest::test_partial_family_availability_rebuilds PASSED                                                                                                                          [ 34%]
build_tools/github_actions/tests/stage_reuse_decision_test.py::GuardrailTest::test_full_ci_trigger_skips_baseline_query PASSED                                                                                                                                 [ 39%]
build_tools/github_actions/tests/stage_reuse_decision_test.py::GuardrailTest::test_no_diff_is_conservative PASSED                                                                                                                                              [ 43%]
build_tools/github_actions/tests/stage_reuse_decision_test.py::GuardrailTest::test_step_summary_shows_baseline_and_availability PASSED                                                                                                                         [ 47%]
build_tools/github_actions/tests/stage_reuse_decision_test.py::DefaultBaselineSelectorTest::test_empty_history_disables_commit_rule PASSED                                                                                                                     [ 52%]
build_tools/github_actions/tests/stage_reuse_decision_test.py::DefaultBaselineSelectorTest::test_history_fetch_error_disables_commit_rule PASSED                                                                                                               [ 56%]
build_tools/github_actions/tests/stage_reuse_decision_test.py::DefaultBaselineSelectorTest::test_history_is_fetched_and_threaded PASSED                                                                                                                        [ 60%]
build_tools/github_actions/tests/stage_reuse_decision_test.py::DefaultBaselineSelectorTest::test_no_sha_means_no_history_fetch PASSED                                                                                                                          [ 65%]
build_tools/github_actions/tests/stage_reuse_decision_test.py::PlatformAwareAvailabilityTest::test_no_platforms_selected_disables_auto_reuse PASSED                                                                                                            [ 69%]
build_tools/github_actions/tests/stage_reuse_decision_test.py::PlatformAwareAvailabilityTest::test_single_platform_default_is_linux PASSED                                                                                                                     [ 73%]
build_tools/github_actions/tests/stage_reuse_decision_test.py::PlatformAwareAvailabilityTest::test_stage_reused_only_when_available_on_all_platforms PASSED                                                                                                    [ 78%]
build_tools/github_actions/tests/stage_reuse_decision_test.py::PlatformAwareAvailabilityTest::test_stage_reused_when_present_on_both_platforms PASSED                                                                                                          [ 82%]
build_tools/github_actions/tests/stage_reuse_decision_test.py::PlanStageReuseTest::test_plan_none_changed_files_is_full_rebuild PASSED                                                                                                                         [ 86%]
build_tools/github_actions/tests/stage_reuse_decision_test.py::PlanStageReuseTest::test_plan_returns_candidates_without_baseline PASSED                                                                                                                        [ 91%]
build_tools/github_actions/tests/stage_reuse_decision_test.py::TargetFamiliesTest::test_always_includes_generic PASSED                                                                                                                                         [ 95%]
build_tools/github_actions/tests/stage_reuse_decision_test.py::TargetFamiliesTest::test_dedupes_and_appends_generic PASSED                                                                                                                                     [100%]

========================================================================================================================= 23 passed in 0.06s =========================================================================================================================
=== Deciding job configuration ===
[STAGE-REUSE] mode=dry-run
[STAGE-REUSE] conservative full rebuild: no stages eligible for reuse.
[STAGE-REUSE]   reason: 'build_tools/github_actions/configure_multi_arch_ci.py' matches a conservative full-CI trigger
[STAGE-REUSE]   reason: 'build_tools/github_actions/stage_reuse_decision.py' matches a conservative full-CI trigger
[STAGE-REUSE]   reason: 'build_tools/github_actions/tests/stage_reuse_decision_test.py' matches a conservative full-CI trigger

Writing job summary:

### Stage reuse analysis

- mode: `dry-run`
- full rebuild required: `True`
- baseline run checked: _none_
- unaffected candidates: _none_
- available in baseline: _none_
- applied: _none_
- reasons:
  - 'build_tools/github_actions/configure_multi_arch_ci.py' matches a conservative full-CI trigger
  - 'build_tools/github_actions/stage_reuse_decision.py' matches a conservative full-CI trigger
  - 'build_tools/github_actions/tests/stage_reuse_decision_test.py' matches a conservative full-CI trigger

Submission Checklist

@therock-pr-bot

therock-pr-bot Bot commented Jun 30, 2026 •

Copy link
Copy Markdown

✅ All Checks Passed — Ready for Review

Check Status Details
🌿 Branch Name ✅ Pass —
📝 PR Title/Description ✅ Pass —
⛔ Forbidden Files ✅ Pass —
🧪 Unit Test ✅ Pass —
🔎 pre-commit ✅ Pass —
🚫 Draft PR 🔜 To Be Enabled —
🚩 Feature Flag 🔜 To Be Enabled —
📊 Code Coverage 🔜 To Be Enabled —
🤖 therock-pr-bot ✅ Pass —

🎉 All checks passed! This PR is ready for review.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

@therock-pr-bot therock-pr-bot Bot added the Not ready to Review PR has unresolved policy failures — reviews blocked label Jun 30, 2026
@therock-pr-bot

therock-pr-bot Bot commented Jun 30, 2026 •

Copy link
Copy Markdown

🎉 All checks passed! This PR is ready for review.

@therock-pr-bot therock-pr-bot Bot removed the Not ready to Review PR has unresolved policy failures — reviews blocked label Jun 30, 2026
@ammallya
ammallya requested a review from Copilot June 30, 2026 19:54
@therock-pr-bot therock-pr-bot Bot added the Not ready to Review PR has unresolved policy failures — reviews blocked label Jun 30, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Introduces an automatic “stage reuse” decision layer for the multi-arch CI setup so that unaffected build stages can be satisfied by copying artifacts from a compatible baseline workflow run. The feature is gated behind STAGE_REUSE_MODE, defaulting to report-only (dry-run) and enabling actual skipping in skip-stage.

Changes:

  • Added stage_reuse_decision.py to compute candidate stages, verify baseline artifact availability, and generate console/step-summary reporting.
  • Integrated auto stage-reuse evaluation into configure_multi_arch_ci.py, merging eligible stages into prebuilt_stages when enabled.
  • Plumbed a new stage_reuse_mode workflow input through the setup and multi-arch CI workflows.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
build_tools/github_actions/stage_reuse_decision.py New logic for stage impact + baseline verification + reporting for stage reuse.
build_tools/github_actions/configure_multi_arch_ci.py Invokes stage reuse logic and merges results into per-stage build decisions.
.github/workflows/setup_multi_arch.yml Adds stage_reuse_mode input and exports related env vars for the setup step.
.github/workflows/multi_arch_ci.yml Adds stage_reuse_mode input and passes it into the setup workflow.
Comments suppressed due to low confidence (1)

build_tools/github_actions/configure_multi_arch_ci.py:675

  • In skip-stage mode this code marks stages as PREBUILT, but it does not set baseline_run_id from the selected baseline run. That will cause downstream workflows to skip build stages without copying artifacts (copy step is gated on baseline_run_id != '').
    # Only skip-stage mode returns non-empty applied_reuse_stages.
    for stage in auto.applied_reuse_stages:
        stage_decisions.setdefault(stage, JobAction.PREBUILT)
    build_rocm = BuildRocmDecision(
        action=JobAction.RUN,
        stage_decisions=stage_decisions,
        baseline_run_id=ci_inputs.baseline_run_id,
    )

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +657 to +661
auto = compute_auto_stage_reuse(
changed_files=git_context.changed_files,
mode=StageReuseMode.from_environ(),
target_families=target_families,
)
Comment on lines +121 to +130
def compute_auto_stage_reuse(
*,
changed_files: Sequence[str] | None,
mode: StageReuseMode,
platform: str | None = None,
target_families: Sequence[str] = (),
topology=None,
baseline_selector: BaselineSelector | None = None,
) -> AutoStageReuse:
"""Compute auto stage-reuse decisions, verified against a baseline run."""
@therock-pr-bot therock-pr-bot Bot removed the Not ready to Review PR has unresolved policy failures — reviews blocked label Jun 30, 2026

@erman-gurses erman-gurses left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me overall - please see my comments.

Comment thread .github/workflows/setup_multi_arch.yml Resolved
render_step_summary,
)
target_families = _stage_reuse_target_families(ci_inputs)
auto = compute_auto_stage_reuse(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

compute_auto_stage_reuse() now performs stage-impact analysis, baseline selection, artifact verification, and reporting. Have we considered separating the planning portion from the reporting/execution plumbing to keep the decision logic independently reusable?

Comment on lines +650 to +654
from stage_reuse_decision import (
StageReuseMode,
compute_auto_stage_reuse,
render_step_summary,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason these imports are local rather than module-level? If it's to avoid import cycles, it might be worth adding a short comment.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope - follow the style guide. These imports must be moved to the top. https://github.com/ROCm/TheRock/blob/main/docs/development/style_guides/python_style_guide.md#import-organization

No import cycles here. If there are cycles we should break them instead of use local imports.

Comment thread .github/workflows/setup_multi_arch.yml Outdated
Comment on lines +162 to +164
STAGE_REUSE_CURRENT_SHA: ${{ steps.checkout.outputs.commit }}
STAGE_REUSE_MAX_AGE_HOURS: "72"
STAGE_REUSE_COMMIT_HISTORY: "50"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are becoming part of the stage-reuse interface - will make sense to document this environment contract in the module docstring.

@erman-gurses erman-gurses changed the title [DRAFT: DO NOT REVIEW] Adding Artifact reuse and stage skip functionality to multi-arch CI feat: Adding Artifact reuse and stage skip functionality to multi-arch CI Jul 1, 2026
@erman-gurses
erman-gurses requested a review from geomin12 July 1, 2026 20:50
Comment thread .github/workflows/multi_arch_ci.yml Resolved
Comment thread .github/workflows/setup_multi_arch.yml Resolved
PREBUILT_STAGES: ${{ inputs.prebuilt_stages }}
BASELINE_RUN_ID: ${{ inputs.baseline_run_id }}
STAGE_REUSE_MODE: ${{ inputs.stage_reuse_mode }}
STAGE_REUSE_CURRENT_SHA: ${{ steps.checkout.outputs.commit }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can actaully get this from ${{ github.sha }} (current sha)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It depends on what commit ref you want.

steps.checkout.outputs.commit gets the TheRock commit even when running in rockrel (and maybe rocm-libraries / rocm-systems?), see the workflow output:

For runs in rocm-libraries and rocm-systems, we may want to use baseline runs from those repositories instead of TheRock though.

Ultimately I think the best artifact reuse we'll be able to get will be some query against a database where we say "here are the artifact fingerprints that my current source code produces, fetch me artifacts for any with matching fingerprints from the database", which we'd want to abstract away from the github actions runs API entirely. We wouldn't need to scan commit history or workflow runs with that architecture.

@therock-pr-bot

therock-pr-bot Bot commented Jul 6, 2026

Copy link
Copy Markdown

Pre-commit check failed

⛔ pre-commit failed

Please run locally:

  • python -m pip install pre-commit
  • pre-commit install
  • pre-commit run --all-files --show-diff-on-failure

This repo uses .pre-commit-config.yaml.

@erman-gurses erman-gurses left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks all of the comments were addressed or added TODOs - Please see my comment below:

@erman-gurses
erman-gurses requested a review from geomin12 July 6, 2026 22:15
@erman-gurses
erman-gurses force-pushed the users/ammallya/reuse-artifacts branch from 8e94a55 to 397e824 Compare July 7, 2026 20:33
@erman-gurses
erman-gurses force-pushed the users/ammallya/reuse-artifacts branch from 5dc1564 to e7086a2 Compare July 7, 2026 20:36
@lumachad

lumachad commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Have we considered the case where we have ASAN build stages and we want to reuse those as well?

@marbre
marbre requested a review from ScottTodd July 8, 2026 10:32
@marbre

marbre commented Jul 8, 2026

Copy link
Copy Markdown
Member

@ammallya, I only glanced at this but I know @ScottTodd worked on a prototype (#3399 (comment)) and thus he has opinion for sure and it would be valuable to wait for his review before this is merged.

@ScottTodd ScottTodd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The general approach taken here and plumbing through the scripts is reasonable. I spotted quite a few issues with the specific changes though. Please watch for these coding errors more carefully (local review iterations via coding agents can help there)

Do you have a plan for which sorts of PR runs to first get working with prebuilt artifacts? If you are aiming to get rocm-systems and rocm-libraries using them, a faster path would be to directly set which workflow runs and prebuilt artifacts to use in the workflows whenever the TheRock commit is updated, as that can use the existing manual inputs and none of this extra code:

  • rocm-libraries can use a prebuilt compiler-runtime stage
  • rocm-systems with changes to hip/CLR unfortunately can't reuse artifacts this way since the compiler and runtimes are together in a stage, we'd need finer-grained artifact fprint-based reuse for that
  • rocm-systems with changes to RCCL can reuse all but the comm-libs stage
  • rocm-systems with changes to the "storage libs" / "profiler apps" / etc. stages can reuse other stages

Comment thread .github/workflows/multi_arch_ci.yml Resolved
PREBUILT_STAGES: ${{ inputs.prebuilt_stages }}
BASELINE_RUN_ID: ${{ inputs.baseline_run_id }}
STAGE_REUSE_MODE: ${{ inputs.stage_reuse_mode }}
STAGE_REUSE_CURRENT_SHA: ${{ steps.checkout.outputs.commit }}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It depends on what commit ref you want.

steps.checkout.outputs.commit gets the TheRock commit even when running in rockrel (and maybe rocm-libraries / rocm-systems?), see the workflow output:

For runs in rocm-libraries and rocm-systems, we may want to use baseline runs from those repositories instead of TheRock though.

Ultimately I think the best artifact reuse we'll be able to get will be some query against a database where we say "here are the artifact fingerprints that my current source code produces, fetch me artifacts for any with matching fingerprints from the database", which we'd want to abstract away from the github actions runs API entirely. We wouldn't need to scan commit history or workflow runs with that architecture.

Comment thread docs/development/stage_reuse.md Resolved
Comment on lines +650 to +654
from stage_reuse_decision import (
StageReuseMode,
compute_auto_stage_reuse,
render_step_summary,
)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope - follow the style guide. These imports must be moved to the top. https://github.com/ROCm/TheRock/blob/main/docs/development/style_guides/python_style_guide.md#import-organization

No import cycles here. If there are cycles we should break them instead of use local imports.

@lumachad

Copy link
Copy Markdown
Contributor

The general approach taken here and plumbing through the scripts is reasonable. I spotted quite a few issues with the specific changes though. Please watch for these coding errors more carefully (local review iterations via coding agents can help there)

Do you have a plan for which sorts of PR runs to first get working with prebuilt artifacts? If you are aiming to get rocm-systems and rocm-libraries using them, a faster path would be to directly set which workflow runs and prebuilt artifacts to use in the workflows whenever the TheRock commit is updated, as that can use the existing manual inputs and none of this extra code:

Shameless plug, but debug-tools is fairly simple to get working with prebuilt artifacts. It only depends on compiler-runtime. I could help test that setup (I've been using some local scripting to do that).

@ScottTodd ScottTodd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes. No more major concerns from me. Having this default to dry-run will let us evaluate it and then choose when to enable. Notably we won't want to reuse artifacts for release builds (the release workflows already don't use that data).

Comment on lines +491 to +504
class BuildPlatformsTest(unittest.TestCase):
def test_no_families_is_empty(self):
self.assertEqual(srd._build_platforms((), ()), ())

def test_linux_only(self):
self.assertEqual(srd._build_platforms(["gfx94x"], ()), ("linux",))

def test_windows_only(self):
self.assertEqual(srd._build_platforms((), ["gfx110x"]), ("windows",))

def test_both_platforms(self):
self.assertEqual(
srd._build_platforms(["gfx94x"], ["gfx110x"]), ("linux", "windows")
)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are these tests checking?

The test cases are pretty hard to read with the function call and checks inlined and positional arguments being used instead of named keyword arguments.

@ammallya
ammallya merged commit ee7bf10 into main Jul 21, 2026
147 of 154 checks passed
@ammallya
ammallya deleted the users/ammallya/reuse-artifacts branch July 21, 2026 17:05
@github-project-automation github-project-automation Bot moved this from TODO to Done in TheRock Triage Jul 21, 2026
@ScottTodd

Copy link
Copy Markdown
Member

Something in here nearly doubled the time it takes to run project unit tests, see #6752

locally I see this with pytest build_tools/github_actions/tests/ -vv --durations=0:
240.15s call github_actions/tests/configure_multi_arch_ci_test.py::TestFamilyTestFilters::test_submodule_bump_tests_only_enables_tests_with_submodule_changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

8 participants