[raramakr]

Add build_stages allowlist input to filter CI build graph

Use case: Partial builds in downstream repos (rocm-systems) that only
need specific stages. For example, rocm-systems only needs
compiler-runtime - it doesn't need math-libs, ml-libs, comm-libs, etc.
With build_stages, the workflow skips unneeded stages, reducing CI time.

Add a build_stages allowlist input to setup_multi_arch.yml that lets
callers specify which stages to build. Stages outside the allowlist
are skipped entirely (no build, no artifact copy).

Changes:

• Add build_stages input to setup_multi_arch.yml and multi_arch_ci.yml
• Add skip_stages input to multi_arch_build_portable_linux.yml and
  multi_arch_build_windows.yml
• Add _resolve_skipped_stages() to convert allowlist to skip list
• Add CIInputs.validate() with fail-fast behavior: raises error if
  test labels require stages not in build_stages
• Disable package builds when stages are skipped (packages require
  complete artifacts; prebuilt/reused stages are OK)
• Fix is_asan check to include host-asan variant

ISSUE ID: #7202

Test job: https://github.com/ROCm/TheRock/actions/runs/33142065664

[therock-pr-bot]

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

[therock-pr-bot]

🎉 All checks passed! This PR is ready for review.

[ScottTodd]

skip_stages on its own LGTM, I don't think the validate_artifact_structure change should be needed though.

BTW, I added similar "skip arbitrary stages" to another CI system before, see https://iree.dev/developers/general/contributing/#ci-behavior-manipulation and the code like if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'test_android') in https://github.com/iree-org/iree/blob/main/.github/workflows/pkgci.yml

[geomin12]

the whole purpose of multi-arch is to re-use or re-build all of ROCm, so we can make sure downstream packaging (pytorch, python packages, tarball packaging) is not impact (so release packages don't break). if we skip, we won't be able to test downstream packages

i did exact work here: #6709 but aligned with team to understand the purpose of multi-arch and its importance

[raramakr]

the whole purpose of multi-arch is to re-use or re-build all of ROCm, so we can make sure downstream packaging (pytorch, python packages, tarball packaging) is not impact (so release packages don't break). if we skip, we won't be able to test downstream packages

i did exact work here: #6709 but aligned with team to understand the purpose of multi-arch and its importance

Thanks for the context on multi-arch's purpose. I agree that for release builds and others we need the full pipeline to validate downstream packaging.

The build_stages input is intended for specialized CI configurations, not to replace the default multi-arch flow. The primary use case is #7202: host-asan
asan-presubmit builds from rocm-systems only need compiler-runtime and runtime-tests

This allows callers to create focused ASAN presubmit workflows that:

• Build only the two required stages
• Run on every PR to catch memory errors early
• Stay cost-effective by skipping unrelated stages

The default behavior remains unchanged — callers that don't set build_stages still get the full multi-arch pipeline.

[ScottTodd]

the whole purpose of multi-arch is to re-use or re-build all of ROCm, so we can make sure downstream packaging (pytorch, python packages, tarball packaging) is not impact (so release packages don't break). if we skip, we won't be able to test downstream packages

We want both:

• the ability to build/test all of rocm, including frameworks like jax and torch
• the ability to build/test only a subset, e.g. media-libs, debug-tools

choosing between those will be "defcon" levels documented at https://github.com/ROCm/TheRock/blob/main/docs/rfcs/RFC0013-Consumer-Based-Test-Selection.md#background-gating-levels

see also the diagrams I posted originally at #3343 (comment)

in particular, case 3 looks like it could benefit from skipping entire build stages

• change just rocGDB
• use prebuilt artifacts for the compiler
• decide to limit testing to just debug-tools, don't opt-in to package builds and tests

[geomin12]

lgtm, based on Scott's diagram 3!

[ScottTodd]

Let's expose this to https://github.com/ROCm/TheRock/blob/main/.github/workflows/multi_arch_ci.yml for workflow_dispatch also, as that will

1. Let us test the feature itself without needing to edit other workflows, see https://github.com/ROCm/TheRock/blob/main/TESTING.md#github-actions-workflows---validation-methods

   If a workflow does not run on pull_request events, test manually with workflow_dispatch (https://docs.github.com/en/actions/how-tos/manage-workflow-runs/manually-run-a-workflow) and link the test runs in the pull request description.

2. Use this for https://github.com/ROCm/TheRock/blob/main/docs/development/ci_behavior_manipulation.md#workflow-dispatch. I'd like to use this to test just building compiler-runtime (or just the LLVM subproject actually, so a "build_subprojects" or "enable_subprojects" input that gets mapped to build stages and test subprojects could also be useful too)

---

I'm also looking over the current integration of these workflows into https://github.com/ROCm/rocm-libraries and https://github.com/ROCm/rocm-systems and I think we should consider changing https://github.com/ROCm/rocm-libraries/blob/develop/.github/workflows/therock-multi-arch-ci.yml from

jobs:
  ci-env:
     ...
  configure:
     ...
  resolve_therock_ref:
     ...
  setup:
    uses: ROCm/TheRock/.github/workflows/setup_multi_arch.yml@eb36dd0822a5e9a53c47fae4abad0b4a978fd6f5 # 08/21/2026
     ...
  linux_build_and_test:
    uses: ROCm/TheRock/.github/workflows/multi_arch_ci_linux.yml@eb36dd0822a5e9a53c47fae4abad0b4a978fd6f5 # 08/21/2026
     ...
  windows_build_and_test:
    uses: ROCm/TheRock/.github/workflows/multi_arch_ci_windows.yml@eb36dd0822a5e9a53c47fae4abad0b4a978fd6f5 # 08/21/2026
     ...
  multi_arch_ci_summary:
     ...

to:

jobs:
  multi_arch_ci:
    uses: ROCm/TheRock/.github/workflows/multi_arch_ci.yml@eb36dd0822a5e9a53c47fae4abad0b4a978fd6f5 # 08/21/2026

that would be less code to maintain in each repository, though it will require making all customized behaviors available to multi_arch_ci.yml (which will also let us centralize logging, testing, monitoring, etc.)

[jayhawk-commits]

I agree with the goal on having a singular uses call. That was my desired design for the original de-dupe effort on the legacy CI system.

[raramakr]

Exposed "build_stages" to multi_arch_ci.yml
https://github.com/ROCm/TheRock/actions/runs/33016801269

[raramakr]

@ScottTodd Can we merge this if it looks good? If any non-blocking changes are required, can we address them in a separate PR?

[ScottTodd]

This label processing code needs close review.

All CI configuration code should fit within the existing stages documented at the very top of this file:

TheRock/build_tools/github_actions/configure_multi_arch_ci.py

Lines 7 to 15 in c3621af

	This script is a pipeline of data transformations:
	1. Parse Inputs — read GitHub event context → CIInputs, GitContext
	2. Check Skip CI — gate: should we skip CI entirely?
	3. Select Targets — trigger type + labels → per-platform GPU families
	4. Decide Jobs — changed files + topology + targets → per-job-group
	decisions
	5. Build Configs — families × variant → per-platform build configs
	6. Write Outputs — JSON → GITHUB_OUTPUT + GITHUB_STEP_SUMMARY

Rather than add loose post-processing like this, it could live within the context of the other steps so decisions are applied in the expected order. Here is where labels are currently parsed:

TheRock/build_tools/github_actions/configure_multi_arch_ci.py

Lines 242 to 252 in c3621af

	# Test labels come from two sources:
	# 1. LINUX/WINDOWS_TEST_LABELS env vars (workflow_dispatch inputs)
	# 2. PR test:* labels (apply to both platforms)
	pr_test_labels = [label for label in pr_labels if label.startswith("test:")]
	linux_test_labels = (
	_parse_comma_list(os.environ.get("LINUX_TEST_LABELS", "")) + pr_test_labels
	)
	windows_test_labels = (
	_parse_comma_list(os.environ.get("WINDOWS_TEST_LABELS", ""))
	+ pr_test_labels
	)

Also note that labels are only one way that tests are selected, typically as part of opting in to running a known list of jobs (https://github.com/ROCm/TheRock/blob/main/docs/development/ci_behavior_manipulation.md#pull-request). If we compute the stages to skip dynamically (not just from fixed workflow inputs), we'll need to align both the build and test jobs to know about that. I don't think that's necessary yet here though.

If only handling fixed labels in workflow files or manual inputs, we could also treat mismatched labels as errors or just proceed and let test jobs fail. Fail fast behavior is preferred over silently falling back.

What behavior are you looking for out of the code here? How do you intend on using this? The PR description is pretty sparse on details so I'm left guessing.

[raramakr]

1. Fail-fast behavior: Changed from silent filtering to raising ValueError if test labels require stages not in build_stages. Incompatible labels now fail
   immediately during setup rather than at test runtime.
2. Code organization: Moved validation into CIInputs.validate(), called from from_environ() in step 1 "Parse Inputs" where labels are parsed. Removed the
   post-processing from configure().

[ScottTodd]

This build_tools/github_actions/configure_multi_arch_ci.py script is currently fully generic - it does not know anything about specific build stages or subproject names. I do think it should know about the github labels it handles (https://github.com/ROCm/TheRock/blob/main/docs/development/ci_behavior_manipulation.md#pull-request), but maybe we can find a separate file to know about these mappings, like https://github.com/ROCm/TheRock/blob/main/build_tools/github_actions/fetch_test_configurations.py?

WDYT @geomin12 ?

[raramakr]

@ScottTodd FYI, Added auto-selection of test labels when none are provided.
The refactor with fail-fast commit only validated explicit labels but didn't handle the case where no labels were specified - causing all tests to run and fail. Now when
build_stages is set:

• Explicit labels → fail-fast validation
• No labels → auto-select compatible tests from STAGE_TO_TEST_LABELS

Latest run passed with only kfdtest triggered for compiler-runtime stage. :
https://github.com/ROCm/TheRock/actions/runs/33142065664 (Latest test passed)

[ScottTodd]

Ok, looks good. As we get more sophisticated with enabling/disabling/reusing builds and tests I think we'll want to do some deeper refactoring here, to keep build_tools/github_actions/configure_multi_arch_ci.py more streamlined. It's starting to reach a complexity level that is difficult to follow.