Skip to content

fix(ci): pull rocm-libraries DVC data on external rocm-systems builds #9544

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
geomin12 merged 1 commit into from
Oct 28, 2026
Merged

fix(ci): pull rocm-libraries DVC data on external rocm-systems builds #9544

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion build_tools/github_actions/detect_external_repo_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,10 @@
"cmake_source_var": "THEROCK_ROCM_SYSTEMS_SOURCE_DIR",
"submodule_path": "rocm-systems",
"skip_submodules": ["rocm-systems"],
"dvc_projects": ["external-rocm-systems"],
# --dvc-projects replaces fetch_sources.py's default project list rather
# than extending it, so "rocm-libraries" is listed explicitly to pull the
# rocSPARSE/hipSPARSE test matrices ClientMatrices.cmake needs.
"dvc_projects": ["external-rocm-systems", "rocm-libraries"],
Comment on lines -59 to +62

@ScottTodd ScottTodd Oct 27, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok if this works to unblock for now, but I'm not sure why we have this "replace the default project list for an external repo" setup in this way. We should be fetching

  • rocm-systems DVC files together with the rocm-systems submodule
  • rocm-libraries DVC files together with the rocm-libraries submodule

Here the config is "rocm-systems" but it now always will fetch rocm-libraries DVC files, even if the build stage is not doing anything with rocm-libraries. I believe we've had the same issue on the other side where rocm-libraries always fetches rocm-systems DVC files for all build stages.

We can also scope which DVC files are fetched based on which subprojects are being built by a particular stage as part of a sparse checkout, which will be especially useful when truly large files (e.g. tensile config files) move into DVC.

See https://github.com/ROCm/rocm-systems/actions/runs/40588237180/job/122486562625?pr=14418 for example, where Windows files and other files unrelated to the emulation build stage are fetched:

dvc config detected in /__w/rocm-systems/rocm-systems/external-rocm-systems, fetching large files
pull: 38 pointer file(s) from s3://therock-dvc/rocm-systems
  fetching  amd_comgr_2.dll (109.9 MiB)
  fetching  amdhip64_6.dll (17.9 MiB)
...

@geomin12 geomin12 Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agree, this hasn't been touched in months.... those scoping of dvc calls will definitely help with resources but i'll land this once math libs in the test PR pass and work on scoping as a follow-up

},
"rocgdb": {
"cmake_source_var": "THEROCK_ROCGDB_SOURCE_DIR",
Expand Down
8 changes: 5 additions & 3 deletions build_tools/github_actions/tests/dvc_external_repo_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,11 +36,13 @@ def test_rocm_systems_has_dvc_projects(self):
"rocm-systems config must have dvc_projects for wsl-rocdxg builds",
)

# Verify it contains external-rocm-systems
# "rocm-libraries" is required for the rocSPARSE/hipSPARSE test matrices:
# --dvc-projects replaces fetch_sources.py's default list rather than
# extending it, so dropping it breaks the math-libs stage.
self.assertEqual(
config["dvc_projects"],
["external-rocm-systems"],
"dvc_projects should contain external-rocm-systems path",
["external-rocm-systems", "rocm-libraries"],
"dvc_projects should contain external-rocm-systems and rocm-libraries",
)

def test_rocm_systems_has_all_required_keys(self):
Expand Down
Loading