Skip to content

GHA workflow PoC to configure submodule remote, branch #594

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
amd-justchen wants to merge 10 commits into from
Closed

GHA workflow PoC to configure submodule remote, branch #594

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
5bc74b9
Add build workflow inputs to config submodule
amd-justchen May 10, 2025
5d1bc89
Add desc to amdgpu_families in build linux package
amd-justchen May 10, 2025
2deb92e
Add default git name and email to bump commit
amd-justchen May 10, 2025
8977331
Move name and email config args before git commit
amd-justchen May 10, 2025
02f5fa2
Use cmd git submodule update over fetch_sources.py
amd-justchen May 10, 2025
439fe21
Remove commented out fetch_sources.py comman
amd-justchen May 10, 2025
76b7471
Move submodule fetch own step and detect defaults
amd-justchen May 11, 2025
afb9a31
Remove trailing whitespace to satisfy pre-commit
amd-justchen May 11, 2025
c89f985
Remove trailing whitespace to satisfy pre-commit
amd-justchen May 11, 2025
4871fdf
Remove trailing whitespace to satisfy pre-commit
amd-justchen May 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 77 additions & 0 deletions .github/workflows/build_linux_packages.yml
View file
Open in desktop
Copy link
Copy Markdown
Member

@ScottTodd ScottTodd May 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

High level feedback: I see two main ways to build out #559 with GitHub Actions, and this PR currently sits a bit awkwardly between them, while I think we should lean strongly into one or the other.

  1. Modify an existing build/test workflow to edit the source tree as it runs.
  2. Create a branch with a modified source tree and run existing workflows on that branch.

For (1), you can either edit the source tree as you have it now, with commands like git submodule set-url or you can run fetch_sources.py first, then go into the submodule and make local changes using e.g. git fetch. Since the source tree is only used within the context of that workflow, there is no strong need to make those changes persistent (e.g. by writing patches to https://github.com/ROCm/TheRock/tree/main/patches or editing .gitmodules).

For (2), we would create branches like what @marbre has done for PRs like #532 (no patches affected) and #562 (one patch dropped).

  • The advantages to creating branches are that we can then run any workflows we want and any developer can also checkout that branch to try their own testing.
  • A disadvantage to creating a branch is that such branches can easily get orphaned and litter the repository. Branch naming, cleanup actions, or storing such branches in a fork could all help there.

I like option 2 for how well it composes with other development activities. If we took what you have for "set one subproject to this remote url and branch" and put it in a python script, we could use that script both "try jobs" (this PR) and for routine submodule updates.

That being said, option 1 is simpler so long as our build/test/etc. workflows are also simple. A try job that runs just

  1. change_subproject_ref.py
  2. fetch_sources.py
  3. cmake --build ...
    Should be pretty easy to reason about and maintain.

Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,16 @@ on:
default: ADHOCBUILD
amdgpu_families:
type: string
description: "(etc. gfx110X-dgpu)"
project_name:
type: string
description: "name of subproject (etc. hipBLASLt) to configure remote url and branch to fetch. (disable w/ null name and url)"
project_seturl:
type: string
description: "url of subproject (etc. https://github.com/ROCm/hipBLASLt) (disable w/ null name and url)"
project_branch:
type: string
description: "name of branch (etc. develop) to configure remote url and branch to fetch"
Comment on lines +12 to +20
Copy link
Copy Markdown
Member

@ScottTodd ScottTodd May 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's try to keep each workflow focused on one specific task. If there was one primary way to interface with the project then having a monolithic workflow could make sense, but we have a complex development model here with several ways to build/test/use artifacts. This new code could go in a workflow that composes with the existing workflows, to have:

  • modifying the source tree
  • building packages
  • running tests

With the "create a branch" approach mentioned in my other comment, we could have try_ci_from_remote.yml that does this:

  1. create a branch using inputs: project_name, project_url, project_ref (generalization of branch to also include commit hash or tag)
  2. Use something like https://github.com/benc-uk/workflow-dispatch (or a native way, or the GH CLI) to trigger ci.yml, build_linux_packages.yml, or whatever else we want using that branch
  3. Somehow wait for the results and report them back to the caller
  4. Delete the branch? Or leave cleanup to some other automation?


workflow_call:
inputs:
Expand All @@ -16,6 +26,16 @@ on:
default: ADHOCBUILD
amdgpu_families:
type: string
description: "(etc. gfx110X-dgpu)"
project_name:
type: string
description: "name of subproject (etc. hipBLASLt) to configure remote url and branch to fetch. (disable w/ null name and url)"
project_seturl:
type: string
description: "url of subproject (etc. https://github.com/ROCm/hipBLASLt) (disable w/ null name and url)"
project_branch:
type: string
description: "name of branch (etc. develop) to configure remote url and branch to fetch"

jobs:
build_linux_packages:
Expand Down Expand Up @@ -59,8 +79,65 @@ jobs:
restore-keys: |
linux-build-packages-manylinux-v2-

- name: Fetch submodule from remote
run: |
proj_name="${{ inputs.project_name }}"
proj_branch="${{ inputs.project_branch }}"
proj_seturl="${{ inputs.project_seturl }}"
Comment on lines +82 to +86
Copy link
Copy Markdown
Member

@ScottTodd ScottTodd May 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's move all this bash to a Python script so we can

  1. Reuse it between workflows
  2. Run it locally
  3. Add tests for it
  4. Add more sophisticated logging and error handling


echo "[*] Fetch submodule from remote using GHA inputs:"
echo " proj_name: [$proj_name]"
echo " proj_path: [$proj_submodule_path]"
echo " proj_seturl: [$proj_seturl]"
echo " proj_branch: [$proj_branch]"

#Only update submodule if set in github action inputs
if [ "$proj_name" ] || [ "$proj_seturl" ]; then
if [ ! "$proj_name" ]; then
proj_name=$(basename "$proj_seturl")
echo "[*] proj_name GHA Input not set, deriving from proj_seturl: [$proj_name]"
else
echo "[*] proj_name GHA Input set to: [$proj_name]"
fi

proj_submodule_path=$(git config --file .gitmodules --get submodule.$proj_name.path)

if [ "$proj_submodule_path" ]; then
echo "[*] Existing submodule config for [$proj_name]:"
git config --file .gitmodules --get-regexp ^submodule.$proj_name\.

echo "[*] Submodule status [$proj_name]:$(git submodule status -- $proj_submodule_path)"

echo "[*] Setting remote url and branch for submodule [$proj_name]"

[ "$proj_seturl" ] && git submodule set-url -- $proj_submodule_path $proj_seturl
[ "$proj_branch" ] && git submodule set-branch --branch $proj_branch -- $proj_submodule_path

echo "[*] Modified submodule config for [$proj_name]:"
git config --file .gitmodules --get-regexp ^submodule.$proj_name\.


echo "[*] Fetching latest submodule commit from remote"
git submodule sync -- $proj_submodule_path
git submodule update --init --remote --force -- $proj_submodule_path
if [ $? -gt 0 ]; then
echo "[-] Could not update submodule from remote. Aborting..."
else
echo "[+] Bump submodule [$proj_name]:$(git submodule status -- $proj_submodule_path)"
git \
-c user.name=therockbot \
-c user.email=therockbot@amd.com \
commit -a -m "Bump submodule $proj_name with its remote for parent project"
fi
else
echo "[-] Could not find submodule path. Aborting..."
fi
else
echo [*] No project name or url specicfied. Skipping submodule fetch...
fi
- name: Fetch sources
run: |
#Continue with normal fetching of sources
./build_tools/fetch_sources.py --jobs 12

- name: Install python deps
Expand Down
77 changes: 76 additions & 1 deletion .github/workflows/build_windows_packages.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,15 @@ on:
extra_cmake_options:
type: string
default: "-DBUILD_TESTING=ON"
project_name:
type: string
description: "name of subproject (etc. hipBLASLt) to configure remote url and branch to fetch. (disable w/ null name and url)"
project_seturl:
type: string
description: "url of subproject (etc. https://github.com/ROCm/hipBLASLt) (disable w/ null name and url)"
project_branch:
type: string
description: "name of branch (etc. develop) of subproject to fetch"

workflow_call:
inputs:
Expand All @@ -23,6 +32,16 @@ on:
extra_cmake_options:
type: string
default: "-DBUILD_TESTING=ON"
project_name:
type: string
description: "name of subproject (etc. hipBLASLt) to configure remote url and branch to fetch. (disable w/ null name and url)"
project_seturl:
type: string
description: "url of subproject (etc. https://github.com/ROCm/hipBLASLt) (disable w/ null name and url)"
project_branch:
type: string
description: "name of branch (etc. develop) of subproject to fetch"


jobs:
build_windows_packages:
Expand Down Expand Up @@ -113,10 +132,66 @@ jobs:
key: windows-build-packages-v3-${{ github.sha }}
restore-keys: |
windows-build-packages-v3-
- name: Fetch submodule from remote
run: |
proj_name="${{ inputs.project_name }}"
proj_branch="${{ inputs.project_branch }}"
proj_seturl="${{ inputs.project_seturl }}"

echo "[*] Fetch submodule from remote using GHA inputs:"
echo " proj_name: [$proj_name]"
echo " proj_path: [$proj_submodule_path]"
echo " proj_seturl: [$proj_seturl]"
echo " proj_branch: [$proj_branch]"

#Only update submodule if set in github action inputs
if [ "$proj_name" ] || [ "$proj_seturl" ]; then
if [ ! "$proj_name" ]; then
proj_name=$(basename "$proj_seturl")
echo "[*] proj_name GHA Input not set, deriving from proj_seturl: [$proj_name]"
else
echo "[*] proj_name GHA Input set to: [$proj_name]"
fi

proj_submodule_path=$(git config --file .gitmodules --get submodule.$proj_name.path)

if [ "$proj_submodule_path" ]; then
echo "[*] Existing submodule config for [$proj_name]:"
git config --file .gitmodules --get-regexp ^submodule.$proj_name\.

echo "[*] Submodule status [$proj_name]:$(git submodule status -- $proj_submodule_path)"

echo "[*] Setting remote url and branch for submodule [$proj_name]"

[ "$proj_seturl" ] && git submodule set-url -- $proj_submodule_path $proj_seturl
[ "$proj_branch" ] && git submodule set-branch --branch $proj_branch -- $proj_submodule_path

echo "[*] Modified submodule config for [$proj_name]:"
git config --file .gitmodules --get-regexp ^submodule.$proj_name\.


echo "[*] Fetching latest submodule commit from remote"
git submodule sync -- $proj_submodule_path
git submodule update --init --remote --force -- $proj_submodule_path
if [ $? -gt 0 ]; then
echo "[-] Could not update submodule from remote. Aborting..."
else
echo "[+] Bump submodule [$proj_name]:$(git submodule status -- $proj_submodule_path)"
git \
-c user.name=therockbot \
-c user.email=therockbot@amd.com \
commit -a -m "Bump submodule $proj_name with its remote for parent project"
fi
else
echo "[-] Could not find submodule path. Aborting..."
fi
else
echo [*] No project name or url specicfied. Skipping submodule fetch...
fi
- name: Fetch sources
run: |
python ./build_tools/fetch_sources.py --jobs 96
#Continue with normal fetching of sources
python ./build_tools/fetch_sources.py --jobs $(nproc --all)

- name: Checkout closed source AMDGPU/ROCm interop library folder
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
Expand Down
Loading