GHA workflow PoC to configure submodule remote, branch#594
GHA workflow PoC to configure submodule remote, branch#594amd-justchen wants to merge 10 commits into
Conversation
There was a problem hiding this comment.
High level feedback: I see two main ways to build out #559 with GitHub Actions, and this PR currently sits a bit awkwardly between them, while I think we should lean strongly into one or the other.
- Modify an existing build/test workflow to edit the source tree as it runs.
- Create a branch with a modified source tree and run existing workflows on that branch.
For (1), you can either edit the source tree as you have it now, with commands like git submodule set-url or you can run fetch_sources.py first, then go into the submodule and make local changes using e.g. git fetch. Since the source tree is only used within the context of that workflow, there is no strong need to make those changes persistent (e.g. by writing patches to https://github.com/ROCm/TheRock/tree/main/patches or editing .gitmodules).
For (2), we would create branches like what @marbre has done for PRs like #532 (no patches affected) and #562 (one patch dropped).
- The advantages to creating branches are that we can then run any workflows we want and any developer can also checkout that branch to try their own testing.
- A disadvantage to creating a branch is that such branches can easily get orphaned and litter the repository. Branch naming, cleanup actions, or storing such branches in a fork could all help there.
I like option 2 for how well it composes with other development activities. If we took what you have for "set one subproject to this remote url and branch" and put it in a python script, we could use that script both "try jobs" (this PR) and for routine submodule updates.
That being said, option 1 is simpler so long as our build/test/etc. workflows are also simple. A try job that runs just
change_subproject_ref.pyfetch_sources.pycmake --build ...
Should be pretty easy to reason about and maintain.
| project_name: | ||
| type: string | ||
| description: "name of subproject (etc. hipBLASLt) to configure remote url and branch to fetch. (disable w/ null name and url)" | ||
| project_seturl: | ||
| type: string | ||
| description: "url of subproject (etc. https://github.com/ROCm/hipBLASLt) (disable w/ null name and url)" | ||
| project_branch: | ||
| type: string | ||
| description: "name of branch (etc. develop) to configure remote url and branch to fetch" |
There was a problem hiding this comment.
Let's try to keep each workflow focused on one specific task. If there was one primary way to interface with the project then having a monolithic workflow could make sense, but we have a complex development model here with several ways to build/test/use artifacts. This new code could go in a workflow that composes with the existing workflows, to have:
- modifying the source tree
- building packages
- running tests
With the "create a branch" approach mentioned in my other comment, we could have try_ci_from_remote.yml that does this:
- create a branch using inputs:
project_name,project_url,project_ref(generalization ofbranchto also include commit hash or tag) - Use something like https://github.com/benc-uk/workflow-dispatch (or a native way, or the GH CLI) to trigger
ci.yml,build_linux_packages.yml, or whatever else we want using that branch - Somehow wait for the results and report them back to the caller
- Delete the branch? Or leave cleanup to some other automation?
| - name: Fetch submodule from remote | ||
| run: | | ||
| proj_name="${{ inputs.project_name }}" | ||
| proj_branch="${{ inputs.project_branch }}" | ||
| proj_seturl="${{ inputs.project_seturl }}" |
There was a problem hiding this comment.
Let's move all this bash to a Python script so we can
- Reuse it between workflows
- Run it locally
- Add tests for it
- Add more sophisticated logging and error handling
|
@amd-justchen this seems stale. Please close if you do no longer to merge. I will remove myself as reviewer for now. |
Will do! Per #559 (comment) this has been implemented in a different PR
|
4 participants
Implementing #559
This is an initial proof of concept that enables specifying the remote url and branch of a submodule when running the build workflow for linux and windows.
As of now this can be tested when running these build workflows from
users/justchen/build-ci-config-submodule:https://github.com/ROCm/TheRock/actions/workflows/build_windows_packages.yml
https://github.com/ROCm/TheRock/actions/workflows/linux_windows_packages.yml
There are now 3 additional input fields:
To enable, either
name of subprojectorurl of subproject, or both can be inputted.If only
url...is inputted then thename of subprojectwill be derived from the url.if only
name...is inputted then it will use the existing remote url.similarly, if no branch is specified, the existing branch will be used.
Details can be found in the build logs under the
Fetch submodule from remotestep.Internally, proj_path has to be retrieved based on the project name.
Example of url specified but no project name:
Example with default url, with subproject name specified: