Add native Linux package building to CI without S3 upload#3993
Conversation
raramakr
force-pushed
the
users/raramakr/ci-pkgtest
branch
from
4d30fae to
92490fd
Compare
raramakr
requested review from
HereThereBeDragons,
jayhawk-commits and
nunnikri
raramakr
force-pushed
the
users/raramakr/ci-pkgtest
branch
2 times, most recently
from
e41e3f2 to
a7a2e62
Compare
raramakr
force-pushed
the
users/raramakr/ci-pkgtest
branch
from
a7a2e62 to
7f1ed9b
Compare
geomin12
left a comment
geomin12
left a comment
There was a problem hiding this comment.
oh, can we also add the gfx architecture to the job name? I realize it is building for different archs
Integrate native package building into the CI workflow by calling
build_native_linux_packages.yml from ci_linux.yml. Packages are built
for both deb and rpm formats but not uploaded to S3.
Changes:
- Add upload_packages input to build_native_linux_packages.yml
(defaults to true for backward compatibility)
- Make AWS CLI install, credentials config, and S3 upload steps
conditional on upload_packages flag
- Call build_native_linux_packages.yml from ci_linux.yml with
upload_packages: false to build packages during CI tests
- Build both deb and rpm packages via matrix strategy
- Use rocm_package_version and build_variant_suffix from inputs
This enables package validation during CI while keeping S3 uploads
limited to native packaging workflows triggered individually
The build_native_linux_packages workflow doesn't require access to repository secrets, so removing secrets: inherit to limit secret exposure.
raramakr
force-pushed
the
users/raramakr/ci-pkgtest
branch
from
57474fa to
a3fe208
Compare
Done |
ScottTodd
left a comment
ScottTodd
left a comment
There was a problem hiding this comment.
Thanks for expanding coverage on presubmit.
Here's some post-merge feedback (my review time is limited, but any work on CI workflows needs to be aligned with our plans, the cost of needing to make changes later is higher)
Please also tag related issues when doing work in these areas, #3177 or a sub-issue would apply here (and that issue should be updated now that this is merged)
There was a problem hiding this comment.
This workflow is deprecated. Please no new feature work here. Needs to go in https://github.com/ROCm/TheRock/blob/main/.github/workflows/multi_arch_ci_linux.yml instead/also
There was a problem hiding this comment.
We just switched the default pipeline from ci.yml to multi_arch_ci.yml today with #4204. This workflow now only runs on postsubmit or opt-in on PRs with the ci:run-non-multi-arch label.
| - name: Upload Package repo to S3 | ||
| if: ${{ !inputs.skip_package_upload }} | ||
| id: upload-packages | ||
| run: | | ||
| echo "Uploading to s3 bucket : ${{ inputs.release_type }}" | ||
| python ./build_tools/packaging/linux/upload_package_repo.py \ |
There was a problem hiding this comment.
We should upload to S3 on CI too.
We're planning on reworking the release (CD) workflows to be based on the workflows currently used for CI:
- Always upload to therock-ci-artifacts (or therock-ci-artifacts-external) bucket
- Release workflows copy those artifacts to a release bucket after testing them
| build_native_linux_packages: | ||
| needs: [build_portable_linux_artifacts] |
There was a problem hiding this comment.
This can be moved up in the workflow. Sort by dependency ordering, ROCm core first, then frameworks:
- build rocm artifacts
- test rocm artifacts
- build rocm native linux packages
- test rocm native linux packages
- build rocm python packages
- build pytorch python packages
- test pytorch python packages
- build jax python packages
- test jax python packages
There was a problem hiding this comment.
Thanks @ScottTodd for the feedback. This is exactly the plan for native packaging.
The test PR #3126 and multi arch PR #3561 and coming to shape. Once this is fully ready, we will get the same flow as you suggested. This current PR is a stop gap mechanism till these matures and hence the upload was skipped.
Integrate native package building into the CI workflow by calling build_native_linux_packages.yml from ci_linux.yml. Packages are built for both deb and rpm formats but not uploaded to S3. Changes: - Add upload_packages input to build_native_linux_packages.yml (defaults to true for backward compatibility) - Make AWS CLI install, credentials config, and S3 upload steps conditional on upload_packages flag - Call build_native_linux_packages.yml from ci_linux.yml with upload_packages: false to build packages during CI tests - Build both deb and rpm packages via matrix strategy - Use rocm_package_version and build_variant_suffix from inputs This enables package validation during CI while keeping S3 uploads limited to native packaging workflows triggered individually --------- Co-authored-by: raramakr <raramakr@amd.com>
4 participants
Integrate native package building into the CI workflow by calling
build_native_linux_packages.yml from ci_linux.yml. Packages are built
for both deb and rpm formats but not uploaded to S3.
Changes:
This enables package validation during CI while keeping S3 uploads
limited to native packaging workflows triggered individually