New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable ROCM in CI #999

Merged

amdfaa merged 4 commits into main from msaroufim-patch-21

Jan 17, 2025

Member

msaroufim commented Oct 3, 2024 •

edited by jithunnair-amd

Loading

Salient points:

Add ROCm nightly wheels to test matrix
Refactor conda-builder -> almalinux-builder pytorch#140157
The above PR shows that we've migrated to almalinux-builder due to the EOL CENTOS 7. Changes to regression_test.yml to not install devtoolset-10 have been made in accordance with this switch.
Fix bug in torchao/utils.py in invocation of torch.cuda.get_device_properties()

Needs changes in pytorch/test-infra#6104

pytorch-bot bot commented Oct 3, 2024 •

edited

Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/999

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 1 Pending

As of commit 593fb78 with merge base cedadc7 ():

NEW FAILURES - The following jobs have failed:

Run Regression Tests / test (CUDA 2.4, linux.g5.12xlarge.nvidia.gpu, torch==2.4.0, cuda, 12.1) / linux-job (gh)
test/quantization/test_quant_api.py::TestQuantFlow::test_quantized_tensor_subclass_int8_dyn_quant
Run Regression Tests / test (CUDA 2.5.1, linux.g5.12xlarge.nvidia.gpu, torch==2.5.1 --index-url https://download.pytorch... / linux-job (gh)
test/quantization/test_quant_api.py::TestQuantFlow::test_quantized_tensor_subclass_int8_dyn_quant
Run Regression Tests / test-nightly (ROCM Nightly, linux.rocm.gpu.2, --pre torch --index-url https://download.pytorch.or... / linux-job (gh)
test/sparsity/test_marlin.py::SparseMarlin24::test_quant_sparse_marlin_layout_eager

This comment was automatically generated by Dr. CI and updates every 15 minutes.

pytorch-bot bot added ciflow/rocm module: rocm labels

facebook-github-bot added the CLA Signed label

pytorch-bot bot commented Oct 3, 2024

No ciflow labels are configured for this repo.
For information on how to enable CIFlow bot see this wiki

Member Author

msaroufim commented Oct 9, 2024

@atalman im not sure the no-sudo flag does anything. Tried a few variants for the value like true or "true" and same result

atalman reviewed

View reviewed changes

.github/workflows/regression_test.yml Outdated Show resolved Hide resolved

atalman reviewed

View reviewed changes

.github/workflows/regression_test.yml Outdated Show resolved Hide resolved

atalman reviewed

View reviewed changes

.github/workflows/regression_test.yml Outdated Show resolved Hide resolved

This was referenced Oct 15, 2024

Enable Intel GPU #753

Closed

Multibackend tracker #1082

Open

atalman mentioned this pull request

Add support for Rocm Runners to linux_job pytorch/test-infra#5766

Open

Collaborator

amdfaa commented Nov 19, 2024

@pytorchbot rebase

1 similar comment

Collaborator

amdfaa commented Nov 19, 2024

@pytorchbot rebase

amdfaa mentioned this pull request

Enable linux_job.yml workflow for ROCm pytorch/test-infra#6003

Closed

amdfaa reviewed

View reviewed changes

.github/workflows/regression_test.yml Outdated

+                    docker-image: ${{ matrix.gpu-arch-type == 'rocm' && format('pytorch/manylinux2_28-builder:{0}{1}',
+                                                                                    matrix.gpu-arch-type,
+                                                                                    matrix.gpu-arch-version)
+                                                                         || 'pytorch/almalinux-builder' }}

Collaborator

amdfaa Dec 5, 2024

pytorch/pytorch#140157
We've migrated to almalinux-builder due to the EOL CENTOS 7.

amdfaa reviewed

View reviewed changes

.github/workflows/regression_test.yml

                     script: |
                       conda create -n venv python=3.9 -y
                       conda activate venv
                       echo "::group::Install newer objcopy that supports --set-section-alignment"
-                      yum install -y  devtoolset-10-binutils
-                      export PATH=/opt/rh/devtoolset-10/root/usr/bin/:$PATH

Collaborator

amdfaa Dec 5, 2024

The gcctoolset is installed through the dockerfile. Mentioned in this PR: pytorch/pytorch#140157

Collaborator

amdfaa Dec 5, 2024

The same is installed for rocm in this PR: pytorch/pytorch#141609

msaroufim added the topic: not user facing label

jithunnair-amd mentioned this pull request

[ROCm] Update setup-rocm for almalinux-based images pytorch/pytorch#143590

Closed

jithunnair-amd requested changes

View reviewed changes

.github/workflows/regression_test.yml Outdated Show resolved Hide resolved

.github/workflows/regression_test.yml Outdated

                   with:
                     timeout: 120
+                    no-sudo: ${{ matrix.gpu-arch-type == 'rocm' }}
+                    rocm: ${{ matrix.gpu-arch-type == 'rocm' }}
+                    continue-on-error: ${{ matrix.gpu-arch-type == 'rocm' }}

Collaborator

jithunnair-amd Dec 19, 2024

This should definitely not be checked-in, since it's only for us to gather a complete list of test failures. @msaroufim Would we merge this PR only after ROCm CI is fully clean? I'd rather get all these infra changes merged, so that we run torchao CI on ROCm regularly, and maybe skip any failing tests for ROCm while we work separately to enable them.

Member Author

msaroufim Dec 20, 2024 •

edited

Loading

It's up to you, the main constraint is we can't really be having CI per commit or on main run red since then it just causes confusion and people slowly learn to ignore seeing red. So if you'd like to merge some variant of this PR without running on commits to or on main then we can try to merge this more quickly

Personally I'd favor merging the skip tests as part of this work and we can do enablement for tests one by one easily while maintaining a green CI

Collaborator

jithunnair-amd Jan 8, 2025

@petrex Please note that torchao team would like to have this PR be merged with a clean signal for ROCm, so please skip any failing tests as part of this PR

Collaborator

petrex Jan 14, 2025

done . #1563 based on the latest ROCm CI run

pytorchmergebot pushed a commit to pytorch/pytorch that referenced this pull request


          [ROCm] Update setup-rocm for almalinux-based images (#143590)

a3b04d4

Needed for pytorch/test-infra#6003 and pytorch/ao#999

Pull Request resolved: #143590
Approved by: https://github.com/atalman

Co-authored-by: Jithun Nair <[email protected]>

amdfaa mentioned this pull request

Enable linux_job_v2.yml workflow for ROCm pytorch/test-infra#6104

Merged

Collaborator

petrex commented Jan 6, 2025

happy new year @jithunnair-amd @amdfaa Is this feature/PR ready to deploy?

Collaborator

jithunnair-amd commented Jan 8, 2025

happy new year @jithunnair-amd @amdfaa Is this feature/PR ready to deploy?

2 pending items:

Resolve issue with aws credentials causing some steps to fail in CI job eg. https://github.com/pytorch/ao/actions/runs/12656214677/job/35268262299#step:17:39 : this one needs pytorch infra team to enable "AWS trust policy" for torchao repository cc @huydhn to submit a PR for that
Skip any failing unit tests for ROCm as part of this PR as per Enable ROCM in CI #999 (comment) cc @petrex to add skips to this PR

Contributor

huydhn commented Jan 8, 2025

1. Resolve issue with aws credentials causing some steps to fail in CI job eg. https://github.com/pytorch/ao/actions/runs/12656214677/job/35268262299#step:17:39 : this one needs pytorch infra team to enable "AWS trust policy" for torchao repository cc @huydhn to submit a PR for that

The credential is working now. There is a new failure w.r.t chown on the CI job https://github.com/pytorch/ao/actions/runs/12656214677/job/35334719646, but it’s a different story I think

amdfaa force-pushed the msaroufim-patch-21 branch from f75c6a0 to 17289e7 Compare

January 10, 2025 23:44


          Enable ROCM in CI

cb1331d

amdfaa force-pushed the msaroufim-patch-21 branch from 17289e7 to cb1331d Compare

January 11, 2025 00:01

petrex mentioned this pull request

Skip Unit Tests for ROCm CI #1563

Merged

Collaborator

petrex commented Jan 14, 2025

@msaroufim I dont seem to have access to this branch so #1563 instead.
@amdfaa please kick off a new ROCm CI run with #1563
thanks!

Collaborator

petrex commented Jan 14, 2025

happy new year @jithunnair-amd @amdfaa Is this feature/PR ready to deploy?

2 pending items:

Resolve issue with aws credentials causing some steps to fail in CI job eg. https://github.com/pytorch/ao/actions/runs/12656214677/job/35268262299#step:17:39 : this one needs pytorch infra team to enable "AWS trust policy" for torchao repository cc @huydhn to submit a PR for that

Skip any failing unit tests for ROCm as part of this PR as per Enable ROCM in CI #999 (comment) cc @petrex to add skips to this PR

amdfaa added a commit to pytorch/test-infra that referenced this pull request


          Enable linux_job_v2.yml workflow for ROCm (#6104)

15110a3

Relates to pytorch/ao#999

Needs pytorch/pytorch#143590

amdfaa added 3 commits

January 16, 2025 13:57


          Update regression_test.yml

e54e154


          Update regression_test.yml

7ca8cdf


          Update regression_test.yml

593fb78

amdfaa merged commit d96c6a7 into main

15 of 18 checks passed

Contributor

andrewor14 commented Jan 17, 2025

Hi @amdfaa, looks like this PR is still failing tests when landed. It's causing other unrelated PRs to fail the same tests: https://hud.pytorch.org/pr/pytorch/ao/1580#35788489448. Please make sure the tests are passing before landing.

andrewor14 added a commit that referenced this pull request


          Revert "Enable ROCM in CI (#999)"

b07f714

This reverts commit d96c6a7.

This was referenced Jan 17, 2025

Revert "Enable ROCM in CI" #1583

Merged

Revert "Skip Unit Tests for ROCm CI" #1580

Merged

andrewor14 added a commit that referenced this pull request


          Revert "Enable ROCM in CI" (#1583)

1240b19

Revert "Enable ROCM in CI (#999)"

This reverts commit d96c6a7.

andrewor14 mentioned this pull request

[Reland] ROCm CI (Infra + Skips) #1581

Merged

petrex pushed a commit to petrex/ao that referenced this pull request


          Enable ROCM in CI (pytorch#999)

5bc0709

* Enable ROCM in CI

---------

Co-authored-by: amdfaa <[email protected]>

petrex pushed a commit to petrex/ao that referenced this pull request


          Enable ROCM in CI (pytorch#999)

6d6f203

* Enable ROCM in CI

---------

Co-authored-by: amdfaa <[email protected]>

jcaip pushed a commit that referenced this pull request


          [Reland] ROCm CI (Infra + Skips) (#1581)

ed361ff

This PR to skip the unit test failures for ROCm + infra changes to enable ROCm CI.

**NOTE:**
This PR aims to enable the ROCm CI testing for torchao _only for pushes to main branch_. The ROCm tests should start showing up here once this PR is merged: https://hud.pytorch.org/hud/pytorch/ao/main/1?per_page=50&name_filter=regression
Torchao PRs can also trigger the ROCm CI runs using the `ciflow/rocm` PR label (#1749).
Enabling ROCm CI testing on *all* torchao PRs will be done in a follow-up PR.

This pull request introduces the `skip_if_rocm` decorator across various test files to skip tests that are not yet supported on ROCm. The changes ensure that tests are conditionally skipped if ROCm is detected, improving the test suite's compatibility with different environments.

# Key changes include:

### Cherry-pick ROCm CI infra changes from #999 

### Configure workflow to trigger ROCm CI only for pushes to main branch, OR on PRs with the `ciflow/rocm` label

### Introduction of `skip_if_rocm` decorator:

* Added `skip_if_rocm` import in multiple test files to conditionally skip tests not supported on ROCm. (`test/dtypes/test_affine_quantized.py`, `test/dtypes/test_floatx.py`, `test/float8/test_base.py`, `test/hqq/test_hqq_affine.py`, `test/integration/test_integration.py`, `test/kernel/test_galore_downproj.py`, `test/prototype/test_awq.py`, `test/prototype/test_low_bit_optim.py`, `test/prototype/test_splitk.py`, `test/quantization/test_galore_quant.py`, `test/quantization/test_marlin_qqq.py`, `test/sparsity/test_marlin.py`, `test/test_ops.py`, `test/test_s8s4_linear_cutlass.py`, `torchao/utils.py`) [[1]](diffhunk://#diff-31b1ffcd78674b79cc65749176354ea4743683070120034709c1da7a3eac31f6R24) [[2]](diffhunk://#diff-0e811fa3416cd87d9a25b4fb680890098c69aa33ca4db4d347d4a10cc41e0eb3L30-R30) [[3]](diffhunk://#diff-05925b4469eb63ab854cc9891f088f570fa3822cdaeb4de109e0b1b9ab5038a7R21) [[4]](diffhunk://#diff-a9708dc28f15bb9cf665417e6c66601f9e8e2f1f672d1858603b74fa879a3357R13) [[5]](diffhunk://#diff-a977c33299f20a626cf650b2b6f0a49ef8fad7c97be21a5618e600b588b14b15R83) [[6]](diffhunk://#diff-4b0ddf8d1e85f4b4f1067f8d1d3e6b4d48785b3675c7202bf49bfbb1079d682fR14) [[7]](diffhunk://#diff-66249d5a8ed995b0a8e22c6354d6b270c5feeb982cb79a28f7c1b929700e89f4L8-R12) [[8]](diffhunk://#diff-244d33d1e8c30e765556011a4d3b76509f61433a346ba12ffc3115144e895aedR33) [[9]](diffhunk://#diff-2bcf3336ff64bfef786e6126813db46040b93628cab5faff3f0f5ed2cb077bf2L16-R24) [[10]](diffhunk://#diff-51ddab022797064be44ca38c87a56c6e87cd69444f4c6151a11b7f0141aef2b9R21) [[11]](diffhunk://#diff-133d8c7492ee2e7536328c8391545610750774e43d128d258380cb6787bb9e93L22-R22) [[12]](diffhunk://#diff-a58427e02fb5b05d26e03e8c2d216e5ae379d82084fd14bf77ea127b5505a43cL18-R18) [[13]](diffhunk://#diff-d183f2afc51d6a59bc70094e8f476d2468c45e415500f6eb60abad955e065156R22-R24) [[14]](diffhunk://#diff-85cc98d31eb8056e082ebdfbf2979aaa046ffc08bbacd4a65a31795b51998645R10-R12) [[15]](diffhunk://#diff-d2a11602a79e83305208472f1abe6a4106f02ce62a7f9524007181813863fcf6R10)

### Application of `skip_if_rocm` decorator:

* Applied `@skip_if_rocm("ROCm development in progress")` to multiple test functions to skip them when running on ROCm. (`test/dtypes/test_affine_quantized.py`, `test/dtypes/test_floatx.py`, `test/float8/test_base.py`, `test/hqq/test_hqq_affine.py`, `test/integration/test_integration.py`, `test/kernel/test_galore_downproj.py`, `test/prototype/test_awq.py`, `test/prototype/test_low_bit_optim.py`, `test/prototype/test_splitk.py`, `test/quantization/test_galore_quant.py`, `test/quantization/test_marlin_qqq.py`, `test/sparsity/test_marlin.py`) [[1]](diffhunk://#diff-31b1ffcd78674b79cc65749176354ea4743683070120034709c1da7a3eac31f6R93) [[2]](diffhunk://#diff-31b1ffcd78674b79cc65749176354ea4743683070120034709c1da7a3eac31f6R173) [[3]](diffhunk://#diff-31b1ffcd78674b79cc65749176354ea4743683070120034709c1da7a3eac31f6R186) [[4]](diffhunk://#diff-0e811fa3416cd87d9a25b4fb680890098c69aa33ca4db4d347d4a10cc41e0eb3R111) [[5]](diffhunk://#diff-05925b4469eb63ab854cc9891f088f570fa3822cdaeb4de109e0b1b9ab5038a7R427) [[6]](diffhunk://#diff-a9708dc28f15bb9cf665417e6c66601f9e8e2f1f672d1858603b74fa879a3357R114) [[7]](diffhunk://#diff-a977c33299f20a626cf650b2b6f0a49ef8fad7c97be21a5618e600b588b14b15R571) [[8]](diffhunk://#diff-a977c33299f20a626cf650b2b6f0a49ef8fad7c97be21a5618e600b588b14b15R690) [[9]](diffhunk://#diff-a977c33299f20a626cf650b2b6f0a49ef8fad7c97be21a5618e600b588b14b15R710) [[10]](diffhunk://#diff-a977c33299f20a626cf650b2b6f0a49ef8fad7c97be21a5618e600b588b14b15R904) [[11]](diffhunk://#diff-a977c33299f20a626cf650b2b6f0a49ef8fad7c97be21a5618e600b588b14b15R924) [[12]](diffhunk://#diff-4b0ddf8d1e85f4b4f1067f8d1d3e6b4d48785b3675c7202bf49bfbb1079d682fR33) [[13]](diffhunk://#diff-66249d5a8ed995b0a8e22c6354d6b270c5feeb982cb79a28f7c1b929700e89f4R120) [[14]](diffhunk://#diff-244d33d1e8c30e765556011a4d3b76509f61433a346ba12ffc3115144e895aedR116) [[15]](diffhunk://#diff-2bcf3336ff64bfef786e6126813db46040b93628cab5faff3f0f5ed2cb077bf2L16-R24) [[16]](diffhunk://#diff-51ddab022797064be44ca38c87a56c6e87cd69444f4c6151a11b7f0141aef2b9R86) [[17]](diffhunk://#diff-133d8c7492ee2e7536328c8391545610750774e43d128d258380cb6787bb9e93R48) [[18]](diffhunk://#diff-133d8c7492ee2e7536328c8391545610750774e43d128d258380cb6787bb9e93R70) [[19]](diffhunk://#diff-a58427e02fb5b05d26e03e8c2d216e5ae379d82084fd14bf77ea127b5505a43cR40) [[20]](diffhunk://#diff-a58427e02fb5b05d26e03e8c2d216e5ae379d82084fd14bf77ea127b5505a43cL51-R58)

### Module-level skips for ROCm:

* Added module-level skips for ROCm in specific test files to skip all tests within the module if ROCm is detected. (`test/test_ops.py`, `test/test_s8s4_linear_cutlass.py`) [[1]](diffhunk://#diff-d183f2afc51d6a59bc70094e8f476d2468c45e415500f6eb60abad955e065156R22-R24) [[2]](diffhunk://#diff-85cc98d31eb8056e082ebdfbf2979aaa046ffc08bbacd4a65a31795b51998645R10-R12)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/rocm CLA Signed module: rocm topic: not user facing