Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[release/2.5] AMDSMI/layernorm cherry picks #1764

Open
wants to merge 2 commits into
base: release/2.5
Choose a base branch
from
Open

[release/2.5] AMDSMI/layernorm cherry picks #1764

wants to merge 2 commits into from
+4 −3

Conversation

jataylo
Copy link

@jataylo jataylo commented Dec 4, 2024

Cherry pick amdsmi fix: pytorch@04f569a
Cherry pick layer_norm accuracy fix: pytorch@001f736

jataylo and others added 2 commits December 4, 2024 14:28
@jataylo
[ROCm] Correct numerical issues in layer norm backwards kernel (pytor…
6024e85 
…ch#140259)

It was raised that the backwards layer norm on AMD was slightly off the accuracy of the equivalent NVIDIA implementation.

On AMD we call into a helper kernel `cuLoadWriteStridedInputs` which processes strided input and accumulates the partial gradients into shared memory.

In this kernel (pytorch#87635) we truncated `mean` and `rstd` from T_ACC type to T which causes numerical issues in the warp buffers created in this kernel. This PR will use the correct accumulator type for mean and rstd.

Note: Only AMD call into this call stack for backwards layer norm, so this was not an issue for NV.

Pull Request resolved: pytorch#140259
Approved by: https://github.com/jianyuh

(cherry picked from commit 001f736)
@jataylo
[ROCm] AMDSMI memory usage unification (pytorch#139900)
aea2827 
Fixes pytorch#140638

Old implementation used vram_used, which is not the correct equivalent API for pynvml memory utilization.

Pull Request resolved: pytorch#139900
Approved by: https://github.com/jeffdaily, https://github.com/eqy

(cherry picked from commit 04f569a)
@rocm-mici
Copy link

Jenkins build for aea282715584d464a86a9a0fdcaa597bb0c7a70d commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

Detected error during Pytorch building:

[7958/8668] Building HIPCC object caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/torch_hip_generated_legendre_polynomial_p.hip.o
[7959/8668] Building HIPCC object caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/torch_hip_generated_modified_bessel_i1.hip.o
[7960/8668] Building HIPCC object caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/torch_hip_generated_modified_bessel_i0.hip.o
[7961/8668] Building HIPCC object caffe2/CMakeFiles/torch_hip.dir/__/torch/csrc/distributed/c10d/torch_hip_generated_CUDASymmetricMemoryOps.cu.o
[7962/8668] Building HIPCC object caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/transformers/hip/flash_attn/torch_hip_generated_flash_api.hip.o
FAILED: caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/transformers/hip/flash_attn/torch_hip_generated_flash_api.hip.o /var/lib/jenkins/pytorch/build/caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/transformers/hip/flash_attn/torch_hip_generated_flash_api.hip.o 
cd /var/lib/jenkins/pytorch/build/caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/transformers/hip/flash_attn && /opt/conda/envs/py_3.10/bin/cmake -E make_directory /var/lib/jenkins/pytorch/build/caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/transformers/hip/flash_attn/. && /opt/conda/envs/py_3.10/bin/cmake -D verbose:BOOL=OFF -D build_configuration:STRING=RELEASE -D generated_file:STRING=/var/lib/jenkins/pytorch/build/caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/transformers/hip/flash_attn/./torch_hip_generated_flash_api.hip.o -P /var/lib/jenkins/pytorch/build/caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/transformers/hip/flash_attn/torch_hip_generated_flash_api.hip.o.cmake
In file included from /var/lib/jenkins/pytorch/aten/src/ATen/native/transformers/hip/flash_attn/flash_api.hip:57:
/var/lib/jenkins/pytorch/aten/src/ATen/native/transformers/hip/aotriton_adapter.h:120:10: error: no matching constructor for initialization of 'aotriton::TensorView<0>'
  120 |   return aotriton::TensorView<0>(reinterpret_cast<intptr_t>(q.data_ptr()),
      |          ^                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Dec 11, 2024 •
edited
Loading

Jenkins build for aea282715584d464a86a9a0fdcaa597bb0c7a70d commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Dec 17, 2024 •
edited
Loading

Jenkins build for aea282715584d464a86a9a0fdcaa597bb0c7a70d commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Dec 18, 2024 •
edited
Loading

Jenkins build for aea282715584d464a86a9a0fdcaa597bb0c7a70d commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Dec 19, 2024 •
edited
Loading

Jenkins build for aea282715584d464a86a9a0fdcaa597bb0c7a70d commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Dec 23, 2024 •
edited
Loading

Jenkins build for aea282715584d464a86a9a0fdcaa597bb0c7a70d commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Dec 26, 2024 •
edited
Loading

Jenkins build for aea282715584d464a86a9a0fdcaa597bb0c7a70d commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Jan 6, 2025 •
edited
Loading

Jenkins build for aea282715584d464a86a9a0fdcaa597bb0c7a70d commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Jan 8, 2025 •
edited
Loading

Jenkins build for aea282715584d464a86a9a0fdcaa597bb0c7a70d commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Jan 8, 2025 •
edited
Loading

Jenkins build for aea282715584d464a86a9a0fdcaa597bb0c7a70d commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Jan 10, 2025 •
edited
Loading

Jenkins build for aea282715584d464a86a9a0fdcaa597bb0c7a70d commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Jan 14, 2025 •
edited
Loading

Jenkins build for aea282715584d464a86a9a0fdcaa597bb0c7a70d commit finished as ABORTED
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Jan 15, 2025 •
edited
Loading

Jenkins build for aea282715584d464a86a9a0fdcaa597bb0c7a70d commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Jan 15, 2025 •
edited
Loading

Jenkins build for aea282715584d464a86a9a0fdcaa597bb0c7a70d commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Jan 16, 2025 •
edited
Loading

Jenkins build for aea282715584d464a86a9a0fdcaa597bb0c7a70d commit finished as ABORTED
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Jan 16, 2025 •
edited
Loading

Jenkins build for aea282715584d464a86a9a0fdcaa597bb0c7a70d commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Jan 20, 2025 •
edited
Loading

Jenkins build for aea282715584d464a86a9a0fdcaa597bb0c7a70d commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Jan 24, 2025 •
edited
Loading

Jenkins build for aea282715584d464a86a9a0fdcaa597bb0c7a70d commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Jan 27, 2025 •
edited
Loading

Jenkins build for aea282715584d464a86a9a0fdcaa597bb0c7a70d commit finished as SUCCESS
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Jan 27, 2025 •
edited
Loading

Jenkins build for aea282715584d464a86a9a0fdcaa597bb0c7a70d commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Jan 29, 2025 •
edited
Loading

Jenkins build for aea282715584d464a86a9a0fdcaa597bb0c7a70d commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Jan 31, 2025 •
edited
Loading

Jenkins build for aea282715584d464a86a9a0fdcaa597bb0c7a70d commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Jan 31, 2025 •
edited
Loading

Jenkins build for aea282715584d464a86a9a0fdcaa597bb0c7a70d commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Jan 31, 2025 •
edited
Loading

Jenkins build for aea282715584d464a86a9a0fdcaa597bb0c7a70d commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

rocm-repo-management-api bot commented Feb 3, 2025 •
edited
Loading

Jenkins build for aea282715584d464a86a9a0fdcaa597bb0c7a70d commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

@rocm-repo-management-api
Copy link

Jenkins build for aea282715584d464a86a9a0fdcaa597bb0c7a70d commit is in progress
Links: Blue Ocean view / Build artifacts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pruthvistony pruthvistony Awaiting requested review from pruthvistony

@jithunnair-amd jithunnair-amd Awaiting requested review from jithunnair-amd

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jataylo @rocm-mici