-
Notifications
You must be signed in to change notification settings - Fork 218
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[XNACK] GPU is asleep during the copy and not waking back up when it should. #2941
Comments
@cderb let's search if we have an internal ticket on this or not, if not we should create one for the runtime and driver. Thanks! FYI: @JehandadKhan @atamazov |
@cderb What is the base driver version? 5.6, like mentioned in ROCm/ROCm#2418? |
@atamazov this test docker was on rocm 6.1.0-82 |
@cderb Thanks but the base driver is not included with the image. Can you please provide output of (run it outside the container): modinfo amdgpu | grep -i -E "(version:)|(vermagic:)" or
|
Perhaps that would mean updating the base driver on this machine could resolve this issue? |
However the base version on our CI nodes is 6.2.4 and we observe the same issue I believe. |
@cderb Hmm... The most recent released ROCm is 6.1.0, how CI nodes may have 6.2.4 installed? |
Changing the grub options allowed these tests to pass on my test machine.
ROCm/ROCm#2418 (comment)
Originally posted by @cderb in #2864 (comment)
The text was updated successfully, but these errors were encountered: