Repository navigation
fix(hipthreads): signal pushCount with hipStreamWriteValue32 to fix gfx12 Windows hang - #13503
Merged
DanielMcIntosh merged 4 commits intoOct 29, 2026
Merged
Conversation
…fx12 Windows hang
✅ All Checks Passed — Ready for Review
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🎉 All checks passed! This PR is ready for review. |
Contributor
Author
|
CI run on gfx1201 with the fix: https://github.com/ROCm/rocm-libraries/actions/runs/38381871451/job/115314385968 |
DanielMcIntosh
approved these changes
Oct 28, 2026
DanielMcIntosh
left a comment
Contributor
There was a problem hiding this comment.
Approving this to get the CI green since it seems like it will solve the problem, but I have left some comments that should be addressed in a follow-up PR
DanielMcIntosh
enabled auto-merge (squash)
October 29, 2026 21:09
DanielMcIntosh
deleted the
users/marsavic/hipthreads-fix-gfx120x-win-hang
branch
October 29, 2026 21:14
Contributor
TheRock Submodule Bump ActivityNewest first
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Closes #13487
Closes #13631
On Windows gfx120X, the hipthreads lit tests and the examples hang in TheRock CI. All lit tests that run through
hip::wthreadtime out, and saxpy hits the 30-minute job timeout. The same nightly passes on Windows gfx103X/gfx110X/gfx1151 and on Linux gfx120X.Technical Details
threading_mainis a persistent kernel. It keeps pollingcpuWorkQueue.pushCountand exits once the host replaces the-1Uvalue with the final push count. The host wrote that value withhipMemcpyToSymbolAsync. On Windows gfx12, the running kernel never sees it, even after the copy completes. So the kernel never exits, and the next null-stream operation (thrust::copy,hipDeviceSynchronize) waits on it forever.hipMemcpyToSymbolAsyncvshipStreamWriteValue32:hipMemcpyToSymbolAsyncis a data transfer. HIP only guarantees its result is visible to work that runs after it in the stream. A kernel that is already running is not guaranteed to see it.hipStreamWriteValue32is a stream memory operation meant for signaling work that is already running. It is a single, ordered, aligned 32-bit write.hipThreads already uses this approach for the work queue:
sendToGPUpublishes entries withhipStreamWriteValue64because of a similar Navi4 issue. This PR does the same forpushCount.Changes in
src/hip/thread.cxx:notifyDeviceThereMightNotBeAnyMoreWorkand the-1Ure-arm inprepDeviceForWorknow writepushCountwithhipStreamWriteValue32, on the same enqueuing stream as before. The device address is cached.mainStream.staticcopy buffer is removed, since the value is now passed directly.Test Plan
Windows 11, RX 9070 XT (gfx1201), TheRock nightly
10.2.0a20261012:lit -j 1 test/)Test Result
🤖 Generated with Claude Code