[windows] Patch TENSILE_LIBRARY_DIR in hipBLASLt. by ScottTodd · Pull Request #783 · ROCm/TheRock

ScottTodd · 2025-06-06T01:01:05Z

Tentative fix for #681. Related to #792. Also enabled hipBLASLt tests on our Windows CI, so progress on #544.

This brings the hipBLASLt Tensile library directory structure in line with what rocBLAS uses on Windows.

I've started sending upstream patches for review too:

…dows-blas-library-dirs

ScottTodd · 2025-06-06T23:59:05Z

-    # Currently, this test is not working with Windows. This test will be enabled once this library has been enabled
-    if: ${{ inputs.platform == 'linux' }}


Tests pass on Windows gfx1151: https://github.com/ROCm/TheRock/actions/runs/15495327375/job/43649536258?pr=783

[----------] Global test environment tear-down [==========] 5766 tests from 2 test suites ran. (470772 ms total) [ PASSED ] 5766 tests. hipBLASLt version: 100000 hipBLASLt git version: c630f202 command line: B:\actions-runner\_work\TheRock\TheRock\build\bin\hipblaslt-test.exe --gtest_filter=*pre_checkin*

Odd checkout failures on rocBLAS and rocPRIM though:
https://github.com/ROCm/TheRock/actions/runs/15495327375/job/43649536566?pr=783

Deleting the contents of 'B:\actions-runner\_work\TheRock\TheRock' Error: File was unable to be removed Error: EPERM: operation not permitted, unlink 'B:\actions-runner\_work\TheRock\TheRock\build\bin\amd_comgr0605.dll'

Failed again on a retry: https://github.com/ROCm/TheRock/actions/runs/15495327375/job/43740951661?pr=783#step:2:28

@amd-justchen can you debug on that runner (windows-strix-halo-gpu-rocm-1)?

Sure! I'll take a look! I just about am done monitoring the CI CPU builders. That error looks like the same one before. I'll be able to remote in to the machine to troubleshoot.

I suspect the build is only just now producing those files, so we should see if any permission/owner bits are set differently. Might need to change how those projects are built or make our CI workflows try harder to clean up the files during startup/shutdown.

A common post-job hook for a self-hosted runner deletes the source directory. We used these on IREE: https://github.com/iree-org/iree/tree/41dcee93c7157955d94973addf6770cecf926849/build_tools/github_actions/runner/config/hooks. See pre_job.sh, cleanup_workdir.sh, and https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/running-scripts-before-or-after-a-job.

Argh: https://github.com/ROCm/TheRock/actions/runs/16148483933/job/45579902751

Still failing with "File was unable to be removed" for rocblas, hipblaslt, rocprim, and hipcub tests. Going to retry those to see if the runners are flaky somehow.

amd-justchen · 2025-06-07T02:35:34Z

https://github.com/ROCm/TheRock/actions/runs/15495327375/job/43649536566?pr=783#step:2:32

Just noting this error on the selfhosted gpu test runner

Error: File was unable to be removed Error: EPERM: operation not permitted, unlink 'B:\actions-runner\_work\TheRock\TheRock\build\bin\rocblas.dll'

This reverts commit b099c87.

…dows-blas-library-dirs

amd-justchen

LGTM

geomin12 · 2025-06-30T16:43:17Z

does tensile work with 1151? seems like the rocblas is failing due to no tensile .dat found for 1151 :(

ScottTodd · 2025-06-30T18:29:20Z

does tensile work with 1151? seems like the rocblas is failing due to no tensile .dat found for 1151 :(

Perhaps not...

I'm having trouble debugging locally using the artifacts produced by CI though: https://gist.github.com/ScottTodd/bf9519793b0d99df5e1b89e0be479b89. Might need to build from source to debug further.

ScottTodd · 2025-07-01T23:48:04Z

does tensile work with 1151? seems like the rocblas is failing due to no tensile .dat found for 1151 :(

Perhaps not...

I'm having trouble debugging locally using the artifacts produced by CI though: https://gist.github.com/ScottTodd/bf9519793b0d99df5e1b89e0be479b89. Might need to build from source to debug further.

I had the same issue with a source build on my primary gfx1100 dev machine so I pushed a patch that now lets me run the rocBLAS tests locally on Windows (they all passed). Next I should be able to pull the gfx1151 artifacts and test on gfx1151 dev machine (or build there, but that will take longer).

ScottTodd · 2025-07-02T21:18:51Z

Okay I can reproduce the rocblas test failures locally on two machines (gfx1151 and gfx1100) now. Next step is to double check the directory layout of therock-dist compared against python build_tools/install_rocm_from_artifacts.py --run-id 16010675859 --amdgpu-family gfx110X-dgpu --output-dir %HOME%\.therock\install_16010675859 --blas --tests.

ScottTodd · 2025-07-02T22:07:31Z

Ugh. I wrote this patch for the old CMake, we've since moved to next-cmake, which repeated the same pattern.

ScottTodd · 2025-07-02T22:42:50Z

Validated more carefully on a local build, hopefully works again now...

…dows-blas-library-dirs

ScottTodd · 2025-07-07T18:47:50Z

CI failed again, and I can reproduce locally again... strange. Testing is harder now that we've had a significant compile time regression though: #978

ScottTodd · 2025-07-07T18:52:16Z

🤦 found more source code:

ScottTodd · 2025-07-07T22:11:17Z

Okay, tests are passing again, including newly enabled hipblaslt tests on Windows. I'm going to need reviews from hipblaslt and rocblas maintainers for this one though, so I'll also prepare a PR in https://github.com/ROCm/rocm-libraries and may need to get the attention of some Windows HIP SDK maintainers.

ScottTodd · 2025-07-07T23:37:17Z

Sent two upstream PRs so far:

I'll need at least one more for the rocblas_parse_data.cpp changes, since there are a few other files that have repeated that code pattern of running .py scripts directly instead of via python script.py on Windows. Those changes aren't load bearing for users, though they would help for library developers who want to run tests locally, like me.

ScottTodd · 2025-07-08T17:03:45Z

@ellosel want to review this PR while we work through the CI jobs in the rocm-libraries PRs? Once those land and get propagated to TheRock we can then drop these patches.

ScottTodd · 2025-07-08T23:32:05Z

Test failures from files remaining open/locked are back, and discussion on the upstream PRs is converging on an alternate solution - keeping the lib/ and bin/ directory structure for rocblas and getting hipblaslt to match it as needed.

ScottTodd · 2025-07-09T19:25:22Z

Reworked the patches on this PR and the upstream PRs. PTAL.

ScottTodd · 2025-07-09T22:23:28Z

Feedback is looking good on the upstream PR, going to merge this with the existing approval to start flushing my PR queue. Open to post-submit feedback and reverting if others have objections.

Progress on #827 (!!) Tested at https://github.com/ROCm/TheRock/actions/runs/16185309410, which triggered these: * https://github.com/ROCm/TheRock/actions/runs/16186383600 (gfx1151) * https://github.com/ROCm/TheRock/actions/runs/16186941663 (gfx120X-all) * https://github.com/ROCm/TheRock/actions/runs/16186966931 (gfx110X-dgpu) Some next steps after this, roughly in order: * Monitor scheduled jobs to check that wheels are successfully built and uploaded * Download/install and test the wheels on various systems, exercising components like hipBLASLt * Update support status: * https://github.com/ROCm/TheRock?tab=readme-ov-file#support-status * https://github.com/ROCm/TheRock/blob/main/RELEASES.md#python-packages-support-status * Build torchvision and torchaudio (need some patches) * Connect with other build/test/release workflows so we get pre-submit, post-submit, and scheduled coverage for these builds/releases/tests * Rebase patches onto PyTorch 2.8 (and/or HEAD), land patches upstream * Add features like Triton support (aotriton?) for optimized flash attention kernels, etc. * Expand the list of supported gfx targets --- ~~This depends on some other open PRs, but I have tested parts of it all the way through to dev wheel publishing and then installation and testing on a dev machine: https://github.com/ROCm/TheRock/actions/runs/16181411431/job/45678668536, #827 (comment) ~~See the other PRs:~~ * ~~#783 * ~~#980 * ~~#988 * ~~#990

This gets hipBLASLt tests passing on Windows in TheRock (see ROCm/TheRock#783), for ROCm/TheRock#681. It also enables PyTorch builds on Windows to use hipBLASLt without needing to manually copy folders around in the install directory. This directory structure is now consistent with rocBLAS: https://github.com/ROCm/rocm-libraries/blob/c7fe831061a4cdc3a14e1aacab8c0738c21daf32/projects/rocblas/library/src/CMakeLists.txt#L825-L830

[hipblaslt] Adjust HIPBLASLT_TENSILE_LIBRARY_DIR on Windows. (#522) This gets hipBLASLt tests passing on Windows in TheRock (see ROCm/TheRock#783), for ROCm/TheRock#681. It also enables PyTorch builds on Windows to use hipBLASLt without needing to manually copy folders around in the install directory. This directory structure is now consistent with rocBLAS: https://github.com/ROCm/rocm-libraries/blob/c7fe831061a4cdc3a14e1aacab8c0738c21daf32/projects/rocblas/library/src/CMakeLists.txt#L825-L830

ScottTodd added 2 commits June 5, 2025 17:53

[windows] Patch TENSILE_LIBRARY_DIR in rocBLAS and hipBLASLt.

faf5a86

[windows][ci] Enable hipBLASLt tests on Windows.

b099c87

github-project-automation Bot added this to TheRock Triage Jun 6, 2025

github-project-automation Bot moved this to TODO in TheRock Triage Jun 6, 2025

ScottTodd mentioned this pull request Jun 6, 2025

[Windows] rocblaslt error: Could not load "TensileLibrary_lazy_gfx1100.dat" #681

Closed

Delete extra patch.

598fd07

ScottTodd added the gfx115X-windows label Jun 6, 2025

ScottTodd commented Jun 6, 2025

View reviewed changes

Comment thread .github/workflows/test_hipblaslt.yml Resolved

geomin12 added gfx115X-windows and removed gfx115X-windows labels Jun 6, 2025

Merge remote-tracking branch 'upstream/main' into users/scotttodd/win…

92c99ce

…dows-blas-library-dirs

ScottTodd commented Jun 6, 2025

View reviewed changes

amd-justchen mentioned this pull request Jun 9, 2025

CI - Windows - Checkout failures with EPERM: operation not permitted, unlink ... #792

Closed

ScottTodd added 2 commits June 29, 2025 20:20

Revert "[windows][ci] Enable hipBLASLt tests on Windows."

59ce73a

This reverts commit b099c87.

Merge remote-tracking branch 'upstream/main' into users/scotttodd/win…

7709438

…dows-blas-library-dirs

amd-justchen approved these changes Jun 30, 2025

View reviewed changes

Patch rocblas_gentest.py invocation for better Windows compatibility.

be122d6

Apply patch to rocBLAS next-cmake too.

6bea94d

Merge remote-tracking branch 'upstream/main' into users/scotttodd/win…

374f26b

…dows-blas-library-dirs

ScottTodd added 2 commits July 7, 2025 12:42

Patch sibling search location in rocBLAS.

fdc7610

Enable hipblaslt tests on Windows again.

5d6c552

ScottTodd marked this pull request as ready for review July 7, 2025 22:10

This was referenced Jul 7, 2025

[hipblaslt] Adjust HIPBLASLT_TENSILE_LIBRARY_DIR on Windows. ROCm/rocm-libraries#522

Merged

[rocblas] Unify ROCBLAS_TENSILE_*_DIR across platforms. ROCm/rocm-libraries#523

Closed

ScottTodd requested a review from stellaraccident July 8, 2025 00:38

ScottTodd commented Jul 8, 2025

View reviewed changes

Comment thread patches/amd-mainline/rocBLAS/0004-Run-rocblas_gentest.py-under-python-on-Windows.patch Resolved

Drop gentest.py patch for now.

16d18fa

ScottTodd mentioned this pull request Jul 8, 2025

[Windows] Parse yaml functions are reliant on .py file associations for _gentest.py files ROCm/rocm-libraries#535

Closed

ScottTodd requested a review from davidd-amd July 8, 2025 17:02

ScottTodd marked this pull request as draft July 8, 2025 23:32

ScottTodd removed the request for review from stellaraccident July 8, 2025 23:32

ScottTodd added 2 commits July 8, 2025 16:49

Rework hipBLASLt patch, drop rocBLAS patches.

6493549

Revert changes to patches.

031e3f8

ScottTodd changed the title ~~[windows] Patch TENSILE_LIBRARY_DIR in rocBLAS and hipBLASLt.~~ [windows] Patch TENSILE_LIBRARY_DIR in hipBLASLt. Jul 9, 2025

ScottTodd marked this pull request as ready for review July 9, 2025 19:24

ScottTodd merged commit 36c10f3 into main Jul 9, 2025

github-project-automation Bot moved this from TODO to Done in TheRock Triage Jul 9, 2025

ScottTodd deleted the users/scotttodd/windows-blas-library-dirs branch July 9, 2025 22:23

ScottTodd mentioned this pull request Jul 9, 2025

[torch][windows] Build Windows release PyTorch wheels. #994

Merged

		# Currently, this test is not working with Windows. This test will be enabled once this library has been enabled
		if: ${{ inputs.platform == 'linux' }}

Conversation

ScottTodd commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ScottTodd Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

ScottTodd Jun 9, 2025

Choose a reason for hiding this comment

Uh oh!

amd-justchen Jun 9, 2025

Choose a reason for hiding this comment

Uh oh!

ScottTodd Jun 9, 2025

Choose a reason for hiding this comment

Uh oh!

ScottTodd Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

amd-justchen commented Jun 7, 2025

Uh oh!

amd-justchen left a comment

Choose a reason for hiding this comment

Uh oh!

geomin12 commented Jun 30, 2025

Uh oh!

ScottTodd commented Jun 30, 2025

Uh oh!

ScottTodd commented Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ScottTodd commented Jul 2, 2025

Uh oh!

ScottTodd commented Jul 2, 2025

Uh oh!

ScottTodd commented Jul 2, 2025

Uh oh!

ScottTodd commented Jul 7, 2025

Uh oh!

ScottTodd commented Jul 7, 2025

Uh oh!

ScottTodd commented Jul 7, 2025

Uh oh!

ScottTodd commented Jul 7, 2025

Uh oh!

Uh oh!

ScottTodd commented Jul 8, 2025

Uh oh!

ScottTodd commented Jul 8, 2025

Uh oh!

ScottTodd commented Jul 9, 2025

Uh oh!

ScottTodd commented Jul 9, 2025

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ScottTodd commented Jun 6, 2025 •

edited

Loading

ScottTodd commented Jul 1, 2025 •

edited

Loading