Skip to content

[windows] Preload compiler DLLs in ROCmLibrariesTest. - #980

Merged
ScottTodd merged 2 commits into
mainfrom
users/scotttodd/windows-test-preload
Jul 10, 2025
Merged

[windows] Preload compiler DLLs in ROCmLibrariesTest.#980
ScottTodd merged 2 commits into
mainfrom
users/scotttodd/windows-test-preload

Conversation

@ScottTodd

@ScottTodd ScottTodd commented Jul 7, 2025

Copy link
Copy Markdown
Member

Progress on #827 (see also the specific bug report at #827 (comment)).

Previously, rocm-sdk test failed on my dev machine with errors like FileNotFoundError: Could not find module 'D:\scratch\therock\python_rocm_2\.venv\Lib\site-packages\_rocm_sdk_libraries_gfx110X_dgpu\bin\hipfft.dll' (or one of its dependencies). Try using the full path with constructor syntax.

Expand for full logs 

(.venv) λ rocm-sdk test
testCLI (rocm_sdk.tests.base_test.ROCmBaseTest.testCLI) ... ++ Exec [D:\scratch\therock\python_rocm_2]$ 'D:\scratch\therock\python_rocm_2\.venv\Scripts\python.exe' -P -m rocm_sdk --help
ok
testTargets (rocm_sdk.tests.base_test.ROCmBaseTest.testTargets) ... ++ Exec [D:\scratch\therock\python_rocm_2]$ 'D:\scratch\therock\python_rocm_2\.venv\Scripts\python.exe' -P -m rocm_sdk targets
ok
testVersion (rocm_sdk.tests.base_test.ROCmBaseTest.testVersion) ... ++ Exec [D:\scratch\therock\python_rocm_2]$ 'D:\scratch\therock\python_rocm_2\.venv\Scripts\python.exe' -P -m rocm_sdk version
ok
test_initialize_process_check_version (rocm_sdk.tests.base_test.ROCmBaseTest.test_initialize_process_check_version) ... ok
test_initialize_process_check_version_asterisk (rocm_sdk.tests.base_test.ROCmBaseTest.test_initialize_process_check_version_asterisk) ... ok
test_initialize_process_check_version_mismatch (rocm_sdk.tests.base_test.ROCmBaseTest.test_initialize_process_check_version_mismatch) ... ok
test_initialize_process_check_version_mismatch_warning (rocm_sdk.tests.base_test.ROCmBaseTest.test_initialize_process_check_version_mismatch_warning) ... ok
test_initialize_process_check_version_pattern (rocm_sdk.tests.base_test.ROCmBaseTest.test_initialize_process_check_version_pattern) ... ok
test_initialize_process_env_preload_1 (rocm_sdk.tests.base_test.ROCmBaseTest.test_initialize_process_env_preload_1) ... ok
test_initialize_process_env_preload_2_comma (rocm_sdk.tests.base_test.ROCmBaseTest.test_initialize_process_env_preload_2_comma) ... ok
test_initialize_process_env_preload_2_semi (rocm_sdk.tests.base_test.ROCmBaseTest.test_initialize_process_env_preload_2_semi) ... ok
test_initialize_process_preload_libraries (rocm_sdk.tests.base_test.ROCmBaseTest.test_initialize_process_preload_libraries) ... ok
testConsoleScripts (rocm_sdk.tests.core_test.ROCmCoreTest.testConsoleScripts) ... ok
testInstallationLayout (rocm_sdk.tests.core_test.ROCmCoreTest.testInstallationLayout)
The `rocm_sdk` and core module must be siblings on disk. ... ok
testPreloadLibraries (rocm_sdk.tests.core_test.ROCmCoreTest.testPreloadLibraries) ... ok
testSharedLibrariesLoad (rocm_sdk.tests.core_test.ROCmCoreTest.testSharedLibrariesLoad) ... ok
testConsoleScripts (rocm_sdk.tests.libraries_test.ROCmLibrariesTest.testConsoleScripts) ... ok
testInstallationLayout (rocm_sdk.tests.libraries_test.ROCmLibrariesTest.testInstallationLayout)
The `rocm_sdk` and libraries module must be siblings on disk. ... ok
testSharedLibrariesLoad (rocm_sdk.tests.libraries_test.ROCmLibrariesTest.testSharedLibrariesLoad) ... Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\Python312\Lib\ctypes\__init__.py", line 379, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: Could not find module 'D:\scratch\therock\python_rocm_2\.venv\Lib\site-packages\_rocm_sdk_libraries_gfx110X_dgpu\bin\hipfft.dll' (or one of its dependencies). Try using the full path with constructor syntax.

  testSharedLibrariesLoad (rocm_sdk.tests.libraries_test.ROCmLibrariesTest.testSharedLibrariesLoad) [Check shared library loads] (so_path=WindowsPath('D:/scratch/therock/python_rocm_2/.venv/Lib/site-packages/_rocm_sdk_libraries_gfx110X_dgpu/bin/hipfft.dll')) ... ERROR
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\Python312\Lib\ctypes\__init__.py", line 379, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: Could not find module 'D:\scratch\therock\python_rocm_2\.venv\Lib\site-packages\_rocm_sdk_libraries_gfx110X_dgpu\bin\MIOpen.dll' (or one of its dependencies). Try using the full path with constructor syntax.
  testSharedLibrariesLoad (rocm_sdk.tests.libraries_test.ROCmLibrariesTest.testSharedLibrariesLoad) [Check shared library loads] (so_path=WindowsPath('D:/scratch/therock/python_rocm_2/.venv/Lib/site-packages/_rocm_sdk_libraries_gfx110X_dgpu/bin/MIOpen.dll')) ... ERROR
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\Python312\Lib\ctypes\__init__.py", line 379, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: Could not find module 'D:\scratch\therock\python_rocm_2\.venv\Lib\site-packages\_rocm_sdk_libraries_gfx110X_dgpu\bin\rocfft.dll' (or one of its dependencies). Try using the full path with constructor syntax.
  testSharedLibrariesLoad (rocm_sdk.tests.libraries_test.ROCmLibrariesTest.testSharedLibrariesLoad) [Check shared library loads] (so_path=WindowsPath('D:/scratch/therock/python_rocm_2/.venv/Lib/site-packages/_rocm_sdk_libraries_gfx110X_dgpu/bin/rocfft.dll')) ... ERROR
testCLIPathBin (rocm_sdk.tests.devel_test.ROCmDevelTest.testCLIPathBin) ... ++ Exec [D:\scratch\therock\python_rocm_2]$ 'D:\scratch\therock\python_rocm_2\.venv\Scripts\python.exe' -P -m rocm_sdk path --bin
ok
testCLIPathCMake (rocm_sdk.tests.devel_test.ROCmDevelTest.testCLIPathCMake) ... ++ Exec [D:\scratch\therock\python_rocm_2]$ 'D:\scratch\therock\python_rocm_2\.venv\Scripts\python.exe' -P -m rocm_sdk path --cmake
ok
testCLIPathRoot (rocm_sdk.tests.devel_test.ROCmDevelTest.testCLIPathRoot) ... ++ Exec [D:\scratch\therock\python_rocm_2]$ 'D:\scratch\therock\python_rocm_2\.venv\Scripts\python.exe' -P -m rocm_sdk path --root
ok
testInstallationLayout (rocm_sdk.tests.devel_test.ROCmDevelTest.testInstallationLayout)
The `rocm_sdk` and devel module must be siblings on disk. ... ok
testRootLLVMSymlinkExists (rocm_sdk.tests.devel_test.ROCmDevelTest.testRootLLVMSymlinkExists) ... skipped 'root LLVM symlink only exists on Linux'
testSharedLibrariesLoad (rocm_sdk.tests.devel_test.ROCmDevelTest.testSharedLibrariesLoad) ... ++ Exec [D:\scratch\therock\python_rocm_2]$ 'D:\scratch\therock\python_rocm_2\.venv\Scripts\python.exe' -P -m rocm_sdk path --root
ok

======================================================================
ERROR: testSharedLibrariesLoad (rocm_sdk.tests.libraries_test.ROCmLibrariesTest.testSharedLibrariesLoad) [Check shared library loads] (so_path=WindowsPath('D:/scratch/therock/python_rocm_2/.venv/Lib/site-packages/_rocm_sdk_libraries_gfx110X_dgpu/bin/hipfft.dll'))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\scratch\therock\python_rocm_2\.venv\Lib\site-packages\rocm_sdk\tests\libraries_test.py", line 62, in testSharedLibrariesLoad
    subprocess.check_call(
  File "C:\Program Files\Python312\Lib\subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['D:\\scratch\\therock\\python_rocm_2\\.venv\\Scripts\\python.exe', '-P', '-c', 'import ctypes; import sys; ctypes.CDLL(sys.argv[1])', 'D:\\scratch\\therock\\python_rocm_2\\.venv\\Lib\\site-packages\\_rocm_sdk_libraries_gfx110X_dgpu\\bin\\hipfft.dll']' returned non-zero exit status 1.

======================================================================
ERROR: testSharedLibrariesLoad (rocm_sdk.tests.libraries_test.ROCmLibrariesTest.testSharedLibrariesLoad) [Check shared library loads] (so_path=WindowsPath('D:/scratch/therock/python_rocm_2/.venv/Lib/site-packages/_rocm_sdk_libraries_gfx110X_dgpu/bin/MIOpen.dll'))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\scratch\therock\python_rocm_2\.venv\Lib\site-packages\rocm_sdk\tests\libraries_test.py", line 62, in testSharedLibrariesLoad
    subprocess.check_call(
  File "C:\Program Files\Python312\Lib\subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['D:\\scratch\\therock\\python_rocm_2\\.venv\\Scripts\\python.exe', '-P', '-c', 'import ctypes; import sys; ctypes.CDLL(sys.argv[1])', 'D:\\scratch\\therock\\python_rocm_2\\.venv\\Lib\\site-packages\\_rocm_sdk_libraries_gfx110X_dgpu\\bin\\MIOpen.dll']' returned non-zero exit status 1.

======================================================================
ERROR: testSharedLibrariesLoad (rocm_sdk.tests.libraries_test.ROCmLibrariesTest.testSharedLibrariesLoad) [Check shared library loads] (so_path=WindowsPath('D:/scratch/therock/python_rocm_2/.venv/Lib/site-packages/_rocm_sdk_libraries_gfx110X_dgpu/bin/rocfft.dll'))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\scratch\therock\python_rocm_2\.venv\Lib\site-packages\rocm_sdk\tests\libraries_test.py", line 62, in testSharedLibrariesLoad
    subprocess.check_call(
  File "C:\Program Files\Python312\Lib\subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['D:\\scratch\\therock\\python_rocm_2\\.venv\\Scripts\\python.exe', '-P', '-c', 'import ctypes; import sys; ctypes.CDLL(sys.argv[1])', 'D:\\scratch\\therock\\python_rocm_2\\.venv\\Lib\\site-packages\\_rocm_sdk_libraries_gfx110X_dgpu\\bin\\rocfft.dll']' returned non-zero exit status 1.

----------------------------------------------------------------------
Ran 25 tests in 26.523s

FAILED (errors=3, skipped=1)

Now it passes:

(.venv) λ rocm-sdk test
testCLI (rocm_sdk.tests.base_test.ROCmBaseTest.testCLI) ... ++ Exec [D:\scratch\therock\python_rocm_2]$ 'D:\scratch\therock\python_rocm_2\.venv\Scripts\python.exe' -P -m rocm_sdk --help
ok
testTargets (rocm_sdk.tests.base_test.ROCmBaseTest.testTargets) ... ++ Exec [D:\scratch\therock\python_rocm_2]$ 'D:\scratch\therock\python_rocm_2\.venv\Scripts\python.exe' -P -m rocm_sdk targets
ok
testVersion (rocm_sdk.tests.base_test.ROCmBaseTest.testVersion) ... ++ Exec [D:\scratch\therock\python_rocm_2]$ 'D:\scratch\therock\python_rocm_2\.venv\Scripts\python.exe' -P -m rocm_sdk version
ok
test_initialize_process_check_version (rocm_sdk.tests.base_test.ROCmBaseTest.test_initialize_process_check_version) ... ok
test_initialize_process_check_version_asterisk (rocm_sdk.tests.base_test.ROCmBaseTest.test_initialize_process_check_version_asterisk) ... ok
test_initialize_process_check_version_mismatch (rocm_sdk.tests.base_test.ROCmBaseTest.test_initialize_process_check_version_mismatch) ... ok
test_initialize_process_check_version_mismatch_warning (rocm_sdk.tests.base_test.ROCmBaseTest.test_initialize_process_check_version_mismatch_warning) ... ok
test_initialize_process_check_version_pattern (rocm_sdk.tests.base_test.ROCmBaseTest.test_initialize_process_check_version_pattern) ... ok
test_initialize_process_env_preload_1 (rocm_sdk.tests.base_test.ROCmBaseTest.test_initialize_process_env_preload_1) ... ok
test_initialize_process_env_preload_2_comma (rocm_sdk.tests.base_test.ROCmBaseTest.test_initialize_process_env_preload_2_comma) ... ok
test_initialize_process_env_preload_2_semi (rocm_sdk.tests.base_test.ROCmBaseTest.test_initialize_process_env_preload_2_semi) ... ok
test_initialize_process_preload_libraries (rocm_sdk.tests.base_test.ROCmBaseTest.test_initialize_process_preload_libraries) ... ok
testConsoleScripts (rocm_sdk.tests.core_test.ROCmCoreTest.testConsoleScripts) ... ok
testInstallationLayout (rocm_sdk.tests.core_test.ROCmCoreTest.testInstallationLayout)
The `rocm_sdk` and core module must be siblings on disk. ... ok
testPreloadLibraries (rocm_sdk.tests.core_test.ROCmCoreTest.testPreloadLibraries) ... ok
testSharedLibrariesLoad (rocm_sdk.tests.core_test.ROCmCoreTest.testSharedLibrariesLoad) ... ok
testConsoleScripts (rocm_sdk.tests.libraries_test.ROCmLibrariesTest.testConsoleScripts) ... ok
testInstallationLayout (rocm_sdk.tests.libraries_test.ROCmLibrariesTest.testInstallationLayout)
The `rocm_sdk` and libraries module must be siblings on disk. ... ok
testSharedLibrariesLoad (rocm_sdk.tests.libraries_test.ROCmLibrariesTest.testSharedLibrariesLoad) ... ok
testCLIPathBin (rocm_sdk.tests.devel_test.ROCmDevelTest.testCLIPathBin) ... ++ Exec [D:\scratch\therock\python_rocm_2]$ 'D:\scratch\therock\python_rocm_2\.venv\Scripts\python.exe' -P -m rocm_sdk path --bin
ok
testCLIPathCMake (rocm_sdk.tests.devel_test.ROCmDevelTest.testCLIPathCMake) ... ++ Exec [D:\scratch\therock\python_rocm_2]$ 'D:\scratch\therock\python_rocm_2\.venv\Scripts\python.exe' -P -m rocm_sdk path --cmake
ok
testCLIPathRoot (rocm_sdk.tests.devel_test.ROCmDevelTest.testCLIPathRoot) ... ++ Exec [D:\scratch\therock\python_rocm_2]$ 'D:\scratch\therock\python_rocm_2\.venv\Scripts\python.exe' -P -m rocm_sdk path --root
ok
testInstallationLayout (rocm_sdk.tests.devel_test.ROCmDevelTest.testInstallationLayout)
The `rocm_sdk` and devel module must be siblings on disk. ... ok
testRootLLVMSymlinkExists (rocm_sdk.tests.devel_test.ROCmDevelTest.testRootLLVMSymlinkExists) ... skipped 'root LLVM symlink only exists on Linux'
testSharedLibrariesLoad (rocm_sdk.tests.devel_test.ROCmDevelTest.testSharedLibrariesLoad) ... ++ Exec [D:\scratch\therock\python_rocm_2]$ 'D:\scratch\therock\python_rocm_2\.venv\Scripts\python.exe' -P -m rocm_sdk path --root
ok

----------------------------------------------------------------------
Ran 25 tests in 3.727s

OK (skipped=1)

@ScottTodd
ScottTodd requested a review from stellaraccident July 7, 2025 20:35
This was referenced Jul 7, 2025
Closed
Merged
@ScottTodd ScottTodd mentioned this pull request Jul 9, 2025
Open
@ScottTodd
ScottTodd merged commit 4486a1d into main Jul 10, 2025
@ScottTodd
ScottTodd deleted the users/scotttodd/windows-test-preload branch July 10, 2025 00:07
@github-project-automation github-project-automation Bot moved this from TODO to Done in TheRock Triage Jul 10, 2025
ScottTodd added a commit that referenced this pull request Jul 10, 2025
@ScottTodd
[torch][windows] Build Windows release PyTorch wheels. (#994)
11d7c75 
Progress on #827 (!!)

Tested at https://github.com/ROCm/TheRock/actions/runs/16185309410,
which triggered these:

* https://github.com/ROCm/TheRock/actions/runs/16186383600 (gfx1151)
* https://github.com/ROCm/TheRock/actions/runs/16186941663 (gfx120X-all)
* https://github.com/ROCm/TheRock/actions/runs/16186966931
(gfx110X-dgpu)

Some next steps after this, roughly in order:

* Monitor scheduled jobs to check that wheels are successfully built and
uploaded
* Download/install and test the wheels on various systems, exercising
components like hipBLASLt
* Update support status:
  * https://github.com/ROCm/TheRock?tab=readme-ov-file#support-status
*
https://github.com/ROCm/TheRock/blob/main/RELEASES.md#python-packages-support-status
* Build torchvision and torchaudio (need some patches)
* Connect with other build/test/release workflows so we get pre-submit,
post-submit, and scheduled coverage for these builds/releases/tests
* Rebase patches onto PyTorch 2.8 (and/or HEAD), land patches upstream
* Add features like Triton support (aotriton?) for optimized flash
attention kernels, etc.
* Expand the list of supported gfx targets

---

~~This depends on some other open PRs, but I have tested parts of it all
the way through to dev wheel publishing and then installation and
testing on a dev machine:
https://github.com/ROCm/TheRock/actions/runs/16181411431/job/45678668536,
#827 (comment)

~~See the other PRs:~~

* ~~#783
* ~~#980
* ~~#988
* ~~#990
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@stellaraccident stellaraccident stellaraccident approved these changes

Assignees

No one assigned

Labels

None yet

Projects

TheRock Triage
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ScottTodd @stellaraccident