Skip to content

Add hipSOLVER - #63

Merged
stellaraccident merged 8 commits into
mainfrom
hipsolver
Feb 12, 2025
Merged

stellaraccident merged 8 commits into
mainfrom
hipsolver

Conversation

@marbre

@marbre marbre commented Feb 5, 2025 •

Copy link
Copy Markdown
Member

Includes workarounds for #82, #83

stellaraccident added a commit that referenced this pull request Feb 11, 2025
* Prefetch from #63 to get docker container ready.
* Temporary while working out deps.
@stellaraccident

Copy link
Copy Markdown
Collaborator

I pushed f69fd0d individually to prep the docker container.

@stellaraccident

Copy link
Copy Markdown
Collaborator

Still seems to need some work...

@stellaraccident
stellaraccident force-pushed the hipsolver branch 2 times, most recently from 6524eec to c967ba5 Compare February 12, 2025 01:57
@stellaraccident
stellaraccident merged commit b31d7fe into main Feb 12, 2025
@stellaraccident
stellaraccident deleted the hipsolver branch February 12, 2025 08:21
powderluv added a commit to powderluv/TheRock that referenced this pull request Jul 9, 2026
…uards + diagnostics (WIP, not yet a fix)

Investigation scaffolding + defensive guards for the torch process-teardown hang
(matmul computes correctly, then the process hangs at interpreter shutdown -> the
pytest subprocess times out, rc=-999).

Mechanism (fully diagnosed): a torch finalizer launches a kernel during Python
interpreter finalization; the first-use code-object load does a CPU write over the
PCIe BAR that wedges un-killably in a non-alertable kernel-mode wait on the
by-then-dying driver (survives os._exit / TerminateProcess / taskkill /F). These
wddm-layer guards do NOT fix that -- the wedge is the CPU-BAR memory access, not an
escape. The real fix is torch-side (don't launch a kernel at finalization) or a
deeper loader change (SDMA code-object copy). Kept as defensive/diagnostic.

gpu_init.cpp:
- g_gpuBroughtUpOnce: skip a 2nd full PSP/GFX bring-up per process (recipeBootload +
  wddmGfxBringUp) so a teardown re-init can't re-run the recipe on a live GPU.
- ROCR_LITE_SKIP_POST_BRINGUP_MEMSET (env, default off): skip the post-bring-up
  rcpAllocVram zeroing memset (itself a CPU-over-BAR write).
- ROCR_LITE_WATCHDOG (env): native all-thread stack dumper (Toolhelp + StackWalk64 +
  dbghelp) that survives CPython finalization where faulthandler is blind.
- ROCR_LITE_BACKTRACE (env): symbolized backtrace at rcpAllocVram-fail /
  wddmGfxBringUp-entry.

wddm_lite.cpp:
- escape-guard: fast-fail WddmLite::escape() when RtlDllShutdownInProgress() or
  Py_IsFinalizing() -- don't issue GPU escapes during process teardown. Defensive.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
powderluv added a commit to powderluv/TheRock that referenced this pull request Jul 9, 2026
…utdown guards + diagnostics (WIP, not yet a fix)"

This reverts commit 3b368e8.
powderluv added a commit to powderluv/TheRock that referenced this pull request Jul 9, 2026
…nd driver guards

Points the superproject at the rocm-systems commits that fix the Windows lite::
torch teardown hang (skip SyncAllStreams drain at process exit) and revert the
disproven driver-object bring-up guards.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants