Skip to content

Studio: find the AMD Vulkan driver through the Windows device registrations - #10564

Merged
danielhanchen merged 7 commits into
unslothai:mainfrom
oobabooga:vulkan-icd-device-registry
Sep 9, 2026
Merged

danielhanchen merged 7 commits into
unslothai:mainfrom
oobabooga:vulkan-icd-device-registry

Conversation

@oobabooga

Copy link
Copy Markdown
Member

Problem

Windows Strix Halo can still show Automatic (ROCm) despite having a working Vulkan driver and the gfx1150/gfx1151 preference introduced in #10381.

_amd_vulkan_icd_manifest_paths() only checked HKLM\SOFTWARE\Khronos\Vulkan\Drivers. On the tested Radeon 8060S, that legacy key was absent: AMD registered its driver through device-specific VulkanDriverName values instead. The driver-presence check therefore returned false, and Automatic continued to select ROCm.

Change

  • Read VulkanDriverName from present display adapters and SoftwareComponents before the legacy key, supporting both REG_SZ and REG_MULTI_SZ.
  • Use CfgMgr32 to identify present instances. Removed devices can retain both registrations and driver files, so file existence alone is insufficient. Skip a class if its presence cannot be determined.
  • Reject missing manifests and relative registration paths, and deduplicate results across registration locations.

Existing library validation, 64-bit filtering, environment overrides, and backend-selection guards remain in effect.

A/B

Live Windows run on Ryzen AI MAX+ 395 / Radeon 8060S with Adrenalin 32.0.22018.5. Both sides used llama.cpp release b10840-mix-d5c17a0 and supplied gfx1151 through the existing remembered-architecture mechanism because the runner had no HIP SDK.

Check Baseline Fix
Valid AMD manifest discovered No Yes
AMD Vulkan driver present False True
Automatic resolves to ROCm Vulkan
Selected install kind windows-rocm windows-vulkan

Both backends remained available. The fix found the valid adapter manifest and rejected a stale SoftwareComponent registration whose manifest was missing.

Verification

  • Regression tests cover device registrations, supported value types, removed devices, missing files, deduplication, legacy discovery, and overrides.
  • CfgMgr32 tests exercise enumeration, output parameters, presence filters, and failure handling through a portable test double.
  • test_install_resolve_prebuilt.py, test_llama_backend_selection.py, and test_llama_backend_switch.py: 399 passed, 1 skipped.
  • git diff --check passed. Subsequent comment-only cleanup preserved the executable AST.

Scope

This repairs detection for automatic backend selection. Existing ROCm installations can re-apply Automatic through the existing migration offer.

SoftwareComponents are filtered by presence without checking their parent-adapter association, so discovery is broader than the Vulkan loader's traversal. Relative registration paths remain unsupported.

The live test covered discovery and backend resolution. No model inference or benchmark was run.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 8, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-09T10:29:49.350222Z 8e4ffd8 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@mahiatlinux

Copy link
Copy Markdown
Collaborator

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: 52128e2860

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Follow-ups from reviewing the device-registry discovery, all checked against
windows_get_device_registry_files in the Vulkan loader's loader_windows.c.

Skip devnodes pending reboot. An Adrenalin update writes VulkanDriverName and
drops its manifest before the restart that binds the driver. In between, the
adapter is present, the file is on disk, and the loader skips the devnode
(DN_HAS_PROBLEM with CM_PROB_NEED_RESTART or DN_NEED_RESTART). Counting it
routed a gfx115x host to the Vulkan bundle for a driver that cannot load yet,
which is the silent CPU fallback _amd_vulkan_icd_present exists to prevent.
CM_Get_DevNode_Status now gates each device, and an unreadable status is skipped
for the same reason the loader skips it.

Retry the device-id list on CR_BUFFER_SMALL. A device arriving between the size
query and the read makes the block outgrow the buffer, and the API refuses
rather than truncating. That transient reported presence as unknown, both
classes were skipped, and the host this feature exists for quietly installed
the HIP bundle. The loader loops on the same condition; this bounds the loop at
four attempts so a churning list answers unknown instead of spinning.

Scope the registry catch to one instance rather than the whole class. A single
unreadable entry discarded every remaining instance in that class, and on a
two-adapter host the integrated part often enumerates first, so the AMD
registration was the one lost.

Tests: pending-reboot and unrelated problem codes, an unreadable status, a
device arriving mid-enumeration, a list that never settles, and one unreadable
instance not costing the rest of the class.

Also cover what the existing suite could not reach. _FakeCfgMgr answers through
native ctypes, where sizeof(c_wchar) is 4 on Linux and 2 on Windows, so the
byte-to-element conversion for CM_Get_DevNode_Registry_PropertyW was only ever
exercised at the runner's width; a ctypes shim now runs the probe at Windows'
widths, and reverting the conversion fails it. Registration paths are checked
through ntpath, since tmp_path made every existing case posixpath, where
"C:\..." is not absolute. The loader-override test now asserts that neither the
registry nor cfgmgr32 was touched. The restricted-subkey case gained a
digit-named instance, without which the PermissionError arm was unreachable.
_CR_FAILURE was 0x0D, which is CR_NO_SUCH_DEVNODE; cfg.h puts CR_FAILURE at
0x13.
@danielhanchen

Copy link
Copy Markdown
Member

Reviewed this and pushed a follow-up commit to the branch (ef247c6). Summary of what I checked and what changed.

Verification

Linux host, so no Windows execution, no AMD driver load and no inference. What that leaves answerable is the discovery logic, the routing consequences, and the byte/character arithmetic under a modelled Windows data layout.

  • The four CfgMgr32 contracts check out against the SDK header and MSDN: DEVINST is a DWORD, the device-id list length is in characters, CM_Get_DevNode_Registry_PropertyW's pulLength is in bytes (_Out_writes_bytes_opt_(*pulLength)), and CM_DRP_DRIVER 0x0A / CM_LOCATE_DEVNODE_NORMAL 0 / FILTER_PRESENT 0x100 / FILTER_CLASS 0x200 all match. With no argtypes, ctypes' default str -> wchar_t* and pointer conversions are the right ones here, and nothing truncates on 64-bit.
  • Ran a scenario table across [Windows, Linux, WSL, macOS] x [NVIDIA, AMD, CPU-only] plus the registry shapes, loader overrides, visibility masks, explicit backends, existing markers and the gfx sweep, on the merge base and on the branch: 46 scenarios, 4 changed, and the 4 are the intended fix (adapter and SoftwareComponent registrations, REG_SZ and REG_MULTI_SZ, gfx1150 and gfx1151). Nothing else moved. The Windows registry and cfgmgr32 are untouched off Windows, and the POSIX XDG branch is unchanged.
  • windows-hip stays behind windows-vulkan in the plan, so a Vulkan asset that fails validation does not drop a working ROCm host to CPU. An existing marker recording rocm/hip/cuda still opts out; only auto re-detects.
  • The two red CI jobs are a stale base, not this PR. The same five tests fail identically at b75ed8815 (test_deliberate_crashes_suppress_cores.py, test_windows_amd_gpu_scan_fallback.py), and none of those files are touched here. A rebase should clear them.
  • The wider suites at 399 passed / 1 skipped matched your number exactly.

What the follow-up changes

Three things where the implementation diverged from windows_get_device_registry_files in the loader's loader_windows.c in ways that change the answer.

  1. Pending reboot. An Adrenalin update writes VulkanDriverName and drops its manifest before the restart that binds the driver. In between the adapter is present, the file exists, and the loader skips the devnode (DN_HAS_PROBLEM with CM_PROB_NEED_RESTART or DN_NEED_RESTART). Counting it routed a gfx115x host to Vulkan for a driver that cannot load yet, which is the silent CPU fallback the probe exists to prevent. CM_Get_DevNode_Status now gates each device.
  2. CR_BUFFER_SMALL. A device arriving between the size query and the read makes the block outgrow the buffer, and the API refuses rather than truncating. One CR_BUFFER_SMALL turned presence into "unknown", skipped both classes, and quietly installed the HIP bundle. The loader loops on this; the retry is bounded at four attempts so a churning list answers unknown instead of spinning.
  3. Catch scope. The except Exception sat at the class level, so one unreadable instance discarded every remaining instance in that class. On a two-adapter host the integrated part often enumerates first, so the AMD registration was the one lost. It is now per instance.

Plus test coverage for holes the existing doubles could not reach:

  • _FakeCfgMgr runs on native ctypes, where sizeof(c_wchar) is 4 on Linux and 2 on Windows, so the byte-to-element conversion was only ever exercised at the runner's width. A ctypes shim now runs the probe at Windows' widths; reverting the conversion to a hardcoded // 4 fails it.
  • Registration paths are checked through ntpath. Every existing case used tmp_path, i.e. posixpath, where C:\... is not absolute and a relative Windows path is.
  • The loader-override test now asserts that neither the registry nor cfgmgr32 was read, not just that the returned list was right.
  • ("Properties", "denied") never reached the PermissionError arm, since "Properties".isdigit() is False and the walk short-circuits before OpenKey. Added a digit-named instance that does.
  • _CR_FAILURE was 0x0D, which is CR_NO_SUCH_DEVNODE. cfg.h puts CR_FAILURE at 0x13.

Left as-is, worth recording

SoftwareComponents are still enumerated by presence rather than reached as children of a present display adapter, which is what the loader does (CM_Get_Child / CM_Get_Sibling, filtered on CM_DRP_CLASSGUID). Your description already says this. An orphaned AMD component that the loader would never read can therefore answer for the driver. The blast radius is bounded: _amd_vulkan_icd_present() has exactly one caller, and it only runs when every physical gfx is in {gfx1150, gfx1151} with ROCm present, no physical NVIDIA, no visibility mask and no explicit backend. Not blocking, but if you want to close it, walking the children of each present adapter is the same call shape as the loader uses.

Relative VulkanDriverName values are still skipped rather than resolved against the device's DriverStore directory. A silent miss rather than a wrong answer, and no host has been observed registering one.

Nice fix, and thanks for the live A/B on real hardware.

@danielhanchen

Copy link
Copy Markdown
Member

Also merged current main into the branch, which settles the CI question.

Every one of the five Repo tests (CPU) failures reproduced identically at the merge base b75ed8815 with the branch reverted, and all five pass on the merged tree (156 passed). They were a stale base, not this PR. pre-commit.ci now passes too; it was reformatting 1082 files repo-wide and could not push the fix because the branch had diverged.

linux ubuntu2404-arm-root is still red, and is not ours: built from source: nvidia-npp-cu13 -- these must resolve to wheels on a clean machine. #10588 fails the same job with the same line, and nothing here touches dependencies.

@Imagineer99

Copy link
Copy Markdown
Member

this matches the gap I reproduced in a controlled Windows A/B test, automatic picked ROCm with adapter-only registration and switched to Vulkan when discovery included it, restoring the original discovery switched it back to ROCm, all 13 cases passed, good to see your hardware test confirms it too

danielhanchen added a commit to shimmyshimmer/unsloth-staging-4 that referenced this pull request Sep 9, 2026
@unslothai unslothai deleted a comment from chatgpt-codex-connector Bot Sep 9, 2026
@unslothai unslothai deleted a comment from chatgpt-codex-connector Bot Sep 9, 2026
@danielhanchen

Copy link
Copy Markdown
Member

@codex review

@danielhanchen

Copy link
Copy Markdown
Member

@codex security review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6f6c22a273

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread studio/install_llama_prebuilt.py Outdated
return False
if not status.value & _DN_HAS_PROBLEM:
return True
return problem.value not in (_CM_PROB_NEED_RESTART, _DN_NEED_RESTART)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Check DN_NEED_RESTART in the status bitmask

When a driver update is pending a reboot and CM_Get_DevNode_Status reports DN_NEED_RESTART, that constant is a flag in status, not a value in problem. Comparing problem.value with 0x100 therefore misses this state (and the preceding early return also accepts it when DN_HAS_PROBLEM is clear), so the newly registered manifest can be treated as usable before the driver is bound, routing automatic selection to Vulkan and potentially leaving inference on CPU. Test status.value & _DN_NEED_RESTART directly and reserve the problem comparison for CM_PROB_NEED_RESTART.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, fixed in 8e4ffd8. cfg.h lists DN_NEED_RESTART (DN_LIAR, 0x100) under "Device Instance status flags, returned by call to CM_Get_DevInst_Status", and the CM_PROB_ space stops at 0x39 (CM_PROB_GUEST_ASSIGNMENT_FAILED), so comparing it against pulProblemNumber could never match. Reproduced both halves against the code at head: a devnode with the bit set in status and DN_HAS_PROBLEM clear took the early return and was counted, and so did the bit alongside an unrelated problem code. The check now tests status.value & _DN_NEED_RESTART before the problem word and independently of DN_HAS_PROBLEM, and keeps the problem comparison for CM_PROB_NEED_RESTART alone. The regression test carried the same mistake (it passed 0x100 as a problem code, which is why it passed); it is now parametrized over the problem code, the status bit alone, and the status bit beside an unrelated problem, and the last two fail against the previous check. Worth noting the Vulkan loader has the same expression in windows_get_device_registry_files, so this is stricter than the loader now, which is the right direction here: a device awaiting a restart has no bound driver whether or not the loader reads its key.

…oblem code

cfg.h lists DN_NEED_RESTART (DN_LIAR, 0x100) under "Device Instance status flags,
returned by call to CM_Get_DevInst_Status", so it is a bit in pulStatus. The check
compared it against pulProblemNumber, where CM_PROB_ codes stop at 0x39, so that
half of the comparison could never match. A devnode reporting the status bit
without also raising DN_HAS_PROBLEM took the earlier return and was counted, which
is the pending-reboot window the check exists to exclude: the manifest is
registered and on disk, the driver is not bound until the restart, and automatic
selection would move a gfx115x host to Vulkan and leave inference on CPU.

Test the bit in the status word, before the problem word and independently of
DN_HAS_PROBLEM, and reserve the problem comparison for CM_PROB_NEED_RESTART.

The parametrized regression test carried the same mistake, passing 0x100 as a
problem code; it now covers the status bit alone and the status bit beside an
unrelated problem, and both fail against the previous check.
@unslothai unslothai deleted a comment from chatgpt-codex-connector Bot Sep 9, 2026
@danielhanchen

Copy link
Copy Markdown
Member

@codex review

1 similar comment
@danielhanchen

Copy link
Copy Markdown
Member

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: 8e4ffd86d5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

danielhanchen added a commit to shimmyshimmer/unsloth-staging-4 that referenced this pull request Sep 9, 2026
@danielhanchen
danielhanchen merged commit 0f58608 into unslothai:main Sep 9, 2026
47 of 59 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants