Fix GPU nodes failing to start when the CUDA driver is newer than the installed toolkit - #1195
Conversation
A host with a CUDA 13 driver but only a CUDA 12 toolkit installed selected the cuda13 native runtime and failed to start: libcudart.so.13: cannot open shared object file The host profile filled `toolkit_majors` from `nvidia-smi`'s "CUDA Version", which reports the newest CUDA the *driver* supports, not what is installed. The resolver then did exact set membership against that value. Linux native runtimes link libcudart/libcublas without bundling them, so they only load when a matching toolkit major is present. Windows runtimes ship their own copies and are self contained. Split the two facts: - `toolkit_majors` is probed from installed `libcudart.so.<major>` sonames and `/usr/local/cuda-<version>` directories. - `driver_max_major` comes from `nvidia-smi` and is used only as an upper bound. Selection now accepts a runtime when its major is within the driver bound and either the artifact bundles its own CUDA libraries or a matching toolkit is installed. With no installed-toolkit evidence, it falls back to the driver bound rather than rejecting every CUDA runtime. Rejection messages now name the installed majors instead of only the required one, so the mismatch is visible without extra debugging. Fixes #1127
📝 WalkthroughWalkthroughCUDA host profiling separates installed toolkit majors from the driver-supported maximum. Runtime resolution uses both values, supports fully bundled runtimes, and reports distinct rejection reasons. Documentation and formatting describe the updated CUDA behavior. ChangesCUDA compatibility flow
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Host profiling
participant HostCudaProfile
participant Runtime resolver
participant Runtime candidate
participant Rejection formatter
Host profiling->>HostCudaProfile: record toolkit majors and driver maximum
Runtime resolver->>HostCudaProfile: read CUDA capabilities
Runtime resolver->>Runtime candidate: inspect bundled CUDA libraries
Runtime candidate-->>Runtime resolver: return library evidence
Runtime resolver->>Rejection formatter: format driver or toolkit rejection
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/mesh-llm-native-runtime/src/host.rs`:
- Around line 38-42: Add driver_max_major: None to the HostCudaProfile struct
literal in the README example so it matches the required field and compiles.
In `@crates/mesh-llm-native-runtime/src/resolver.rs`:
- Around line 448-452: Update the CUDA compatibility logic around toolkit_majors
to use the driver-bound fallback only when driver_max_major is present; when
both toolkit_majors and driver_max_major are absent, reject the unbundled
runtime with a distinct reason. Add a regression test covering a GPU-label or
architecture-only profile with no detected libcudart installation and no driver
compatibility bound.
- Around line 718-740: Update
self_contained_cuda_runtime_is_accepted_on_newer_driver to set host.os to
windows after profile() and bundled.platform.os to windows after
cuda_runtime(...). Keep the existing artifact ID and CUDA selection assertions
unchanged so the regression test exercises Windows platform matching.
- Around line 464-468: Update artifact_bundles_cuda_runtime so a CUDA artifact
is considered self-contained only when its complete load-time CUDA dependency
set is present and validated, including CUDART, CUBLAS, and CUBLASLt rather than
matching CUDART alone. Reuse the existing library-name normalization and ensure
every matching artifact declares the full required set, or add an explicit
self-contained marker consumed by this function.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: f9bedeef-7f8d-483c-b0cc-82cbc0ba2ff1
📒 Files selected for processing (5)
crates/mesh-llm-commands/src/runtime_native/formatters.rscrates/mesh-llm-hardware-profile/src/lib.rscrates/mesh-llm-native-runtime/README.mdcrates/mesh-llm-native-runtime/src/host.rscrates/mesh-llm-native-runtime/src/resolver.rs
| /// Highest CUDA major the installed driver supports, as reported by | ||
| /// `nvidia-smi`. This is an upper bound, not evidence that a matching | ||
| /// toolkit is installed. | ||
| #[serde(default, skip_serializing_if = "Option::is_none")] | ||
| pub driver_max_major: Option<u32>, |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Update the HostCudaProfile Rust example.
Line 42 adds a required struct field. The example in crates/mesh-llm-native-runtime/README.md lines 127-131 omits it and cannot compile. Add driver_max_major: None to that literal.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/mesh-llm-native-runtime/src/host.rs` around lines 38 - 42, Add
driver_max_major: None to the HostCudaProfile struct literal in the README
example so it matches the required field and compiles.
i386
left a comment
i386
left a comment
There was a problem hiding this comment.
I reviewed the PR head and ran the focused native-runtime and hardware-profile test suites (26 and 17 tests passed). The four existing CodeRabbit comments identify valid issues; I added one additional correctness comment about directory-only CUDA toolkit detection.
| if let Some(output) = command_output("ldconfig", &["-p"]) { | ||
| majors.extend(cuda_majors_from_soname_listing(&output)); | ||
| } | ||
| for base in ["/usr/local/cuda", "/usr/local"] { |
There was a problem hiding this comment.
Verify runtime libraries, not just the toolkit directory name. A /usr/local/cuda-* directory can exist without usable libcudart/libcublas/libcublasLt files (for example, a partial or stale installation). This code would then add the major even though ldconfig found no runtime library, and the resolver can select an unbundled Linux artifact that still fails at load time. The repository’s scripts/lib/cuda-toolkit.sh::cuda_toolkit_library_dir already requires all three libraries; please use equivalent file checks here before treating the major as installed.
i386
left a comment
i386
left a comment
There was a problem hiding this comment.
Found one additional compatibility gap in toolkit discovery; inline below.
| /// their own copies and do not depend on this. | ||
| fn installed_cuda_toolkit_majors() -> BTreeSet<u32> { | ||
| let mut majors = BTreeSet::new(); | ||
| if let Some(output) = command_output("ldconfig", &["-p"]) { |
There was a problem hiding this comment.
[P2] Probe configured CUDA library roots too. ldconfig -p plus /usr/local/cuda-* misses toolkits installed under a custom prefix or exposed only through LD_LIBRARY_PATH/CUDA_HOME (for example Conda or /opt/cuda-12). On a host with libcudart.so.12 in one of those paths and a driver advertising CUDA 13, toolkit_majors stays empty and the resolver falls back to selecting cuda13—the load failure this PR is intended to prevent. Please include the loader/environment search paths (with the same complete-library validation) or explicitly handle this installation shape.
…inked runtime Review follow-up on the toolkit-vs-driver split. Toolkit detection was unsound in two ways. It treated a `/usr/local/cuda-*` directory name as proof of an installed toolkit, which a stale or partial install satisfies without usable libraries, and it only looked for `libcudart`, so a major missing `libcublas`/`libcublasLt` still counted as present. Detection now probes the loader's own view (the `ldconfig` cache plus `LD_LIBRARY_PATH`) and requires the complete `libcudart` + `libcublas` + `libcublasLt` set at a matching major, mirroring `scripts/lib/cuda-toolkit.sh::cuda_toolkit_library_dir`. Directory names are no longer evidence: a toolkit the loader cannot see cannot be loaded. Selection is now fail-closed. Previously an empty `toolkit_majors` accepted any host-linked runtime within the driver bound, so a host whose toolkit was not detected could still select cuda13 and hit the exact `libcudart.so.13: cannot open shared object file` this change exists to prevent. Such candidates are rejected with a distinct `CudaToolkitNotDetected` reason naming the libraries looked for and the `MESH_LLM_CUDA_TOOLKIT_MAJORS` escape hatch. CPU fallback still applies, so these hosts serve rather than fail. Self-containment likewise requires the full bundled set. Declaring `cudart` alone left `cublas` to the host while claiming independence from it. Also fixes the Windows regression test, which ran against a Linux host profile and artifact platform and therefore never exercised Windows selection, and adds the missing `driver_max_major` field to the README profile example.
|
Thanks both — the review found a real hole in the original approach, and one of my own tests was fake. Pushed First, a correction to my own PR descriptionI claimed CI missed this because "all 18 CI references set The conclusion still holds, for a simpler reason: no CI job invokes host runtime selection at all. There are no @i386 — verify libraries, not directory namesAccepted, and this was the more serious of the two. Directory-name scanning is gone entirely. Detection now requires the complete Verified against the host that triggered this, which does satisfy the full triple: @i386 — [P2] custom prefixesPartly addressed, and I want to be explicit about where I landed rather than quietly not doing it.
I deliberately did not probe Your P2 also identified the case that made me change the design: on such a host @coderabbitai — fail-open fallbackAccepted, and taken further than suggested. You proposed using the fallback only when The fail-open behaviour existed so unprobeable hosts were not left with zero CUDA candidates. Added @coderabbitai — Windows test and READMEBoth correct, both fixed. The Windows test was genuinely fake: it ran against a Linux host profile and a Linux artifact platform, so it never exercised Windows selection at all — it only changed the artifact ID string. It now sets @coderabbitai — explicit self-containment flagAgreed the I have not added an explicit Out of scope, worth a separate look
The deeper fix both of you are circling is to stop inferring loadability and verify it — probe the candidate's full library set in a short-lived subprocess before declaring it compatible, which makes probe accuracy irrelevant. That is a larger change and interacts with the pre-download selection path ( ValidationSix regression tests now cover: newer-driver/older-toolkit, self-contained Windows runtime on a newer driver, partially-bundled artifact not treated as self-contained, host-linked rejected without a detected toolkit, and CPU fallback in that case. |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
git status --short
git show --stat --oneline --decorate a4d16e18
git show --format= --name-only a4d16e18
ast-grep outline crates/mesh-llm-native-runtime/src/resolver.rs --items all --match 'Cuda|cuda|CandidateRejection|Host' || true
ast-grep outline crates/mesh-llm-hardware-profile/src/lib.rs --items all --match 'Cuda|cuda|toolkit|library' || true
ast-grep outline crates/mesh-llm-native-runtime/src/host.rs --items all --match 'Cuda|cuda|Host' || true
ast-grep outline crates/mesh-llm-commands/src/runtime_native/formatters.rs --items all --match 'Cuda|cuda|format' || true
git show --format= --find-renames a4d16e18 -- \
crates/mesh-llm-native-runtime/src/resolver.rs \
crates/mesh-llm-hardware-profile/src/lib.rs \
crates/mesh-llm-native-runtime/src/host.rs \
crates/mesh-llm-commands/src/runtime_native/formatters.rs \
crates/mesh-llm-native-runtime/README.md
printf '\n--- min_driver declarations/usages ---\n'
rg -n -C 3 --glob '*.rs' 'min_driver|CudaRuntimeRequirements' crates
printf '\n--- CI host-selection invocations ---\n'
rg -n -i -C 2 'runtime\s+(list\s+--available|install)|host_runtime_profile' .github 2>/dev/null || trueLength of output: 21750 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- CUDA profile detection and tests ---'
sed -n '285,525p' crates/mesh-llm-hardware-profile/src/lib.rs
rg -n -C 8 --glob '*.rs' \
'(installed_cuda_toolkit_majors|loader_search_dirs|record_cuda_soname|MESH_LLM_CUDA_TOOLKIT_MAJORS|detect_cuda_profile)' \
crates/mesh-llm-hardware-profile
printf '%s\n' '--- Resolver evaluation and diagnostics ---'
sed -n '380,500p' crates/mesh-llm-native-runtime/src/resolver.rs
sed -n '700,870p' crates/mesh-llm-native-runtime/src/resolver.rs
rg -n -C 7 --glob '*.rs' \
'(CudaToolkitNotDetected|CudaToolkitMajorAboveDriver|evaluate_cuda_toolkit_major|artifact_bundles_cuda_runtime|available)' \
crates/mesh-llm-native-runtime crates/mesh-llm-commands
printf '%s\n' '--- Host CUDA profile construction ---'
sed -n '25,48p' crates/mesh-llm-native-runtime/src/host.rsLength of output: 141083
The resolver now rejects host-linked CUDA runtimes when Linux toolkit detection now requires the matching-major The Windows regression test now uses a Windows host and a Windows artifact. The bundled-runtime check now requires the complete CUDA library set. The README includes The test coverage correctly verifies CPU fallback when no host-linked CUDA runtime is safe to select. The diagnostics also report the rejection reason. I could not independently run the reported Cargo validation commands in this sandbox. The source changes and test definitions are consistent with the reported results. The ✏️ Learnings added
If you found this review helpful, would you consider giving us a shout-out on X? Thank you for using CodeRabbit! You are interacting with an AI system. |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/mesh-llm-hardware-profile/src/lib.rs`:
- Around line 335-359: Update installed_cuda_toolkit_majors and the related
evidence handling to retain each observed library’s resolved path, then validate
that target exists, is not dangling, and matches the current ELF architecture
before setting CudaLibraryEvidence flags. Apply the same validation to both
ldconfig output and loader_search_dirs entries, so stale cache records and
unsuitable directories cannot make a major appear installed; add regression
coverage for stale and dangling entries.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 67a965bb-3c44-4add-b028-d67c928195ae
📒 Files selected for processing (4)
crates/mesh-llm-commands/src/runtime_native/formatters.rscrates/mesh-llm-hardware-profile/src/lib.rscrates/mesh-llm-native-runtime/README.mdcrates/mesh-llm-native-runtime/src/resolver.rs
🚧 Files skipped from review as they are similar to previous changes (3)
- crates/mesh-llm-native-runtime/src/resolver.rs
- crates/mesh-llm-commands/src/runtime_native/formatters.rs
- crates/mesh-llm-native-runtime/README.md
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/mesh-llm-hardware-profile/src/lib.rs`:
- Around line 464-480: Update valid_cuda_library_target to open the candidate
file and read only the ELF header bytes it inspects using read_exact, rather
than loading the entire file with fs::read. Preserve the existing
minimum-length, ELF magic, endianness, machine, and current_elf_identity
validation behavior, returning false when opening or reading the required header
bytes fails.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: aca5f94f-af24-4364-a9b7-6c1eb62e1256
📒 Files selected for processing (1)
crates/mesh-llm-hardware-profile/src/lib.rs
i386
left a comment
i386
left a comment
There was a problem hiding this comment.
Reviewed and fixed the actionable CUDA comments. The change now discovers CUDA evidence from configured/custom roots, validates actual library targets against the host ELF architecture, and reads only the required ELF header bytes instead of loading whole shared libraries. Validated with cargo fmt --all -- --check, cargo check -p mesh-llm-hardware-profile, cargo clippy -p mesh-llm-hardware-profile --all-targets -- -D warnings, and cargo test -p mesh-llm-hardware-profile --lib (18 passed). Approving.
There was a problem hiding this comment.
🧹 Nitpick comments (1)
crates/mesh-llm-hardware-profile/src/lib.rs (1)
854-890: 🎯 Functional Correctness | 🔵 Trivial | 🏗️ Heavy liftExercise the production toolkit-discovery path.
This test manually builds
evidenceand checksappend_cuda_root_dirs. It does not call the production scanner that consumesldconfigoutput and configured search directories. The test can pass if custom roots are ignored by the scanner.Add a test seam that accepts search directories or loader input. Assert that the complete CUDA 12 set is discovered and the incompatible CUDA 13 set is rejected.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-llm-hardware-profile/src/lib.rs` around lines 854 - 890, Update cuda_library_probe_includes_custom_roots_and_rejects_invalid_targets to exercise the production CUDA toolkit-discovery scanner rather than manually constructing evidence or testing append_cuda_root_dirs directly. Add a test seam allowing loader input and/or search directories to be supplied, invoke that scanner with the temporary root, and assert it discovers the complete CUDA 12 set while rejecting the incompatible CUDA 13 libraries.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@crates/mesh-llm-hardware-profile/src/lib.rs`:
- Around line 854-890: Update
cuda_library_probe_includes_custom_roots_and_rejects_invalid_targets to exercise
the production CUDA toolkit-discovery scanner rather than manually constructing
evidence or testing append_cuda_root_dirs directly. Add a test seam allowing
loader input and/or search directories to be supplied, invoke that scanner with
the temporary root, and assert it discovers the complete CUDA 12 set while
rejecting the incompatible CUDA 13 libraries.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: bc3b6226-2e0d-46c8-a9c6-81a978ae1ae4
📒 Files selected for processing (1)
crates/mesh-llm-hardware-profile/src/lib.rs
michaelneale
left a comment
michaelneale
left a comment
There was a problem hiding this comment.
🤖 (micn's agent) Reviewed against the v0.75.0 release manifest and ran the test suites locally. The core design is right and this is needed for Linux GPU nodes on 0.75.x — the toolkit/driver split matches reality: I verified all four Linux CUDA artifacts in the published native-runtimes.json bundle no cudart/cublas libraries while windows-x86_64-cuda12 bundles all three, so artifact_bundles_cuda_runtime keys off exactly the right evidence. Local runs: mesh-llm-native-runtime 31 passed, mesh-llm-hardware-profile 18 passed, clippy -D warnings clean on all three touched crates.
Findings, in descending severity:
1. valid_cuda_library_target reads the entire library into memory to check 20 header bytes.
fn valid_cuda_library_target(path: &Path) -> bool {
let Ok(bytes) = fs::read(path) else { return false; };
if bytes.len() < 20 || &bytes[..4] != b"\x7fELF" { ... }libcublasLt.so.12 is routinely 400–700 MB and libcublas.so.12 100+ MB. This runs during host profile detection — i.e., startup — potentially several times per library (ldconfig line + each matching search dir). Please read just the first 20 bytes (File::open + read_exact into a [u8; 20]). Same correctness, avoids transiently allocating ~1 GB on CUDA hosts.
2. PR description contradicts the code on the no-toolkit fallback. The body says "With no installed-toolkit evidence at all it falls back to the driver bound, so hosts we cannot probe are not left with zero CUDA candidates" — but the code (and the host_linked_runtime_is_rejected_without_detected_toolkit test) deliberately rejects host-linked runtimes when toolkit_majors is empty, falling back to CPU. I think the code is right and the description is stale from an earlier iteration; the code comment ("reject rather than guess") gives the correct rationale. Worth fixing the description before merge so 0.75.1 release notes don't inherit the wrong claim.
3. Doc/code mismatch on directory scanning. The doc comment on installed_cuda_toolkit_majors says the probe "deliberately" uses only "the loader's own view — the ldconfig cache and LD_LIBRARY_PATH — rather than guessing from installation directory names", but loader_search_dirs() then scans CUDA_HOME/CUDA_PATH/CUDA_ROOT/CONDA_PREFIX and everything under /usr/local/cuda*. The ELF validation makes that scan safe against empty/wrong-arch dirs, so the behavior is fine — but the comment should be updated to describe what it actually does.
Related question, non-blocking: if the only toolkit evidence is /usr/local/cuda-12/lib64 and that dir is not in the ldconfig cache or LD_LIBRARY_PATH, we mark cuda12 as installed and select the cuda12 runtime — can the runtime actually resolve libcudart.so.12 at load in that case (RPATH? explicit dlopen paths?), or do we reintroduce the same cannot open shared object file failure one step later? If load would fail, evidence from non-loader dirs may need to also extend the load path, or be excluded.
4. SDK surface note. CandidateRejection::CudaToolkitMajorMismatch gained an installed field and two new variants were added. The enum is re-exported through mesh-llm-sdk / mesh-llm-api surfaces, so exhaustive matches in downstream consumers will break at compile time. Fine for a minor release, just deserves a line in the changelog/PR description.
5. Small nit: driver_max_major takes next_back() of all majors parsed from nvidia-smi output. On multi-GPU boxes with mixed driver stacks (rare, but Jetson + dGPU exists) the max across GPUs is the right call for an upper bound, so this is correct — just noting it was considered.
None of these are architectural objections — #1 is the only change I'd insist on before merge, #2 is a description fix, #3 a comment fix. Overall this correctly turns a startup crash into correct selection, and the rejection messages in runtime list --available are a genuine debuggability win.
|
🤖 (micn's agent) Post-merge review against What I verified in the merged code:
Two residual items, neither back-out-worthy:
|
3 participants
What this fixes
A machine with an NVIDIA driver that advertises a newer CUDA than the toolkit actually installed on it can now start and use its GPU.
Before this change, such a host picked the
cuda13native runtime and died at startup:On Windows the same misdetection silently fell back to Vulkan, which then reported 0 GPUs (#1127).
This is common: NVIDIA drivers routinely support a newer CUDA than the toolkit installed alongside them. On the box I hit this on,
nvidia-smireportsCUDA Version: 13.0while the only runtime library present islibcudart.so.12.mesh-llm runtime list --availablenow also explains the mismatch instead of stating only the requirement:Why it happened now
0.74.0 shipped a ~340 MB monolithic binary that linked
libcudart.so.12directly, so it drove the GPU through its own linkage and the misdetection was harmless.0.75.0 is the ~90 MB backend-neutral host with no CUDA linkage — all CUDA now comes from the resolved native runtime. The detection bug is not new, but it only became fatal once runtime selection had to be correct.
Architecture
HostCudaProfileconflated two different facts. They are now separate:toolkit_majorslibcudart.so.<major>sonames,/usr/local/cuda-<version>driver_max_majornvidia-smiLinux runtimes link
libcudart/libcublaswithout bundling them, so they need a matching host toolkit. Windows runtimes ship their own copies (cudart64_12.dll) and are self contained — verified against the v0.75.0 release manifest.Selection accepts a runtime when its major is within the driver bound and either the artifact bundles its own CUDA libraries or a matching toolkit is installed. With no installed-toolkit evidence at all it falls back to the driver bound, so hosts we cannot probe are not left with zero CUDA candidates.
Validation
Reproduced and confirmed on a real RTX 3090 host (driver 580.173.02 advertising CUDA 13.0, toolkit 12.6 only):
Forcing the correct selection made 0.75.0 serve normally on that node — 12 peers, GPU at 58%, inference returning
OK. This PR makes that selection happen automatically.Three regression tests added, covering the newer-driver/older-toolkit case, self-contained Windows runtimes on a newer driver, and the no-evidence fallback.
Note on CI coverage
CI never caught this because all 18 CI/release references set
MESH_LLM_CUDA_TOOLKIT_MAJORexplicitly, so auto-detection is never exercised, and no CI host has the driver-newer-than-toolkit shape. Worth considering a job that leaves detection unset — happy to add it here or follow up separately.Fixes #1127
Summary by CodeRabbit
Bug Fixes
Documentation