Studio: source CPU llama.cpp prebuilts from unslothai/llama.cpp

[oobabooga]

Finishes what #5963 started: moves the last llama.cpp prebuilt artifacts off ggml-org and onto the unslothai/llama.cpp fork. #5963 already moved CUDA, ROCm and macOS. This PR moves the CPU bundles (Linux + Windows, x86_64 + arm64), so the installer no longer pulls any ggml-org prebuilt on the default path.

unslothai/llama.cpp#23 is already merged, so this just needs the daily workflow to run once to publish a release with the CPU bundles (which look like https://github.com/oobabooga/llama.cpp/releases/tag/b9625-mix-2d6bd50).

What changed

Routing to the fork is now unconditional, in three mirrored layers (CPU-only Linux/Windows used to route to ggml-org; only GPU and macOS used the fork):

• setup.sh: replaced the _LINUX_HAS_GPU detection and if/elif/else with a single _HELPER_RELEASE_REPO="unslothai/llama.cpp".
• setup.ps1: $HelperReleaseRepo is now an unconditional fork assignment.
• install_llama_prebuilt.py: dropped published_repo_for_host() (it had become a constant) and the dead amd_tooling probe.

arm64 CPU selectors added to the manifest path (x86_64 CPU already selected from the fork manifest; arm64 was the gap):

• resolve_release_asset_choice: new windows-arm64 branch.
• _linux_published_attempts: the CPU branch is now arch-aware (linux-arm64 vs linux-cpu).
• setup.sh arm64 GPU last-resort fallback now pulls the fork's app-*-linux-arm64-cpu.tar.gz instead of ggml-org's ubuntu-arm64.

The ggml-org filename branches stay in place, reachable only via an explicit --published-repo ggml-org override, so the diff stays small and reversible. There is no s390x/riscv/ppc CPU path to regress; the only hosts that pulled ggml-org CPU artifacts were x86_64 and arm64, both now on the fork.

Follow-up

scripts/unsloth/assemble_metadata.py on unslothai/llama.cpp master still emits two arm64 upstream passthrough checksums (linux-arm64-upstream, windows-arm64-upstream). After this PR gets merged, those checksums can be dropped.

[gemini-code-assist]

Code Review

This pull request simplifies the installation routing logic for llama.cpp prebuilts by unconditionally routing all supported hosts (including CPU-only hosts on Linux and Windows) to the unslothai/llama.cpp fork, which now ships CPU bundles. This removes the previous fallback logic to the upstream ggml-org repository. The review feedback suggests two robust improvements: using defensive variable expansion in the shell script to prevent unbound variable errors under 'set -u', and normalizing line endings in test files to ensure cross-platform compatibility on Windows.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5048bb766d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c0e34f99f8

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[danielhanchen]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: f74ee3e8f6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[danielhanchen]

Reviewed the routing change and replicated the branch onto current main to isolate the CI failures.

Routing logic looks correct and backwards compatible. I simulated the full host matrix (Linux/Windows/macOS x x86_64/arm64 x NVIDIA/ROCm/CPU) against the fork manifest on this branch vs main: every GPU and macOS path resolves identically, and only the intended CPU paths change (Linux/Windows x64+arm64 now pull the fork CPU bundle; unknown Linux arches correctly fall back to a source build). The explicit --published-repo ggml-org override still resolves the same ggml assets. The four CPU bundles exist in the latest fork release with valid checksums, and the in-app update flow reads the marker's repo, so existing installs keep updating from their original source.

On the review comments: the arch and SDK-host points are already handled at head - the CPU kind is gated on is_x86_64/is_arm64 (unknown arch source-builds), and removing the tooling-only override is what keeps SDK-only hosts eligible for the CPU prebuilt. The ROCm-only resolve-prebuilt concern is not a regression: on main this host already routed to the fork via the AMD-tooling probe and resolved the same linux-x64-cpu bundle, so --resolve-prebuilt returns the identical result before and after. The two line-ending / set -u items do not apply here - .py and .sh are pinned to eol=lf in .gitattributes (no CRLF on checkout), and _setup_nvidia_usable is unconditionally initialized before use.

One real blocker, on the Windows CPU path. The fork's Windows x64 CPU llama-server installs and passes --version, but fails to start when serving a GGUF on a runner without Visual Studio. I merged this branch onto current main and re-ran CI on a staging replica: Windows Studio GGUF/API/UI all fail with Failed to load model: llama-server failed to start, while the Linux and macOS Studio jobs pass and Core passes (the Core failure showing on this PR is only because the branch is behind main). So the failure is specific to the fork's Windows CPU bundle, not the routing code - main is green here only because it still pulls the ggml-org Windows CPU bundle that this PR replaces.

Before this merges, the fork's Windows CPU bundle (the unsloth-prebuilt-cpu workflow in unslothai/llama.cpp) needs to produce a llama-server.exe that starts on a clean Windows host - most likely a missing bundled runtime dependency, since the ggml-org bundle it replaces starts fine. Once that build is fixed and the branch is rebased on main, this should go green.

[danielhanchen]

Follow-up: I tested the fork CPU bundles across the full GitHub-hosted CPU runner matrix (Linux and Windows x64 + arm64, macOS Intel + arm64), installing each via install_llama_prebuilt.py exactly as this PR routes, then loading a GGUF through llama-server and chatting.

Correcting my earlier note: the fork's Windows CPU llama-server is not broken. Launched directly it loads a 2B GGUF and answers correctly on every Windows runner (x64 and arm64), including with Studio's exact flags (-c 2048 --parallel 1 --flash-attn on --no-context-shift --fit on --jinja --spec-default). The binary and those flags are fine.

The Windows Studio load failure is a separate thing. llama-server hard-crashes with exit 0xC0000409 (STATUS_STACK_BUFFER_OVERRUN) before writing any log, but only when spawned by Studio, and only with the fork bundle (the ggml-org bundle Studio used before loads fine). I could not reproduce it by invoking llama-server directly with Studio's exact command line, so it is specific to how Studio launches/manages the process on Windows with this bundle (subprocess spawn, the concurrent embedder, or the environment), not the bundle contents or its flags. Pinning it down needs running Studio on Windows and tracing its llama-server spawn.

Separately, the matrix found a real portability bug in the Linux arm64 CPU bundle: it requires GLIBC_2.38 / GLIBCXX_3.4.32, so it fails to load on Ubuntu 22.04 arm64 (glibc 2.35) with a version-not-found loader error, while the x64 bundle runs fine there. The fork's arm64 CPU build should target an older base image to match the x64 one.

Everything else is green: Linux x64 (ubuntu-slim/latest/24.04/22.04/26.04), Linux arm64 24.04/26.04, and macOS all install the fork CPU/Metal bundle and answer correctly.

[oobabooga]

Thanks for running the full matrix, and for the routing simulation earlier. Both follow-up items check out when tested against the artifacts, with one correction on each.

The original Windows blocker: fixed at the source. Your staging failure traced to the packaging step in unslothai/llama.cpp picking the oldest VS redist libomp140 (the 14.29 toolset), which predates __kmpc_dispatch_deinit, so ggml-base.dll failed to load on any clean Windows host. Fixed in unslothai/llama.cpp#31, first shipped in b9899-mix-5dd3721 (2026-07-08 00:01 UTC). The workflow now also smokes llama-server --version so a bad pick fails the build instead of shipping.

This PR's CI re-ran after that release and is green, including the full Windows Studio GGUF CI run (https://github.com/unslothai/unsloth/actions/runs/28908666714) with "Studio install + inference without Visual Studio". Those jobs run the exact flow you describe: install via this PR's routing, assert the prebuilt landed, boot Studio, load a GGUF through /api/inference/load, chat. The branch is also merged with current main, so the stale Core failure you isolated is gone.

The Studio-spawn 0xC0000409: probably a pre-fix bundle. Which release_tag does the install marker in that Studio environment record? Every Windows CPU bundle before b9899 ships the broken libomp, and a pre-fix bundle dies at spawn before writing any log, which matches your symptom. A fresh direct-launch download would have resolved b9899 and worked, so the two tests may have exercised different builds.

If the marker says b9899 or later and the crash still reproduces, there is a real Studio-spawn interaction to trace and I'll dig into it. But the green job's log shows it installed app-b9899-mix-5dd3721-windows-x64-cpu.zip and then loaded and chatted through Studio on windows-latest (https://github.com/unslothai/unsloth/actions/runs/28908666714/job/85761170686), so I'd want the tag first.

The Linux arm64 glibc floor: real, but not a regression of this PR. I compared the fork bundle against ggml-org's llama-b9905-bin-ubuntu-arm64 that it replaces: identical floors down to the same library (libllama-server-impl.so needs GLIBC_2.38 / GLIBCXX_3.4.32 in both), because both recipes build arm64 on ubuntu-24.04-arm. Ubuntu 22.04 arm64 hosts hit the same loader failure before this PR, and in both worlds the installer's validation catches it and falls back to a source build.

Worth fixing in the fork's unsloth-prebuilt-cpu workflow as a follow-up: move the arm64 leg to ubuntu-22.04-arm, after validating its default gcc (the GCC-14 workaround exists because 24.04's stock toolchain miscompiles ggml). The same fix applies upstream, which has the identical floor.

[danielhanchen]

Final update on the Windows Studio question I flagged earlier.

I stood up full Unsloth Studio on a windows-latest runner with the current published fork CPU bundle (b9899-mix-5dd3721, app-b9899-mix-5dd3721-windows-x64-cpu.zip, sha256 verified) and loaded unsloth/Qwen3.5-2B-MTP-GGUF UD-Q4_K_XL through Studio's real /api/inference/load, then chatted with reasoning off. It works end to end:

• Studio spawned the fork llama-server.exe with its full production command: --flash-attn on --no-context-shift --fit on --jinja --spec-type ngram-mod ... --chat-template-kwargs {"enable_thinking": false} --mmproj ...
• llama-server ready then Loaded GGUF model via llama-server
• POST /v1/chat/completions "What is 1+1?" returned content: "2" (reasoning off, 2 completion tokens)

So the 0xC0000409 (STATUS_STACK_BUFFER_OVERRUN) I saw earlier does not reproduce on the current published fork bundle under Studio. It was on an older daily fork build; the daily rebuild resolved it. I could not have caught this from the binary alone: the crash was build-specific, not in the flags, the argv, or Studio's spawn/env/cwd.

Two things worth landing regardless of this PR (both outside its scope, both real):

1. Linux arm64 CPU bundle portability: the fork arm64 CPU build requires GLIBC_2.38 / GLIBCXX_3.4.32, so it fails to load on Ubuntu 22.04 arm64 (glibc 2.35) with a version-not-found loader error, while the x64 bundle runs fine there. Building the arm64 CPU bundle against an older base image (to match x64) would fix it.

2. Studio's xet->HTTP download fallback: on Windows, hf_transfer can hard-error (RuntimeError) rather than stall, so the stall-timeout fallback never triggers and the model load fails before llama-server is ever spawned. Catching that error and retrying over plain HTTP (or defaulting HF_HUB_ENABLE_HF_TRANSFER=0 on Windows, as Studio's own hf_download worker already does) would make loads robust.

Net: the routing this PR adds is correct and backward compatible, the CPU bundles exist and are hash-valid across the full GitHub-hosted runner matrix (Linux x64/arm64, Windows x64/arm64, macOS Intel/arm64), and the specific Windows Studio scenario now works on the current bundle.