[staging CI] unslothai/unsloth#7238 - #68
Closed
danielhanchen wants to merge 1003 commits into
Closed
danielhanchen wants to merge 1003 commits into
danielhanchen wants to merge 1003 commits into
Conversation
* fix: keep LoRA reloads working with PEFT 0.19 * test: exercise the PEFT tensor-parallel symbol extractor * test: prove the full PEFT tensor-parallel seam * fix: harden PEFT tensor-parallel shims * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: fall back when PEFT tensor-parallel source inspection fails --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* add models for /update endpoint * add logic for identifying out of date hf models * add endpoint for updating hf models * add relevant field to GgufVariantDetail * make exception handling better * add update_available flag for cached_models, and moved /update endpoint from inference -> models * hook up /update endpoint on the frontend * implement update scenarios for the model picker * fix bug where downloaded flag for an older revision was being wrongly set to false * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix import and make hf calls async * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove has_vision from UpdateRequest * fix ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * clear cancel event before updating gguf variant * set _cancel_event back if it was set initially * add hf_token to get_paths_info * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio: harden model update endpoint and update checks - update_hf_model: pass snapshot_download local_dir (local_path is not a valid kwarg and 500s when updating bicodec audio models) - get_gguf_variants: wrap the remote update check so a network, rate-limit, gated, or offline failure degrades to "no update info" instead of failing the whole variant listing, matching list_cached_models - add regression tests for both paths * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: HF model update detection and Update action for cached models Surface an "Update available" cue and a managed Update action for cached on-device models. /api/hub/update-status compares each cached main GGUF file's local blobs against the remote main revision using set membership across all cached revisions, so a repo that was already updated (and still holds the old snapshot alongside the new one) is not falsely flagged. The Update action re-downloads through the download manager so it shows in the Downloads panel with progress and cancel. The frontend wires the Update button into the GGUF, on-device, and model-selector cards and keeps the quant label fully visible when the action buttons crowd the row. Adds regression tests for the multi-revision update check. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: accept force_download kwarg in hf_xet_fallback test double The download seam now passes force_download to the attempt callable; the _FakeAttempt mock did not accept it, failing 6 tests with TypeError. Add the keyword (default False) so the scripted-results double matches the seam. * Fix Studio model update regressions * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address Studio update review feedback * Address Studio update edge cases * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Share GGUF update status helper * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix GGUF update detection and cache cleanup * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix cached GGUF update badges --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: shimmyshimmer <107991372+shimmyshimmer@users.noreply.github.com> Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
…drift (#6770) check_js_file put the bundle's KB size inside the finding's check label, which is part of the baseline match key (package, file, check). When tensorboard's projector_binary.js grew from 1918 KB to 1933 KB, the reviewed baseline entry stopped matching and the benign HIGH resurfaced, red-failing the studio and extras scan-packages shards. Move the size into the evidence field (shown for review, not matched) and keep the check label constant, then update the one tensorboard baseline entry to the size-agnostic label. The finding is suppressed again and will not re-break when the bundle grows by a few KB. Scanner self-tests pass unchanged. Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Condense the verbose comments and docstrings added by the recent chat template, GPT-OSS detection, PEFT tensor-parallel, and Studio inference proxy fixes. Comments and whitespace only; no code changes.
…t (#6777)
* MLX CI: find llama-cli where save_pretrained_gguf actually installs it
The GGUF reload step hardcoded the CWD-relative paths llama.cpp/llama-cli and
llama.cpp/build/bin/llama-cli, but save_pretrained_gguf builds and installs llama.cpp
under unsloth_zoo's LLAMA_CPP_DEFAULT_DIR ($UNSLOTH_LLAMA_CPP_PATH, else
~/.unsloth/llama.cpp), so the reload could not find the binary and failed the Mac M1
job with "llama-cli not found". _find_llama_cli now searches that install directory
(and honors the env override) before falling back to the old CWD layout, with a
recursive glob as a last resort. The search is a strict superset of the previous
paths, so it cannot regress a layout that already worked.
* MLX CI: return an absolute llama-cli path from the locator
Resolve the located binary to an absolute path. If UNSLOTH_LLAMA_CPP_PATH is a
relative directory (e.g. "."), Path(".") / "llama-cli" normalizes to the bare name
"llama-cli", and subprocess.run treats a separator-less argument as a PATH lookup
rather than a file to execute, raising FileNotFoundError. resolve() makes the returned
path absolute so it always runs the intended binary.
* MLX CI: give llama-cli EOF on stdin so GGUF reload cannot hang
With the binary now found, the GGUF reload actually invokes llama-cli and it timed
out after 300s generating 24 tokens on a 270m model, which is a stdin block rather
than slow generation: subprocess.run captured stdout/stderr but left stdin inherited,
so -no-cnv still left llama-cli waiting for interactive input. Pass
stdin=subprocess.DEVNULL so it receives an immediate EOF and runs the single prompt to
completion.
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
…(#6775) * Studio RAG: disable trust_env on loopback llama-server httpx clients The RAG embedder health probe (embed_llama_server.py), its pooled httpx.Client, and the vision captioner (captioner.py) call the local 127.0.0.1 llama-server with httpx's default trust_env=True, so an ambient HTTP(S)_PROXY that returns 503 for loopback breaks embedder startup and captioning. Set trust_env=False on these loopback clients, matching the existing fix on the main llama_cpp and inference clients. External provider calls are untouched. Follow-up to the loopback trust_env fix; covers the remaining local llama-server clients in the RAG path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio RAG tests: accept trust_env kwarg in captioner httpx.post mocks The loopback captioner now passes trust_env=False; update the _vision_complete fake_post stubs to accept it and assert it is False. * Trim comments in Studio RAG trust_env fix (comment-only) * RAG trust_env test: explicit UTF-8 read + scan all package .py files Addresses review: utf-8 open avoids a Windows decode error, and scanning every .py in core/rag catches any future file that adds an httpx call. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
…ined files are not auto-suppressed (#6552) * scan_packages: key baseline on matched-code hash The baseline matched on (package, package-relative file, check), which excluded the matched code, so a future finding of the same check in the same file was suppressed regardless of what the code did. A malicious future version of an already-baselined package could place a payload in the same file under the same check and pass the enforcing gate. Key the baseline on a hash of the matched code too. The hash is over the deduped, sorted set of matched spans with L<NN>: line markers stripped, so version bumps, line shifts and match reordering stay stable while new or changed flagged code reopens the finding. Version is left out of the key so routine dependency bumps do not reopen every entry. The hash is capped and recomputable from the stored evidence. Regenerate scan_packages_baseline.json against the current dependency set; the hf-stack, studio and extras scan shards pass enforcing (no active CRITICAL or HIGH). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan_packages: refresh baseline for newer unsloth-zoo release A newer unsloth-zoo published after the first regenerate added tests/test_mlx_save_export_regressions.py, a benign test fixture (temporary_location="/tmp/ignored") that trips the /tmp dropper check. Regenerate the hf-stack shard against the current set so the entry is allowlisted; studio and extras are unchanged. * scan_packages: harden baseline loading against malformed JSON Guard against a non-dict top-level baseline and non-dict entries so a corrupt or hand-edited allowlist warns and fails closed instead of crashing with AttributeError, and treat an explicit evidence: null as empty. * scan_packages: hash the full match set, keep indentation, strip only the marker Address the evidence-hash review feedback: - Capture every matching line, not the first three, so a payload appended after existing matches in a baselined file and check reopens the finding instead of riding the sample. - Preserve leading indentation so a flagged line moved out of a guarded block reads as changed. - Strip only each span's prefix up to the first L<NN>: marker, so an L<NN>: inside the matched code is kept and a change to it reopens the finding. Evidence and its hash are stored in full and stay recomputable from the stored field. Regenerate the baseline; hf-stack, studio and extras pass enforcing with no active CRITICAL or HIGH. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan_packages: bind baseline evidence to full matched code Address review feedback on the evidence-hash baseline key: - Split evidence only on real span delimiters (" | " before an L<NN>: marker, or a newline), so a bitwise-or or union type in matched code is no longer split apart into separate spans. - Record matched lines in full (drop the 160-char per-line cap) and record every distinct multiline match, so code appended past the cap or a second cross-line match reopens the finding instead of riding the first one. - Give the large-JS-bundle and .pth base64-blob findings a content digest instead of empty or prefix-only evidence, and record all .pth import lines, so a changed bundle, blob or import no longer inherits a baselined empty or truncated key. - Warn when a loaded baseline has entries without evidence_hash so a legacy baseline is regenerated rather than silently degraded. Regenerate scripts/scan_packages_baseline.json against the current dep set and add regression tests for each case. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan_packages: harden multiline and duplicate evidence handling Follow-up hardening so the evidence hash tracks the full matched code: - For DOTALL patterns that match across lines, record every line the match spans (not just the start line), so a change on a continuation line (the URL inside a baselined C2 loop, a swapped credential path) reopens the finding. A pathological greedy span is bounded to its head line plus a digest of the rest. - Keep duplicate spans in the canonical evidence so a second identical matched line in a new code path changes the key instead of deduping away. - Anchor the evidence prefix to strip only a genuine leading label or line-number marker, leaving a marker-like "L<NN>:" inside raw .pth code intact. - Make the legacy-baseline warning explicit that entries without an evidence_hash reopen rather than suppress under a coarse key. Regenerate scripts/scan_packages_baseline.json (same finding set; entries for same-file repeated checks are now tracked separately) and add tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan_packages: bind every combo and large finding to its full content Close the remaining asymmetric-evidence gaps so a changed payload cannot ride a reviewed baseline entry: - Digest a capped multiline span from the code without line markers, so a pure line shift stays stable while a continuation-line change reopens. - Give the "Unusually large executable .pth" finding a content digest instead of keying on byte size and import-line count alone. - Record both contributing signals for the JS credential+network stealer, the shell credential+network and persistence-hook combos, and the hidden network+exec docstring payload, so changing the network/exec side reopens. - Allow punctuation in an evidence label prefix so a "network+exec:" label is stripped and line shifts do not change the key. Regenerate scripts/scan_packages_baseline.json and add tests for each case. * scan_packages: bind remaining Python combos; key npm baseline on evidence Python scanner: the openssl+key, anti-analysis, DNS-exfil and base64+exec+blob combos recorded only one contributing signal, so a changed payload on the other side could ride a reviewed baseline entry. Each now binds every co-occurring signal (and the blob is digested, since it can sit on a separate line from the decode call). npm scanner: scan_npm_packages.py keyed its allowlist on (package, path, pattern) only, the same coarse-key bypass the Python scanner just closed. Add an evidence hash to the key (schema v3, fail-closed on older baselines) and store full evidence. The committed baseline stays empty by design. Regenerate scripts/scan_packages_baseline.json and add tests for each case. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan_npm_packages: bind full blob evidence and harden baseline loader Follow-up on the npm evidence-hash key: - _evidence now records every match and, when a snippet is truncated for display, appends a digest of the full match. The obfuscated-blob key was hashing only the truncated first-match snippet, so a changed payload tail or an appended blob in the same package/file/pattern could ride a reviewed entry. - _load_baseline guards that the root is an object, entries is a list, and each entry is a dict before reading it, so a malformed baseline warns and fails closed instead of raising AttributeError. Add tests for a changed blob tail reopening the key and for malformed entries. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan_packages: symmetric baseline-loader guards; bind npm outbound host context - Python _load_baseline now rejects a non-list "entries" with a warning instead of raising TypeError, matching the npm loader. - npm cred-surface-host (outbound) records the host with its URL path / fetch call / host config, so a changed outbound path, headers or body reopens the key rather than riding the bare host literal. Add tests for both. * scan_npm_packages: migrate v2 baselines and bind host-config outbound context - _load_baseline now migrates schema v2 entries by recomputing the evidence hash from stored evidence (with a legacy warning), matching the Python loader, instead of discarding them; only pre-v2 basename schemas are rejected. - The cred-surface-host (outbound) host-config branch now captures the whole line (path, headers, body), so a changed outbound payload on the same hostname line reopens the key instead of riding the bare host snippet. Add tests for v2 migration and the host-config context binding. * scan packages: bind PEM key bodies and npm windowed evidence to baseline keys scan_packages: embedded-key findings now pin the full PEM block (BEGIN..END) via a content digest, so a key body swapped under the same marker reopens the finding instead of riding the unchanged BEGIN line. Single-line and DER keys were already bound by their full matched line; marker-only references with no END block (validation header lists) are unaffected, so the committed baseline is unchanged. scan_npm_packages: _evidence now digests the full containing line whenever the shown snippet is only a window into it (short match on a long line, or a truncated payload), so a changed payload tail outside the display window reopens the key. The npm baseline is empty, so this changes no suppressions. Adds regression tests for both cases. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan packages: bind multi-line evidence and every blob to baseline keys _extract_evidence now extends each single-line match over its bracket continuations, so a multi-line call binds its argument lines and a changed URL or body on a continuation line reopens the key. After the per-line pass it also records cross-line matches the scan cannot otherwise see (a DOTALL regex, or a multi-line construct appended under a check that already had a one-line match), so an appended multiline payload reopens instead of riding the key. _blob_digest hashes every large base64 blob (not just the first) for the base64+exec finding and the .pth large-blob finding, so an appended or swapped second encoded payload reopens; single-blob files keep the same digest. scan_npm_packages _evidence digests the full logical line (the matched line plus its bracket-continuation lines), so a multi-line fetch's option and header lines bind and a changed payload on a following line reopens the outbound key. Regenerated the Python baseline: same package/file/check set, 24 entries pick up the wider multi-line evidence. Adds regression tests for each case. * scan packages: stop giant greedy spans from binding a whole-file digest When a greedy DOTALL pattern (reverse shell socket...subprocess, C2 loop) has its anchor tokens far apart, the match span covers the whole file. Digesting that span bound thousands of unrelated lines, so the evidence hash drifted on any edit between the anchors (a dependency bump reshuffling the file), which made a baselined finding reopen on an upstream release. The multiline pass now skips an oversized span when the per-line pass already bound the signal lines, so the evidence is the stable matched lines; a genuinely appended multi-line construct stays under the cap and is still recorded. Regenerated the Python baseline against Python 3.12 (the version the scan CI shards run) so the resolved dependency set matches CI. Same package/file/check set. Adds a regression test. * scan packages: tighten evidence binding (order, string brackets, span size) Address review follow-ups on the evidence extraction: - _canon_evidence keeps discovery (line) order instead of sorting. Line-shift stability already comes from stripping the L<NN>: markers, so order stays significant and reordering matched lines (a multi-line call's arguments) reopens the finding. - _logical_line_end (Python) and _logical_line_text (npm) blank string literals before counting brackets, so a ) inside a string argument does not close the logical line early and drop later argument lines. - The oversized-span skip now only drops a giant whole-file bridge (over 60 lines); a genuinely appended multi-line construct is recorded so its payload reopens, rather than riding an existing one-line match. - npm _logical_line_text binds the enclosing bracket group, so a host-config object whose { is on a prior line binds its path/headers/body lines. Regenerated the Python baseline (Python 3.12, matching the scan CI shards): same package/file/check set. Adds regression tests for each. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan npm packages: normalize and bound the logical-line digest - _evidence whitespace-normalizes the logical line before digesting (matching _evidence_hash), so a formatter-only reindent of the bound continuation lines does not change the sha256 suffix and reopen an unchanged finding. - _logical_line_text follows a bracket group to its close up to a hard 200-line cap (digest input only), so a config object longer than the backward window still binds its whole tail instead of silently truncating. Adds regression tests. npm baseline is empty, so no regeneration is needed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan: cap single-line evidence and widen npm opener window Cap each rendered evidence line at 200 chars in scan_packages.py: a long or minified one-line file is shown as a bounded prefix plus a sha256 of the full line, so a packed payload cannot dump unbounded content into the CI logs or baseline while a change past the cutoff still changes the digest and reopens the finding. Mirrors how the npm scanner bounds its snippets. Widen the npm backward opener window (_MAX_CONT_LINES 12 to 200, symmetric with the forward cap) so a host deep inside a large options object binds the whole object, not just its own line; a changed path, header, or body on any property reopens. Regenerate the Python baseline with Python 3.12: only the protobuf nspkg.pth and unsloth-zoo compiler.py evidence change, both from the new line cap; the package/file/check key set is unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan: bind all host contexts, deep call continuations, far-back npm openers Three fail-closed evidence gaps surfaced by review of the previous round. scan_npm_packages.py: measure the forward bracket-group cap from the matched line (idx + _MAX_GROUP_LINES) instead of the opener, so an opener found near the widened backward limit no longer consumes the forward budget and drops the path, headers, or body that follow the host. scan_npm_packages.py: _outbound_host_evidence now records every outbound context form for a host (URL, fetch-context, host-config), claiming each non-overlapping match in form order, so a separate host-config request added beside an already-baselined URL changes the evidence and reopens the key. The common single-context case keeps its existing snippet. scan_packages.py: follow a matched Python call over its continuations up to a separate _MAX_CALL_LINES (40), decoupled from the 12-line display threshold, so a multi-line requests.post( binds its whole argument list in the digest and a changed body deep in the call reopens; bounded so a miscounted bracket cannot swallow unrelated code. No baseline change: the current dependency set has no matched call that closes between 13 and 40 lines, confirmed by a Python 3.12 regenerate that produced a byte-identical baseline. * scan: clamp npm depth, pin large bundles, follow backslash and bound .pth dump Four fail-closed evidence gaps surfaced by review of the previous round. scan_npm_packages.py: clamp the backward opener scan at depth 0 so a leading unmatched closer (a preceding block whose opener is outside the backward window) no longer drives depth negative and masks the real enclosing opener that follows; a host-config object after such a block now binds and a changed path reopens. scan_packages.py: a large JS bundle now pins its whole content even when another JS heuristic already fired. The bundle digest was only added when no other finding existed; it is now appended to every finding's evidence on a large bundle, so an unchanged obfuscation signature no longer lets changed payload elsewhere ride the matched-line key. scan_packages.py: _logical_line_end follows explicit backslash line continuations, so a call split with a backslash before its parenthesis binds the continuation line (URL/body) instead of returning at the zero-depth API line. scan_packages.py: the catch-all .pth import evidence is bounded through _cap_line (prefix plus a digest of every line) so a large .pth of benign imports cannot dump the whole member into the logs or baseline while an appended or swapped import still reopens. Baseline regenerated with Python 3.12: key set unchanged; one entry (unsloth-zoo compiler.py) gains the backslash-continued banner lines now bound by the continuation fix. * scan: handle multi-line strings, lifecycle bodies, and de-quadratic evidence Addresses a review round plus a performance audit of the evidence extractor. Correctness (fail-closed): - Bind the UNION of the single-line-blanked and multi-line-blanked bracket spans in both scanners. The multi-line view blanks a triple-quoted Python string or a backtick template literal that spans lines, so a `)` inside such a string no longer closes the enclosing call early and drop later arguments. The single-line view still counts a payload embedded INSIDE a string, so a dropper that hides a call in a string keeps its argument lines bound. Taking the larger span never shrinks the binding below either view, avoiding a fail-open regression. - cred-env-in-lifecycle now pins the whole lifecycle script body via a digest, so a changed non-token line (e.g. adding a curl exfil beside the token reference) reopens, not just a change on the token line. Performance / DoS (the scanner runs on attacker-controlled package files up to the 64 MiB / 16 MiB member caps, with no per-file time budget): - _extract_evidence precomputes newline offsets once and maps match offsets with bisect, removing the O(matches) whole-file content.count per match that made the finditer fallback quadratic (a crafted minified file went from ~13 s/MiB and hours at the cap to linear). - npm _index_text splits and string-blanks the file once per evidence call instead of per match (was O(matches x file) time and allocation). - Bound evidence output: _MAX_EVIDENCE_SPANS (Python) and _MAX_EVIDENCE_MATCHES (npm) fold the remainder into a digest so a file with thousands of matches cannot build a multi-megabyte evidence/baseline blob while an added/removed match past the cap still changes the key. - _outbound_host_evidence caps matches per form and bounds the overlap claim so a host repeated many times cannot make it quadratic. No baseline change: a Python 3.12 regenerate is byte-identical (the union equals the legacy single-line span for every current dependency file; the cap thresholds sit above the largest real entry), so these are forward-looking hardening with no drift. * scan: count all overflow matches, bind their context, blank JS regex literals Follow-ups on the evidence output caps from the previous commit. - _outbound_host_evidence no longer truncates each pattern's match iterator with islice; it iterates every match and runs the overlap dedup only while the display list is below the cap (so claimed stays bounded and the check is O(cap) per match, not quadratic), folding every match past the cap into the overflow digest. A host context beyond the 64th is counted again, so it reopens. - The overflow digest (both scanners, via a shared _overflow_digest) binds each overflow match's logical-line context, not just the regex match text, so a changed payload on an over-cap line reopens even with the matched token unchanged. - The multi-line JS blanked view now blanks regex-literal bodies (tracking the previous significant char for regex-vs-division and char classes for a literal `/` inside `[...]`), so a `)` inside `/)/` no longer closes an outbound call early. The bound span is the union of the single-line and multi-line views, so an imperfect regex decision only ever grows the span, never shrinks it. - The Python overflow digest canonicalizes spans (strips L<NN>: markers via _canon_evidence) before hashing, restoring line-shift stability for the over-cap region. No baseline change: the overflow branches only trigger above the per-finding caps (above the largest real entry), and the npm baseline is empty, so a Python 3.12 regenerate is byte-identical. * scan: refresh baseline for ipython interactiveshell.py span drift A newer ipython release changed the filesystem-enumeration span in IPython/core/interactiveshell.py, so its content digest no longer matched the baselined evidence and the studio scan shard flagged it as a non-baselined CRITICAL. Regenerated with Python 3.12: only the ipython entry's evidence_hash changes; the package/file/check key set is unchanged, and a studio enforcing spot-check exits 0. * Bound scanner evidence memory: stream overflow spans and cap lifecycle baseline size scan_packages.py: _extract_evidence no longer materializes a rendered span per match before slicing at the display cap. Once out holds _MAX_EVIDENCE_SPANS spans, further spans fold straight into a running digest, so a minified or padded file with hundreds of thousands of matching lines keeps memory bounded to the display cap instead of the match count. The fold reproduces _canon_evidence(" | ".join(overflow)) byte for byte, so the overflow digest and every baseline key are unchanged. scan_npm_packages.py: lifecycle-fetch-exec and cred-path-in-lifecycle stored the entire install script body as evidence, so --write-baseline on a package with a multi-MiB lifecycle script bloated the baseline JSON. Both now store a bounded matched snippet plus a body-sha256 digest, matching cred-env-in-lifecycle. The digest still binds the whole body, so a change to any line reopens the finding. Adds tests for the streamed overflow bound and the bounded-but-reopens lifecycle evidence. Baseline unchanged (byte-identical Python evidence; npm baseline empty). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make npm bracket-group scan order-aware so a same-line close-then-open binds _scan_group counted brackets with a per-line net (opens minus closes), which collapses intra-line order: a line that closes a prior block and then opens the host-config object on the same line, e.g. `}); const opts = {`, nets to <= 0, so the trailing `{` was dropped and the group started at the hostname line. A changed path/headers on the following lines then hashed to the same evidence and could ride an existing baseline key. Replace the net count with an order-aware (L, R) reduction per line (L closers needing an opener to the left, R openers needing a closer to the right) and apply it in order in both the backward and forward scans, clamping stray closers at 0. The trailing opener now stays visible so the whole object binds and a changed payload reopens. Per-line cost is unchanged (one C-level bracket findall), so the existing outbound-host evidence is byte-identical on all prior shapes; only the previously-dropped same-line case changes. Adds a regression test for it. * Harden scanner evidence: bound memory and bind Python call tails fail-closed Five fixes across both scanners, none of which change the committed baseline (a full regen of all three pip shards produced a byte-identical 185-key set). scan_npm_packages.py: _evidence and _outbound_host_evidence collected every regex match into a list before applying the 64-match display cap, so a text file under the size cap that repeats a cheap signal (such as NPM_TOKEN) millions of times could allocate a huge list of re.Match objects and stall or OOM before the overflow digest ran. They now stream from finditer and fold overflow as matches arrive via a shared _fold_overflow_match helper, byte-identical to the prior digest. scan_packages.py: - _extract_evidence kept inserting every unique over-cap span into the seen set even after it stopped appending to the display list, so a generated file with millions of one-line matches still grew that set unbounded. It now tracks spans only while filling the display list (per-line spans are unique by line number, so dropping them past the cap cannot miss a dedup). - _scan_line_end counted brackets with a per-line net, so a continued statement that closes on the same line it opens a flagged call (a leading "]" before "requests.post(") had the call's open paren cancelled and bound only the opener line. It now applies brackets in order via _bracket_lr (leading closers clamp at 0), matching the npm bracket fix. - a single-quoted string continued by a trailing backslash was not tracked across lines, so a close paren inside the continued string on the next line closed the call early; _blank_code_strings now carries the continuation. - a call with more argument lines than the soft cap was hashed only through the cap, so a changed data=/headers tail past it stayed suppressed; a closing call is now followed to its real close under a 200-line hard limit (a never-closing opener still stops at the 40-line soft cap so it cannot swallow the file). Adds regression tests for each. npm baseline is empty; the Python baseline is unchanged (verified byte-identical by regenerating all three shards). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Bind giant DOTALL span anchors and add context to constant IOC evidence Two fail-closed gaps where a changed payload could keep the same evidence hash and stay suppressed by the baseline. scan_packages.py: a giant greedy DOTALL span (a cross-line IOC match bridging more than 60 lines, e.g. RE_TEMP_EXEC matching a /tmp line and a much-later subprocess line) was dropped entirely once the per-line pass had any match, so an appended cross-line payload -- a new /tmp line plus a later subprocess line that share no single line, so the per-line pass never binds them -- produced the same evidence and rode the key. The span is no longer dropped: it is bound by its head and tail anchor lines plus a digest over just those (no line numbers, so a pure line shift is stable). An added or moved anchor reopens the finding, while churn in the bridged interior stays stable, so this does not reintroduce whole-file drift. Two baseline entries (multiprocess test, unsloth-zoo scanner file) carry such a span and are refreshed; a full three-shard regen confirmed only those two keys change. scan_npm_packages.py: known-ioc-string and cred-surface-host (always-bad) recorded only the bare needle/host as evidence, so a reviewed tarball that kept the IOC string while altering the adjacent fetch/exfil body produced an identical key. They now bind matched-line context: known-ioc-string via the matched line and its bracket-group continuation, cred-surface-host (always-bad) via the outbound call context (path/headers/body, falling back to the bare host when not in an outbound call). A changed payload on the same call now reopens. Adds regression tests for each. npm baseline is empty; the Python baseline updates only the two giant-span entries. * Hash giant-span interiors, bind exec/eval trigger, JS content, intra-literal whitespace Four fail-closed gaps where a changed payload could keep the same evidence hash. scan_packages.py: - A giant bridged DOTALL span was bound only by its head and tail anchors, so a cross-line payload inserted into the bridged interior between unchanged outer anchors kept the same key. The whole span content is now digested (via _render), so any interior change reopens; a pure line shift stays stable because the digest is over the markerless code. Two baseline entries (multiprocess test, unsloth-zoo scanner file) carry such a span; with full-interior binding, multiprocess resolved at two versions across shards now yields two distinct entries where the anchor digest had collapsed them into one. - The exec/eval-with-hidden-payload findings omitted the visible exec/eval line that makes the hidden string executable, so flipping a harmless eval("1+1") to exec(__doc__) kept the same key while arming the payload. The trigger line from the real-code view is now bound into the evidence. - check_js_file extracted evidence with the Python-string-aware extractor, which does not blank JS backtick template literals, so a template containing a close paren closed a call's bracket span early and omitted later option/body lines. The full file content digest is now pinned to every JS finding (not just large bundles), binding the whole call. scan_npm_packages.py: the evidence canon collapsed all whitespace via split(), erasing whitespace inside JS string literals along with harmless indentation, so a changed request body 'a b' -> 'a b' kept the same key. A new _canon_preserve_strings collapses whitespace only OUTSIDE string literals (reindent-stable) while preserving it INSIDE single/double/backtick literals (intra-payload edits reopen). Used for the evidence hash and the logical-line digests. Adds regression tests for each. npm baseline is empty; the Python baseline updates the two giant-span entries and adds the second multiprocess version's entry. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Pin llm-compressor auto-install to a vetted version range
install_llm_compressor() auto-installs llm-compressor on first use of an FP8/FP4
compressed export when it is not already present. The install command used the bare
package name, so pip resolved to whatever the configured index served; a compromised,
dependency-confused, or inflated-version ("999.0.0") release could then run under the
Unsloth process at install and import time.
Bound the automatic install to a vetted range
(_LLM_COMPRESSOR_SPEC = "llmcompressor>=0.8.0,<0.13"), which the oneshot /
QuantizationModifier API this uses supports, so pip can no longer jump to an arbitrary
future or inflated version. An already-installed newer llm-compressor is still used
as-is (the import short-circuits), so this only constrains the auto-install, never a
user's own install.
Add UNSLOTH_DISABLE_LLM_COMPRESSOR_AUTOINSTALL=1 to forbid the automatic install
entirely and require a manual, vetted install, for locked-down or air-gapped
environments. Update the manual-install hints to the pinned spec.
Add tests/saving/test_llm_compressor_install_pin.py: static (ast) guards that the spec
stays a bounded pin, that the install command never passes an unpinned llmcompressor
literal, and that the opt-out env gate is evaluated before any install runs.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Loosen llm-compressor auto-install ceiling to <1.0 so new models still export
The earlier <0.13 ceiling was too tight: brand-new architectures (for example
Qwen3_5ForConditionalGeneration / qwen3_5, gemma-4 MoE) can require a newer
llm-compressor, and _unsloth_save_compressed_tensors already fails with
"requires a newer llm-compressor" when a scheme is unavailable. Capping the
auto-install at 0.12 would block getting that newer release and break compressed
export for new models.
Widen to llmcompressor>=0.8.0,<1.0. pip still auto-installs the latest 0.x
(where new-architecture support lands), while the <1.0 ceiling continues to block
a jump to an inflated-version ("999.0.0") or 1.0+ dependency-confusion release.
An already-installed newer llm-compressor is still used as-is (the import
short-circuits), and UNSLOTH_DISABLE_LLM_COMPRESSOR_AUTOINSTALL still forbids the
automatic install entirely for locked-down environments.
* Lower llm-compressor floor to 0.6.0 so supported old torch still resolves
The >=0.8.0 floor conflicts with the torch this install pins in its constraints
file. Unsloth supports torch>=2.4, but llm-compressor 0.7.0+ require torch>=2.7
(0.10+ need >=2.9, 0.12+ need >=2.10). On a supported torch 2.4-2.6 box pip then
has no candidate in [0.8.0, 1.0) and FP8/FP4 export fails before quantization.
Lower the floor to 0.6.0 (its metadata only needs torch>=1.7), which never
conflicts with any supported torch. pip still prefers the newest compatible
release, so modern torch continues to get the latest 0.x (0.12.0). The <1.0
ceiling that blocks an inflated-version supply-chain jump is unchanged.
Add a regression test asserting the floor stays <= 0.6.0.
* Cap llm-compressor auto-install ceiling to a vetted minor (<0.13)
A bare <1.0 ceiling still admits any 0.x, so an inflated "0.999.0" served by a
compromised or misconfigured index would win pip's highest-version selection --
the same dependency-confusion this pin is meant to block. Cap the ceiling to the
current vetted minor (<0.13) so that jump is blocked; bump it deliberately, after
vetting, when a newer llm-compressor is needed (e.g. for a brand-new architecture
scheme). Current new models are unaffected: 0.12.0 is < 0.13 and supports them.
The 0.6.0 floor (torch>=1.7 compatible) is unchanged, so resolution still works
across Unsloth's whole supported torch range (2.4 -> 0.6.0 ... 2.12 -> 0.12.0).
Add a regression test asserting the ceiling admits the current vetted release but
blocks an inflated 0.x and the next major.
* Trim comments in the llm-compressor pin (comment-only, no code change)
* Cap llm-compressor auto-install to the exact vetted patch (<=0.12.0)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Extracted and narrowed from unslothai/unsloth#6543 by @TheJagStudio. This keeps the startup/banner and text file encoding hardening separate from the already-merged Python code-exec UTF-8 fix in #6548. Co-authored-by: Jagrat Patel <81472856+TheJagStudio@users.noreply.github.com>
* Studio: opt-in OpenAI /v1 model auto-switch and idle keep-warm The OpenAI-compatible endpoints serve whichever GGUF is loaded and ignore the request model field, so an OpenAI client that changes model never reloads. Add an opt-in setting that, when a /v1 request names a downloaded local GGUF different from the loaded one, loads it before serving by reusing the existing /load path (its dedup, tensor fallback, and threading apply). Unknown names still serve the loaded model, so drop-in compatibility is preserved and no remote download is triggered. Also add an optional idle auto-unload (TTL keep-warm): a pure-ASGI middleware tracks in-flight inference requests so a stream is never unloaded mid-response, and a lifespan loop unloads the model after the configured idle seconds. Both settings default off and live in the app_settings store, exposed via GET/PUT /api/settings/openai-auto-switch. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: variant-aware auto-switch, /v1/responses coverage, keep-warm load stamp Follow-ups from review of the opt-in OpenAI auto-switch path: 1. Variant-aware dedup. _maybe_auto_switch_model compared only the repo id, so requesting another quant of the loaded repo (e.g. Q4_K_M loaded, Q8_0 asked) was served by the old quant. Compare hf_variant too, matching /load dedup. 2. Streaming /v1/responses now calls the auto-switch hook. It went straight into _responses_stream and only checked is_loaded, so stream=True could serve the old model or 400. Non-streaming already routed through chat completions; the hook is idempotent once loaded. 3. resolve_local_gguf tries an exact id match before splitting a trailing :VARIANT, so local ids that contain a colon (e.g. a Windows path) resolve instead of being cut at the drive letter. 4. Idle keep-warm stamps activity on a load/swap transition. _last_active was only refreshed by inference requests, so a model loaded after the server sat idle past the TTL could be unloaded before its first request. Tests cover each case. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make the /v1/responses auto-switch test order-independent The new streaming-responses test passed in isolation but failed under the CI's randomized collection order with "object has no attribute 'state'": it passed a bare object() as the request and stubbed only one dispatcher, so an ordering where the real dispatcher ran hit request.state. Give the request a state and stub both dispatchers; the test still asserts the hook fires before dispatch. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: assert /v1/responses auto-switch wiring on source, not at runtime The behavioral version executed openai_responses and relied on stubbing its callees, which a randomized collection order in CI could defeat (the real dispatcher ran and hit request attributes). Assert on the function source that the hook precedes both dispatchers instead; the hook's runtime behavior is already covered by the direct _maybe_auto_switch_model tests. * Studio: auto-switch on /v1/embeddings, GGUF-only targets, idle-unload race gate Second-pass review follow-ups on the opt-in auto-switch path: 1. /v1/embeddings now calls the auto-switch hook before the loaded-state check, matching the other model-bearing OpenAI endpoints (the keep-warm middleware already treats embeddings as inference). 2. The resolver index is now GGUF-only. The local-model scanners also surface Transformers/safetensors repos; without a filter, auto-switch could unload the GGUF and route a request into the non-GGUF loader. _has_local_gguf checks a direct .gguf, a models-dir folder, and the HF-cache snapshots layout. 3. Idle keep-warm now holds an asyncio gate across the idle check and the unload, and a request bumps inflight under the same gate, so the loop can no longer unload in the window between "looks idle" and the kill. Tests cover each. Broader local-model source parity (LM Studio, Ollama, legacy caches, custom scan folders) is a follow-up; missing one of those today just falls through to the loaded model. * Studio: variant-aware local resolver, count_tokens + audio auto-switch coverage Third-pass review follow-ups on the opt-in auto-switch path: 1. The resolver is now variant-aware via list_local_gguf_variants. It indexes only the quants actually on disk, recursing snapshots and quant subdirs such as the nested per-quant folders, so a requested repo:VARIANT resolves only when that quant is local and a bare repo resolves to a concrete local quant. This fixes two gaps: the previous shallow glob rejected nested-variant GGUF repos, and a request for an uncached quant could send /load down the remote download path, breaking the local-only contract. 2. /v1/messages/count_tokens now auto-switches like its sibling /v1/messages, so a count uses the requested model's tokenizer. 3. /api/inference/audio/generate (direct GGUF TTS) is now tracked as in-flight inference, so the idle loop cannot unload the model mid-generation. Tests cover each. Two reviewer items are left as follow-ups: indexing the remaining local sources (LM Studio, Ollama, legacy/default caches, custom scan folders), which fails safe today by falling through to the loaded model; and fully serializing concurrent different-model requests, an inherent limit of the single-slot llama backend that the opt-in feature is not designed around. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make local GGUF resolver fail-safe so a bad model name cannot 500 The auto-switch hook calls resolve_local_gguf without its own guard, and /v1/completions and /v1/embeddings pass body.get("model") through unchanged. A non-string model (e.g. {"model": 123}) or any internal scan failure would then raise out of the resolver and turn a request that would otherwise be served by the loaded model into a 500, breaking the drop-in compatibility the feature is built on. Guard the resolver at its boundary: reject non-string input up front and wrap the lookup so any failure returns None (fall through to the loaded model). Add regression tests for both paths. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: per-model launch flags for auto-switched GGUF models * Studio: list switch-eligible GGUFs in /v1/models when auto-switch is on * Studio: settings UI for OpenAI model auto-switch and idle auto-unload * Studio: show save error over the disabled-idle hint in auto-switch settings * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: address gemini review (case-insensitive /v1/models retrieve, idle-input empty guard) * Studio: address codex review (deterministic override args, exclude probe/embedding models from discovery) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep-warm count_tokens, gate idle on auto-switch, drop hidden models Three hardening fixes to the opt-in auto-switch path surfaced while reviewing the work that builds on it: 1. count_tokens keep-warm. /v1/messages/count_tokens counts via the loaded tokenizer and already auto-switches, but the keep-warm middleware did not track it, so idle auto-unload could free the model mid-count. It is now a tracked in-flight path. 2. "Off means unchanged" for idle unload. get_auto_unload_idle_seconds now reports 0 while auto-switch is disabled. Idle unload only makes sense with auto-switch on (an unloaded model returns only via the next request's swap), so a stray TTL can no longer trigger a destructive unload while the feature is off, keeping the disabled state identical to pre-feature behavior. 3. Hidden models are not switch targets. The resolver index now skips what Studio hides from its own pickers (the llama.cpp validation probe, RAG embedding weights) via _is_hidden_model, so they can never be auto-switched to by name. Tests added for each. * Studio: bare-id reuse, responses validation order, in-flight tracking Review follow-ups after folding in the per-model overrides and discovery work: 1. A bare model id (no :VARIANT) is now satisfied by any loaded quant of that repo. Previously a bare name resolved to the largest local quant, so it could force a slow reload when a different quant of the same repo was already serving. An explicit repo:VARIANT request still honors the quant. 2. /v1/responses now runs the auto-switch hook after the empty-input validation so a request that 400s can no longer trigger a multi-minute model load before being rejected. The hook still precedes both dispatchers, so streaming requests switch. 3. The keep-warm middleware now tracks in-flight requests whenever auto-switch is enabled rather than only when the idle TTL is already positive, so a stream that starts with the TTL at 0 is still protected if idle-unload is enabled mid-stream. Off still passes straight through. Tests added for each. * Studio: tighten auto-switch code comments Comment/docstring-only pass over the OpenAI auto-switch feature: collapse multi-line blocks, drop a comment that restated the gate it sits next to, and trim verbose docstrings on internal helpers while keeping the load-bearing rationale (concurrency, API behavior, drop-in compat, gotchas). No logic change: verified comment-only with the AST/printer signature check. * Studio: bind auto-switch locks per running loop Review follow-up. The auto-switch swap lock and the keep-warm unload gate were module-level asyncio.Lock objects. That is safe under the single uvicorn loop and on Python 3.10+ (the Lock resolves the running loop lazily on acquire), but a module-level Lock binds to one loop on pre-3.10, which can raise a loop mismatch in multi-loop runners. Resolve each lock through a per-loop accessor backed by a WeakKeyDictionary so every running loop gets its own Lock and stale loops are collected. No behavior change under the server's single loop. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: auto-switch re-review fixes (body codes, coverage, swap, alias, tracking) Follow-ups from a second review pass over the opt-in OpenAI auto-switch feature: 1. OFF-state status codes: /v1/completions and /v1/embeddings moved the body read ahead of the loaded-state check, so a malformed/empty body with no model loaded returned 500 instead of the prior 503. A shared helper reads the body defensively (an unparseable/non-dict body yields no model), and the handler re-reads after the 503 gate to surface the original parse error exactly as before. OFF behavior is unchanged. 2. Local-model coverage: the resolver index only scanned ./models and the active HF cache, while the model picker also lists the legacy/default HF caches, LM Studio dirs, and user scan folders. A request for one of those named models silently served the loaded model instead. _build_index now scans the same roots (Ollama's symlink-creating scanner is skipped on the request path), and resolution is offloaded with asyncio.to_thread so the wider scan never blocks the event loop. 3. Swap vs in-flight stream: a cross-model swap killed the llama-server while another client was still streaming from it. The hook now tracks how many requests are streaming on the loaded model (in-flight minus those still inside the hook) and returns 409 instead of swapping while one is active. Concurrent same-model requests never reach this path, so they are unaffected. 4. Idle-unload + alias: after idle-unload freed the model, an unknown/alias name resolved to nothing and 503'd, though it served the active model before the TTL. Idle-unload now remembers the freed id and an alias request reloads it (only an already-local model, so no remote download), cleared once a model is loaded again. 5. In-flight tracking: the keep-warm middleware tracked in-flight only while the feature was on, so a stream started while off could be unloaded if idle-unload was enabled mid-stream. It now tracks on every inference path; counting is cheap and invisible to clients. Tests added for each. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove stray async_task_outputs files committed by mistake * Studio: auto-switch review round 3 (revert swap guard, hardening) Addressing a third review pass: - Revert the cross-model swap guard. It counted keep-warm in-flight (which includes external-provider calls that never touch the local model) and so could 409 a local swap spuriously, and it still left a same-model request able to start streaming on the model a concurrent swap was unloading. A correct fix needs a request-lifetime reader/writer barrier; a partial guard was worse than the honest single-slot behavior, so concurrent different-model use is back to being serialized (documented), like llama-swap's single slot. - Non-string request model (e.g. {"model": 123} on a raw-body endpoint) is now treated as absent, so it falls through instead of raising in the membership checks once an idle-unload stash exists. - Idle-unload now stashes and replays the freed quant: an alias reload restores the exact (id, variant) that was freed rather than the largest local quant. - Anthropic /v1/messages validates max_tokens before the auto-switch hook, so a request that 400s never triggers a model load. - Keep-warm tracks a pending count for requests waiting on the unload gate, so the idle loop cannot unload the model out from under a request that is blocked on the gate but not yet counted as in-flight. - The idle-unload task is awaited after cancel on shutdown to avoid pending-task warnings. - The resolver's HF cache scan is None-safe and logs at debug instead of letting a bad root abort the whole index build. - upsert_app_setting_map_entry rolls back explicitly on error. Tests updated/added for each. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep saved idle-unload seconds when auto-switch is toggled off * Studio: auto-switch hardening (thread-safe lock maps, body validation) Defensive fixes from review: - Guard the per-loop WeakKeyDictionary get-or-create for both the unload gate and the auto-switch lock with a threading lock, since WeakKeyDictionary mutation is not thread-safe when two event loops run on different threads. - Build the resolver index under the cache lock so concurrent callers with an expired cache don't all run the multi-dir scan at once. - /v1/completions and /v1/embeddings return a clean 400 for a valid JSON body that is not an object (e.g. a list), instead of a 500 from body.get(...). - The keep-warm middleware only tracks POST requests (inference is always POST), so CORS preflight (OPTIONS) is not counted, and tolerates a None path. Tests added for the list-body 400 and the non-POST skip. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: auto-switch review round 4 (local-path load, swap guard, idle fixes) From a 10-reviewer pass: - HF-cache entries now load by a concrete local path, not the bare repo id. The resolver records a load_path (the snapshot dir for a models--* cache repo, the file/dir otherwise) so /load takes the local branch and can never trigger a download to satisfy a partial cache. The advertised loader_id (repo id) is kept as the launch-override key. resolve_local_gguf now returns (load_path, variant, loader_id). - Re-add a single-slot swap guard: a cross-model swap returns 409 model_switch_busy while another inference request is active rather than killing its stream (the caller is excluded from the count), and holds the keep-warm gate across the load so no new inference starts mid-swap. Concurrent same-model requests never reach this path. A residual spurious 409 is possible while a concurrent or external- provider request is active; that is the documented single-slot tradeoff. - Idle keep-warm tracks (model_identifier, hf_variant): reloading the same repo at a different quant counts as a fresh model, so it is not unloaded before one TTL. - Track Studio's own /api/inference/generate/stream so the idle loop can't unload the model mid-stream on that route. - A successful manual /load clears the idle-unload reload stash synchronously, not only on the next idle poll. Also merged origin/main (the branch had fallen behind, which would have reverted unrelated files on merge). Tests added/updated for each. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: auto-switch review round 5 (concurrency, identity, load gate) From a 10-reviewer pass (9 request-changes, 1 approve): - Concurrent same-target requests load once instead of each returning 409. The count-based busy guard could not tell "another request wants the same model" (safe, load once) from "another request is using the loaded model" (refuse). Track in-flight auto-switch requests per (target, variant) and subtract same-target waiters from the busy count; a cross-model swap still 409s while a genuinely different request is active. - Fix the identity confusion introduced when round 4 began loading by concrete local path: the backend identifier became a filesystem path. Record the advertised repo id on the backend after an auto-switch load and use it so (a) a model loaded manually by repo id is recognized as already serving (no spurious reswap/409), (b) /v1/models reports the repo id, never a host path or a duplicate, and (c) the idle-unload stash keeps the override keyed by the repo id, so an alias reload after TTL keeps the user's saved launch flags. - Gate the manual /load route with the keep-warm lifecycle gate so idle auto-unload can't unload a model mid-load. load_model now wraps _load_model_impl in the gate; auto-switch calls _load_model_impl directly since it already holds the gate. - Restore default-off parity on Anthropic /v1/messages: an unloaded backend with auto-switch disabled 503s before the max_tokens 400 check, as it did pre-feature. When the feature is on, request-shape validation still runs before any load. Tests added for each; full backend suite diff vs baseline is unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: auto-switch review round 6 (concurrency ordering, leaks, unload gate) From a second 10-reviewer pass (8 request-changes, 2 approve): - Same-target concurrency: register a waiter by the raw requested model before the (slow) resolve, and exclude pending requests from the swap busy count. The middleware counts a concurrent same-model request as in-flight before it resolves and joins the resolved-target waiter map, so the prior fix could still 409 it. The guard now subtracts max(same resolved-target, same raw-request) waiters and ignores pending (a pending request is blocked in the middleware, not generating, so a swap can't interrupt it). - External-provider requests no longer block a local swap. The keep-warm middleware counts every inference-path POST, but external-provider chat returns before the auto-switch hook and never touches the local GGUF. The chat handler now untracks itself before proxying, so its in-flight stream can't trip model_switch_busy on a concurrent local auto-switch. The middleware skips its own end-decrement for an untracked request. - Manual /unload is gated like load and idle-unload: it holds the lifecycle gate and returns 409 rather than tearing down llama-server while an inference request is in flight. - Response model id no longer leaks the load path. /v1/models already advertised the repo id; chat, completions, embeddings, Anthropic messages, and audio response bodies now use the same _llama_public_model_id helper instead of the concrete on-disk model_identifier. - Chat completions validates the non-system-message requirement before the auto-switch hook (as /responses and /messages already do), so an invalid request can't swap the resident model before returning 400. Tests added for each; full backend suite diff vs baseline is unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: auto-switch review round 7 (teardown policy, Unsloth-active swap, training) From a third 10-reviewer pass (9 request-changes, 1 approve), all on the same asymmetric-teardown theme. Resolved per the intended policy that only automatic paths defer to an active stream; deliberate user actions stay interrupting: - Revert the manual /unload in-flight guard added last round. A manual /load or /unload is a deliberate action and tears down immediately, as before; only the automatic idle-unload loop and auto-switch defer to an active request. This removes the asymmetry the reviewers flagged (manual /load, the /unload Unsloth branch, and the opposite-backend swaps inside _load_model_impl) by not extending the guard to deliberate paths, rather than spreading it. - Auto-switch now refuses a swap whenever another inference request is in flight, not only when a GGUF is already loaded. _load_model_impl also unloads an active Unsloth/transformers backend before loading a GGUF, so the busy guard must cover that case too; otherwise an Unsloth stream could be killed by an auto-switch. - Refuse API-initiated training while inference is active. When Studio is driven as an inference API (sk-unsloth key auth), POST /api/training/start returns 409 if a request is in flight, since training frees VRAM by unloading the chat model and would kill the stream. The Studio UI (session auth) still starts training and coexists/frees VRAM as before. A mixed UI+API session is not yet special-cased. Adds auth.authentication.authenticated_via_api_key. Tests added/updated for each; full backend suite diff vs baseline is unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: add UNSLOTH_MODEL_IDLE_TTL env override for idle-unload Borrowed from PR 6517: a startup env var that sets the idle-unload TTL without the settings UI. Unlike the stored setting (gated on auto-switch), the env value is a standalone default that enables idle-unload even with auto-switch off, for headless/container deploys. An explicit UI/API value still overrides it and stays gated. The settings GET reflects the env default when nothing is stored. * Studio: auto-switch fixes from review (paths, embeddings input, env idle reload) - /v1/models advertises a client-facing alias instead of a filesystem path: the ./models and LM Studio scanners report the on-disk path as the model id, so the index now prefers model_id/display_name as the advertised/override id and keeps the concrete path internal as load_path, still resolvable by path. - /v1/embeddings validates input before auto-switch: a request with a model but no input now 400s before the hook (like chat/responses/messages), so an invalid embeddings request cannot unload or swap the resident model. - Standalone UNSLOTH_MODEL_IDLE_TTL reloads the freed model: the hook now runs when auto-switch or idle-unload is active, and with auto-switch off it skips the resolver and only restores the idle-unloaded model, so the first idle timeout no longer leaves later /v1 requests with nothing loaded. - Do not resurrect a stale GGUF over an active Unsloth model: the reload-stash path bails when a non-GGUF backend is loaded, so an unknown /v1 name cannot tear down a live Transformers/Unsloth model. - Defensive HF cache scan: each cache root's resolve/dedup is wrapped so a missing or malformed root skips that root rather than aborting the index. - Single-model retrieve checks the id is a string before lowercasing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: fix automatic-load asymmetry, audio reload, preview, idle timer The standalone UNSLOTH_MODEL_IDLE_TTL reload is a second automatic-load trigger, but several validate-before-switch guards and reload hooks only checked the auto-switch toggle. Add a shared _automatic_model_load_may_run() (auto-switch on, or idle TTL > 0) and route every guard through it. - /v1/completions validates prompt before any automatic load (it was the one model-bearing route with no pre-check). - /v1/chat/completions and /v1/embeddings pre-checks gate on the shared predicate so a standalone idle TTL cannot reload then reject. - /v1/messages no longer 503s before the reload hook can restore an idle-freed model when auto-switch is off. - Raw completions/embeddings with no model field pass a non-empty sentinel so the idle-stash reload runs, restoring the legacy "omit model, use loaded" path. - /api/inference/audio/generate gains the reload hook (after message validation) so an idle-freed audio GGUF is restored. - Public preview opts out of auto-switch via a request-scope flag, so a caller's model field cannot swap away from the pinned checkpoint; preview chat streams are now matched by _is_inference_path so idle-unload cannot kill them. - Keep-warm no longer stamps activity on request start, and external-provider untracking decrements without restamping, so periodic external traffic can no longer keep the local GGUF warm forever. Merges origin/main (the branch had fallen behind, which also brought in the preview route the review flagged). * Studio: surface model auto-switch in the API tab and demo it in examples The OpenAI auto-switch toggle previously lived only in Settings -> General. Add the same toggle to the API tab's usage-examples panel (it shares the settings cache), and make the examples reflect it: when on, the Python examples append a second call naming a different downloaded GGUF (so the model field visibly selects which model serves), and the curl examples gain a one-line note. Reuses the existing settings API client and i18n keys. * Studio: harden OpenAI auto-switch reload-only path and Anthropic tool validation - Omitted-model raw-body requests pass a reload-only sentinel so the idle-stash reload still restores an idle-freed model, but the resolver never matches a downloaded GGUF literally named "default". - Reject malformed Anthropic client tools before _maybe_auto_switch_model so an invalid request can no longer evict the loaded model. * Studio: extend auto-switch reload-only and tool validation to schema endpoints - Schema-backed endpoints (chat completions, responses, count_tokens, messages, audio) defaulted an omitted model to "default" and passed it to the switch hook, so a downloaded GGUF named "default" could be swapped to. Route the hook through a helper that switches only on an explicitly set model, else reload-only. - Propagate the explicit-set status when building the chat request from a Responses request, so the non-streaming chat re-check stays reload-only too. - Validate Responses function tools before the switch hook so a malformed tool returns 400 without evicting the loaded model. * Studio: serialize auto-switch swaps across event loops with a process-wide gate The auto-switch lock is a per-event-loop asyncio.Lock, so two /v1 swaps on different loops in one process could both pass it and race the single model slot (the backend and _load_model_impl are process-wide). Add a process-wide threading gate around the swap, acquired off the loop so a cross-loop wait never blocks it, layered with the existing per-loop lock. Add a cross-loop test that fails without the gate (two slow loads overlap) and passes with it. * Studio: make the auto-switch swap gate wait cancellation-safe _acquire_swap_gate awaited asyncio.to_thread(lock.acquire) when another loop held the process-wide gate. to_thread cancellation doesn't stop the worker thread, so a /v1 request cancelled mid-wait (client disconnect during a cross-loop swap) would have its thread acquire the gate after the fact, while the finally that releases it never runs -- permanently deadlocking later auto-switch swaps. Poll a non-blocking acquire off a short asyncio.sleep instead: it still keeps the wait off the loop and serializes across loops, but a cancel now lands during the sleep, when the gate is not held, so nothing leaks. Add a test that deadlocks the to_thread variant (it times out) and passes with the poll. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: validate modality and tool-confirmation before auto-switch Two more request shapes could load a named GGUF and only then 400, evicting the resident model: - An image request naming a different text-only GGUF. The switch hook now takes require_vision and rejects a swap to a non-vision target before loading it; a GGUF's vision capability is its companion mmproj, knowable without a load, and matches the post-load guard. Only the resolver branch is checked, never the reload-stash restore. - confirm_tool_calls=true with stream=false and local tools. /v1/chat/completions now rejects that shape before the hook, mirroring the local tool path's bypass_permissions exemption and intent signal. The vision probe threads the ambient HF token to keep the capability-probe invariant. Reload-only and idle-reload paths are unaffected. * Studio: extend validate-before-switch and make the lifecycle gate process-wide - /v1/messages/count_tokens now rejects malformed client tools before the switch hook, like /messages (shared _validate_anthropic_client_tools helper), so a count request can't evict the loaded model. - /v1/chat/completions rejects a malformed tool_choice forcing object (a {"type":"function","function":{}} with no name) before the switch hook. - The inference lifecycle gate that blocks new inference during a swap is now process-wide (a poll-acquired threading lock, cancellation-safe), not a per-loop asyncio lock, so a request on another event loop can't start inference while a swap tears the single backend down. - Usage examples no longer hard-code a switch-demo repo most users lack; the model is an explicit placeholder the user replaces. * Studio: extend the auto-switch modality guard to /v1/responses and /v1/messages The pre-load vision check that guards /v1/chat/completions now also runs on /v1/responses and /v1/messages, so an image request naming a text-only GGUF is rejected before the swap and never evicts the resident vision model. Run the vision capability probe off the event loop. Make the /v1/models retrieve loaded fast-path case-insensitive, and never advertise a host path from the resolver. Remove the dead list_switch_eligible_ids helper, superseded by the /v1/models catalog. * Studio: filter /v1/models to GGUF, per-loop catalog lock, reject system-only Responses Address review findings on the auto-switch path: - /v1/models advertises only GGUF models the API can actually switch to; a safetensors/LoRA entry would be selectable but never loadable via llama.cpp. - The /v1/models catalog cache uses a per-loop lock (like the auto-switch path) so a second event loop awaiting it can't hang in a multi-loop process. - /v1/responses rejects system/developer-only input before the switch, mirroring chat, so an invalid request can't evict the resident model. - _build_index guards each scan source on its own so one bad root drops only that source; the vision probe logs a real detection failure instead of swallowing it. * Studio: list cached GGUFs in /v1/models by inspecting files, not model_format The HF-cache scanner leaves model_format unset for GGUF snapshots, so the previous model_format == "gguf" filter dropped every downloaded HF-cache GGUF from /v1/models and the retrieve fallback. Decide GGUF-ness from the on-disk files via the resolver (info_has_local_gguf) instead, run off the event loop, so the catalog advertises exactly what /v1 can serve. * Studio: fix /v1/messages/count_tokens route binding plus auto-switch review fixes The @router.post decorator for /messages/count_tokens had been separated from anthropic_count_tokens by the _validate_anthropic_client_tools helper, so the route bound to the validator and dropped its auth dependency. Move the decorator back onto the handler. Add route-binding tests asserting each /v1 endpoint maps to its handler with the auth dependency, so a decorator/handler split is caught at the route level (the direct-call tests missed it). Also from review: - update_openai_auto_switch writes both settings keys in one transaction so a PUT can't leave one updated and the other stale (drop the now-unused single setters). - max_seq_length override rejects 0 at the boundary (ge=1) instead of accepting then silently dropping it. - Document that embeddings auto-switch is best-effort: GGUF pooling has no cheap pre-load probe like vision's mmproj, so a guard would false-reject GGUF embedders. - Add a positive idle-unload test (loop frees the model and stashes it for reload). * Studio: validate Responses tool_choice + Anthropic mixed tools before switch, filter Ollama from catalog More auto-switch review findings: - /v1/responses rejects a forcing-function tool_choice with no name before the switch, mirroring chat, so a malformed request can't evict the resident model. - /v1/messages rejects mixing Anthropic server tools with custom client tools before the switch (the check depends only on the payload, so it moves up cleanly). - /v1/models no longer advertises Ollama-link models: info_has_local_gguf excludes .studio_links / ollama_links entries, which the resolver skips and can't switch to, so an advertised id never silently falls through. * Studio: guard chat audio input before switch; surface env-backed idle unload in settings UI A chat request carrying audio_base64 rides the same companion mmproj projector as a vision request, so a text-only target cannot serve it either. Flag require_vision for audio input as well so the multimodal probe runs before the switch and a rejected request never evicts the working model. Generalize the reject message to cover image and audio. The settings response now reports idle_unload_active (effective TTL > 0) so the UI can distinguish idle-unload that is active via the UNSLOTH_MODEL_IDLE_TTL env var from the case where it needs the toggle enabled. * Studio: harden auto-switch eviction guards (count_tokens vision, TTS reload-only, mmproj/stash) Four eviction/correctness fixes on the opt-in /v1 auto-switch path: - /v1/messages/count_tokens now carries the same require_vision guard as /messages, so an image count naming a text-only GGUF can't evict a loaded vision model for a swap that can't serve the request. - /audio/generate is now reload-only. A local GGUF's audio-input capability is not a cheap pre-load probe (the companion mmproj signal can't tell an audio projector from a vision one, and codec TTS ships no projector), so resolving the client model could load a text/vision-only target and evict the working audio model before the audio check fails. Only the idle-stash restore runs here; switching TTS models is an explicit /load. - The resolver no longer treats a standalone mmproj .gguf as a servable model. _scan_models_dir's standalone-file pass does not filter mmproj the way its directory scan does, so /v1/models could advertise a projector and a switch could load it over the real weights. - A non-GGUF (Transformers/Unsloth) load and a deliberate /unload now clear the idle reload stash, so a manual load/unload is never superseded by a stale idle-freed GGUF that the next /v1 request resurrects. * Studio: report advertised repo id consistently after an auto-switch Two model-id reporting fixes so an auto-switched cached HF GGUF is named by its repo id everywhere, not its snapshot path: - Streamed /v1/responses envelopes now derive the model id from _llama_public_model_id (which prefers _openai_advertised_id) instead of the raw model_identifier. After an auto-switch the identifier is the snapshot path while the repo id lives in _openai_advertised_id, so the stream used to report a snapshot basename while /v1/models, chat completions, and non-streaming Responses all reported the repo id. - When an advertised alias already resolves to the loaded model (a model loaded by local path, requested by its repo or LM Studio id), the already-serving early return now records the alias as the advertised id, so /v1/models and responses report the alias and mark it loaded instead of the path-derived basename. Resolver branch only; safe lock-free because an in-flight request blocks any concurrent swap via the single-slot busy guard. * Studio: validate request shapes before auto-switch (prompt/input/audio/mcp confirm) Four more validate-before-switch guards so a deterministic client error never evicts the resident model on the opt-in /v1 auto-switch path: - /v1/completions rejects an object/number prompt (only a string or array is valid) before the switch, instead of loading the named GGUF and letting llama-server reject the shape afterward. - /v1/embeddings rejects an object/number input the same way. - Chat rejects an oversized audio_base64 upload (413) before the switch. The size cap is a cheap, target-independent length check; the decode itself stays post-switch to avoid decoding a valid upload twice. - The chat confirm-without-stream pre-switch guard now mirrors the tool loop's actual enablement: _effective_enable_tools (honoring a CLI --enable-tools policy) and mcp_enabled (which opens the tool loop on its own but defers to a CLI --disable-tools policy). Previously a confirm+no-stream request with only mcp_enabled slipped past and 400'd after the swap. * Studio: fix model-id retrieval, streaming n>1, resolver cache TTL, keep-warm auth Four fixes from review: - GET /v1/models/{id} legacy raw-path fallback now maps the raw identifier to the same public id its /v1/models entry uses. After an auto-switch load the identifier is the snapshot path while the entry is keyed by the advertised repo id, so a client that cached the old absolute path no longer 404s on a model that is in fact loaded. - stream=true with n>1 is now rejected before the switch. Only the non-streaming GGUF path returns multiple choices, so streaming n>1 is invalid on every local serving path; both fields are known pre-switch, so it must not load model B only to 400 and evict model A. Non-streaming n>1 stays post-switch where the serving path decides. - The resolver index cache is stamped after _build_index, not with the pre-scan timestamp. On installs with enough local models for the multi-root scan to exceed the 5s TTL, the cache was stored already expired and every request rebuilt it. - The keep-warm middleware no longer stamps model activity for 401/403 responses. It runs before FastAPI auth, so unauthenticated probes used to refresh the idle timer without touching llama.cpp; they now decrement the in-flight count without keeping the model warm. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>
--------- Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>
…d_embedding on tied/vLLM models (#6774)
* Fix gpt-oss offload_embedding and generate() logits_to_keep on fused models
offload_embedding=True moved embed_tokens to CPU but left the input/output device-shuffling forward hooks commented out ('[TODO] Doesn't seem to work!'), so an eager forward/generate with CUDA input_ids hit the CPU embedding and raised a device-mismatch RuntimeError. Re-implement them in a testable helper _install_offload_embedding_hooks that saves the origin device on the module (the pre-hook returns a new tensor, so a device stashed on the original input is lost) and runs the lookup on the embedding weight's CURRENT device. Reading the weight device at call time (not a hard-coded cpu) also handles a non-quantized (bf16) embedding that a later model.to(...) pulls back onto the GPU, which the hard-coded version broke in the opposite direction.
unsloth_base_fast_generate injected logits_to_keep/num_logits_to_keep whenever an inner submodule forward accepted it, but transformers validates generate kwargs against the top-level prepare_inputs_for_generation (plus forward when it takes kwargs). On fused/PEFT-wrapped gpt-oss this raised 'model_kwargs are not used by the model: [logits_to_keep]'. Only inject when the top level would accept it, mirroring transformers _validate_model_kwargs. Behavior is unchanged for every model that works today.
Adds tests/test_offload_embedding_hooks.py and tests/test_generate_kwarg_gate.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss offload hooks: store origin device on the tensor, not the shared module
The pre-hook stashed the input device on embed_tokens itself, which races when
concurrent forwards share the module (serving). Ride it on the moved tensor and
read it from the post-hook args instead: stateless and thread-safe.
* Also strip mm_token_type_ids that generate() rejects (Qwen3-VL vision GRPO)
The vision processor (Transformers 5.x path) emits mm_token_type_ids, which
Qwen3-VL's generate() then rejects in _validate_model_kwargs on transformers
4.x, so vision GRPO fails at the first rollout:
ValueError: The following `model_kwargs` are not used by the model:
['mm_token_type_ids']
Unlike logits_to_keep this is an incoming kwarg rather than one we inject, so
drop it in unsloth_base_fast_generate when the top level generate does not
accept it, reusing the same _unsloth_generate_accepts_kwarg gate. Extends the
GPU-free gate test with the accept/reject mm_token_type_ids cases (7/7 pass).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim mm_token_type_ids comment
* Trim comments in gpt-oss offload/logits fix (comment-only)
* gpt-oss offload: return embedding output to the decoder device, not the input's
When offload_embedding moves the embedding to CPU, model.device can become CPU and
inputs then arrive on CPU, so returning the output to the input device left it on CPU
and the CUDA decoder hit a device mismatch. Capture the decoder device before offload
and always return there. This also drops the per-request tensor state (stateless, so
concurrent forwards stay correct).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss offload: refuse offload_embedding for tied word embeddings
Tied models share embed_tokens.weight with lm_head, so offloading the weight
to CPU strands the output projection there (device mismatch at generate) and
saves no VRAM since lm_head still needs it on GPU. Detect the shared weight via
get_output_embeddings and raise NotImplementedError instead of loading into a
crash. Untied models (gpt-oss, Llama-3.1-8B) offload unchanged.
Adds tests/test_offload_tied_guard.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss offload: skip embedding offload on fast_inference (vLLM)
vLLM manages its own weights, so offload_embedding cannot apply on the
fast_inference path (previously it was silently ignored). Disable it with a
notice, mirroring the WSL and Windows skips.
* Trim offload embedding comments (comment-only)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss offload: track decoder device live so it survives model.to()
The post-hook returned the embedding output to a device captured at load time.
If a model is loaded on CPU then moved with model.to(cuda), that device is
stale and the output lands on the wrong device. Read the decoder device live
from the (untied) output embeddings, keeping the captured device as a fallback.
Adds a stale-fallback regression test.
* Make generate-kwarg-gate cases pytest-collectable
Cases lived in run(), which pytest does not collect, so CI never exercised the
gate. Expose them as test_generate_kwarg_gate; still runnable via __main__.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss offload: skip a meta (disk-offloaded) lm_head as the return device
A device_map that disk-offloads an untied lm_head leaves its weight on the meta
device until that module's own hook runs, so reading it as the decoder device
would move real hidden states to meta. Skip meta (and a missing weight) and fall
back to the captured device. Adds a regression test.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add a shared fits-on-device filter to the model selects The chat model selector gains an Only show models that fit on this device tick under its filter row, and the Hub page gains a matching Fits device pill next to the sort menu. Both read one persisted preference (unsloth_models_fit_on_device_only), so toggling either applies to both. The filter reuses the Recommended sort's existing fit math, extracted into hfModelFitsDevice: size from safetensors metadata, GGUF param count, or the repo name, against the 0.7 GPU + 0.7 RAM budget, with unsizable models hidden. In the chat selector it extends the fit filtering to the Trending and Recent sorts and to search results; downloaded models stay visible regardless. An unknown device budget keeps everything. The preference is cleared by Reset all local preferences like the other picker toggles. * Move the device-fit toggle into the sort dropdowns * Tighten sort menu footer spacing and shorten the label * Align the footer checkbox with the option text * Make the footer checkbox circular with a smaller tick * Clear menu highlight when the pointer leaves the options * Address review: fit filter coverage and sizing Exempt on-disk models from the Hub fit filter, apply it to the feed trending rows and curated search results, size safetensors and MLX rows by the quantized load estimate instead of checkpoint bytes, and replace the native title hint with the app Tooltip. * Make the whole device-fit row toggle the filter
* Studio RAG: fix RTL/Indic PDF corruption and dropped DOCX tables The RAG parser prefers pymupdf4llm.to_markdown for PDFs, but that rebuilds text from positioned glyphs and mangles complex-shaping scripts (RTL Arabic/Hebrew come back as shaped Presentation Forms, Indic matras drop to U+FFFD) and can silently drop most of a heavy-RTL page. _pdf now compares the Markdown against PyMuPDF's logical-order get_text() per page and falls back to it when the Markdown looks corrupted (shaped Presentation Forms or U+FFFD above a small floor/ratio) or holds far fewer letters than the raw layer. Latin PDFs are unaffected and keep their Markdown tables/headings. _docx walked document.paragraphs, which excludes table cells, so DOCX tables were dropped entirely. It now walks body content in document order via iter_inner_content, emitting each table row as pipe-joined cells (deduped across merged cells); the preview locator already anchors on pipes. Adds parser tests for the corruption and incompleteness fallbacks and for DOCX table extraction. These mirror the chat document-extractor guard raised in the unslothai/ unsloth#5351 review; the RAG parser is a separate module and needed its own fix. * RAG DOCX: keep empty table cells and collapse in-cell newlines Skipping empty cells shifted later cells left and broke column alignment across rows; a cell with internal paragraphs (newlines) also broke the pipe-joined row. Keep every cell (dropping the row only when all are empty) and normalize each cell with " ".join(split()) so multi-paragraph cells stay on one row. Adds a test for both. * RAG DOCX: dedup merged table cells on the <w:tc> element directly Store the shared <w:tc> lxml element in the seen set instead of its id(); it is hashable and compares by the underlying node, so it dedups spanned/merged cells the same way without relying on id(). Adds a merged-cell test. * RAG DOCX: align merged cells, pad skipped grid columns, flatten nested tables * RAG DOCX: walk cells in document order so nested tables keep in-cell position * RAG DOCX: dedup vertically merged cells so a spanning label is indexed once --------- Co-authored-by: danielhanchen <michaelhan2050@gmail.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
… reorganization (#6800) * Add customizable RAG embedding model setting and reorganize settings tabs Chat with files, project sources, and knowledge bases previously always embedded with unsloth/bge-small-en-v1.5. This adds a Settings option to pick any Hugging Face embedding model (or local path), with HF search autocomplete, server-side verification that the repo is actually an embedding model, and a save anyway escape hatch for offline or local models. The setting persists in app_settings and applies at runtime to both the sentence-transformers and llama-server GGUF embedder backends without a restart. Also reorganizes the General settings tab: Documents & RAG sits above Uploads, Helper LLM moved above the danger zone, and Model auto-switch (OpenAI API) moved to the bottom of the API tab. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Support local model paths on the GGUF embedder and normalize default saves Found by simulation testing of the embedding model setting: Local paths saved as the embedding model now work on the llama-server GGUF backend (the default backend on macOS and CPU). A path to a .gguf file is used directly and a directory is scanned for a variant-matching non-mmproj .gguf, with a clear error when none exists. Previously a local path was sent to the HF hub API and failed with a repo lookup error. Saving the default model explicitly no longer stores an override, so is_custom stays false and the UI does not show a reset button for the default value. * Address review: stale-vector handling, GGUF derivation, save-time guards Review follow-ups, each verified by new tests: Re-uploading a document after an embedding model change now re-indexes instead of deduping by content hash. Documents record the embedder that produced their vectors (lazy embedding_model column, NULL legacy rows keep deduping) and a mismatch replaces the old document. A vector width change no longer bricks the dense index. ensure_vec drops and recreates chunks_vec when the dim changes (old vectors are in a foreign space and only block inserts) and search_dense returns empty on a width mismatch instead of surfacing a vec0 error, so lexical search keeps working until documents are re-uploaded. Saving a local sentence-transformers folder with no .gguf now returns 409 with a clear message when the install embeds via llama-server, instead of failing at first index. force still saves. A custom RAG_EMBEDDING_MODEL env without RAG_EMBED_GGUF_REPO now derives the -GGUF companion repo instead of silently keeping the bge GGUF on CPU and macOS installs. The resolved GGUF path is tagged with the repo captured at entry, so a setting change during a download cannot mark the old model as current. GGUF repo detection matches gguf as a whole name segment rather than a substring, hf_token is trimmed before verification, and the settings combobox drops a redundant state mirror of its controlled value. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Shrink embedding model font to 11px in the input and dropdown The combobox wrapper applies className to the outer input group, so the size utility must target the inner input element; the previous text-xs never reached it and the field rendered at the browser default. * Show curated unsloth embedding models when the search field is empty The empty-query listing was the global top-downloads page, which holds no unsloth mirrors for the unsloth-first float to reorder, so the dropdown opened on third-party models. Match the model picker: curated unsloth listing when empty, whole-Hub search once a query is typed. * Address review: settings resilience and index consistency Keep the last known embedding model on settings store errors, remove the re-entrant dim lock in the llama-server backend, accept local GGUF saves and verify GGUF availability for HF repos on that backend, match local path embedders exactly in model list filters, drop same-width stale vectors from dense search, pin the embedder per ingestion job, and only replace completed documents after the re-index succeeds. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Consolidate the GGUF repo derivation tests * Trim to a single core embedding-model test * Address review: GGUF repo saves and cache race Accept a GGUF-named HF repo on the llama-server backend by verifying GGUF availability instead of the sentence-transformers metadata gate, and guard the settings cache with a generation counter so a read overlapping a save cannot repopulate it with the pre-save value. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
…lls (#6805) * Studio: self-heal a pre-#6483-fix anyio>=4.14 stuck in existing installs The <4.14 cap in constraints.txt/no-torch-runtime.txt only constrains new anyio resolutions. An install made before that cap existed can already be sitting on anyio 4.14+, and since it already satisfies mcp/fastmcp's anyio>=4.5 floor, every later constrained install skips it as already-satisfied -- so affected installs never recover and keep hitting the cancel-scope RuntimeError on every request (#6797, a recurrence of #6483). Force-reinstall anyio<4.14 whenever a stuck 4.14+ is detected. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: also repair anyio on the update fast path setup.sh's _SKIP_PYTHON_DEPS and setup.ps1's $SkipPythonDeps skip install_python_stack.py entirely once the installed package version already matches PyPI latest, so an install stuck on anyio>=4.14 with an otherwise up-to-date package never reaches the repair added in install_python_stack.py. Probe anyio on that fast path too and fall through to the full dependency pass when it's still >=4.14, mirroring the existing ROCm/CPU-torch override right below it. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
… monitor (#6509) * feat: Implementation of the Portuguese (Brazil) language and VRAM/RAM monitor. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update studio/frontend/src/hooks/use-gpu-utilization.ts Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update studio/backend/main.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update studio/backend/main.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update studio/backend/utils/hardware/hardware.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update studio/backend/utils/hardware/hardware.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update studio/frontend/src/features/settings/components/usage-examples.tsx Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update studio/backend/main.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update studio/backend/utils/hardware/hardware.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update studio/frontend/src/features/studio/sections/progress-section.tsx Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * fix: resolve automated review feedback on API shape * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix review issues for PR #6509: Cpu icon, VRAM percent, system polling - model-inspector: use the exported CpuIcon (Cpu is not a Hugeicons export) - app-sidebar: guard the VRAM percent on totalVram to avoid Infinity, and reset the system poll cache only after each request settles so a slow probe is reused instead of stacking overlapping requests - use-gpu-info: populate CPU/RAM on hosts without a GPU - progress-section: label GPUs by visible_ordinal instead of array index - hub-page: base the RAM label on systemRamTotalGb - usage-examples: emit JS sampling and tool options at the top level instead of nesting them under extra_body (the JS SDK does not unwrap extra_body) - main: read torch and transformers versions from package metadata instead of importing the libraries on every system poll, and guard the VRAM math against null values - hardware: translate a leftover comment to English * Harden /api/system: guard psutil.boot_time for PR #6509 Simulating restricted containers and some VMs (where psutil.boot_time can raise) showed the /api/system endpoint would 500 on the unguarded boot_time call, the same failure class already handled for cpu_freq, disk_usage, and Process. Wrap boot_time and return uptime_seconds as null when it is unavailable so the sidebar monitor degrades gracefully instead of breaking. Widen the uptime_seconds type to number | null to match. * Studio: make the sidebar hardware monitor a toggle (default on) for PR #6509 Adds a "Show hardware monitor" switch under Settings > Appearance > Layout, backed by a localStorage preference (default on), mirroring the existing useSidebarPin pattern. When turned off, the sidebar hides the VRAM/RAM meters and useSystemInfo stops the 3s /api/system poll entirely, so no nvidia-smi / SMI probes run while the monitor is disabled. Adds the en and pt-BR strings. * Studio: default the sidebar hardware monitor to off (opt-in) for PR #6509 * Studio pt-BR: fix three small translation defects for PR #6509 - learningRateDescription: "5e-5 for CPT" -> "5e-5 para CPT" (leftover English) - exportScopeRecents: "Recents" -> "Recentes" (untranslated) - relativeMonthsAgo/relativeYearsAgo: add the missing space ("há {count} meses"/ "há {count} anos") so they no longer render as "há 3meses" * Studio pt-BR: translate the last 10 fallback keys for PR #6509 Adds the settings.general.storage block (Armazenamento) and the settings.chat.modelDisclaimer pair, so pt-BR now covers all en keys (679/679) with no English fallbacks. * Studio: hide sidebar VRAM row on CPU-only hosts for PR #6509 * Studio: tighten and trim code comments for PR #6509 * fix: UI issue in the stop button dialog box (fine-tuning) * Studio pt-BR: translate 18 new keys from main merge (password dialog, GGUF export, dataset streaming) for PR #6509 * Rounding to GB * Fix/adjust System resources tab for PR #6509 * Fix/adjust GPU monitor review items for PR #6509 * Fix/adjust remaining GPU monitor review items for PR #6509 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix/adjust MLX resource fallback for PR #6509 * floating window implementation * resize for floating window * Fix resource monitor review items * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore frontend optional dependency lock entries * Make GPU selection tests hermetic * Fix GPU monitor CI test failures * Bound MLX GGUF reload smoke * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix MLX GGUF reload smoke exit --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: wasimysaid <wasimysdev@gmail.com>
…813) --------- Co-authored-by: Ayushman Paul <ayushman@HP>
--------- Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
…ma.cpp (#6472) * Add --with-llama-cpp-dir flag to install.ps1 and install.sh Users can now pass --with-llama-cpp-dir /path/to/llama.cpp to the installer to skip downloading or building llama.cpp and use a local directory instead. A junction (Windows) or symlink (Linux/macOS) is created at the canonical install location, bypassing both the prebuilt download (Phase 3) and source build (Phase 4) steps in setup.ps1/setup.sh. The path is passed via UNSLOTH_LOCAL_LLAMA_CPP_DIR env var which setup.ps1 and setup.sh read directly. Ported from the idea in unslothai/unsloth#4384, reimplemented against current Studio architecture. * test: add static wiring test for --with-llama-cpp-dir flag Cross-checks install.sh, install.ps1, studio/setup.sh and studio/setup.ps1 so the flag's contract (parse -> UNSLOTH_LOCAL_LLAMA_CPP_DIR env var -> link local dir, skip prebuilt download and source build) can't silently regress. Wired into studio-backend-ci.yml alongside the other tests/sh installer tests. * Address review feedback on --with-llama-cpp-dir flag - setup.ps1: delete an existing junction/symlink via DirectoryInfo.Delete() instead of a recursive remove, which can traverse the link and wipe the user's real llama.cpp directory on PowerShell 5.1. - setup.ps1: short-circuit the build chain when a local dir is linked so CMake never runs inside the user's checkout when it lacks a Windows-layout binary. - install.sh / setup.sh: resolve paths with CDPATH= cd -P so a set CDPATH cannot corrupt the resolved path. - install.sh: seed _WITH_LLAMA_CPP_DIR from UNSLOTH_LOCAL_LLAMA_CPP_DIR so an exported env var (piped-install style) is honored instead of being clobbered. - setup.sh: create the root llama-quantize shim when linking a local source build so GGUF export's check_llama_cpp() still finds it. - setup.sh / setup.ps1: drop a stale link before the custom-home ownership assert so re-runs with the flag stay idempotent. - test: pin the new linked-dir build short-circuit. * Harden --with-llama-cpp-dir against Codex/Gemini review findings - install.sh: error when --with-llama-cpp-dir is the final arg with no path, matching the existing --package/--python post-loop guards (was a silent fallback to the normal prebuilt/source install). - studio/setup.sh: canonicalize LLAMA_CPP_DIR before the self-link no-op compare. _RESOLVED_LOCAL is fully resolved while LLAMA_CPP_DIR was textual, so a symlinked $HOME made the guard miss and the rm -rf could wipe the user's real llama.cpp tree. - studio/setup.sh: make the llama-quantize shim non-fatal; it writes through the link into the user's tree, which may be read-only (shared/CI cache), and under set -e a failed ln aborted an otherwise-good reuse. - studio/setup.ps1: detect a broken junction via Get-Item -Force instead of Test-Path so a dangling link from a prior run is removed and mklink can relink to a new valid directory. - studio/setup.ps1: use Copy-Item -LiteralPath so a source path containing [ ] isn't treated as a wildcard in the junction copy fallback. - tests: update the wiring assertions for the LiteralPath copy and the canonicalized compare. * Validate/reuse local llama.cpp tree and guard the in-use case Addresses the second Codex pass on the --with-llama-cpp-dir flag: - Validate the linked tree before disabling installs (setup.sh + setup.ps1): reusing a local dir skips BOTH the prebuilt download and the source build, so the dir must already contain a runnable llama-server (build/bin on Linux/macOS, build\bin\Release\llama-server.exe on Windows). Bail out with a clear message instead of linking an unbuilt/wrong-platform checkout and leaving Studio with no usable binary. - Treat a canonical-path target as already linked when it holds a build (setup.sh + setup.ps1): point the flag at ~/.unsloth/llama.cpp itself and an existing build is reused (skip prebuilt + source) rather than clobbered by the staged prebuilt installer (which uses os.replace()/replace). An empty canonical dir still falls through to the normal in-place install. - Abort when an in-use llama.cpp can't be removed on Windows (setup.ps1): Remove-Item -ErrorAction SilentlyContinue can silently leave a locked tree in place; detect that and stop with the same active-process message + exit 3 the prebuilt path uses, instead of junctioning over a half-present dir. Left as follow-up (already tracked by the PR author as a non-blocker): the in-app "Update llama.cpp" updater does not yet recognize a local-link install as externally managed; that fix belongs in studio/backend/utils/llama_cpp_update.py. * Accept all backend llama-server layouts in --with-llama-cpp-dir validation The linked-tree validation only accepted build/bin[/Release]/llama-server, but LlamaCppBackend._layout_candidates() resolves a root-level llama-server first, then build/bin, then build/bin/Release on Windows. A `make` build or a flat release extract (binary at the dir root) was therefore rejected with a hard installer failure even though Studio would have run it. Validate the same candidate set the backend uses in both setup scripts, and add wiring-test assertions so the check can't silently narrow again. * Treat --with-llama-cpp-dir local links as externally managed A --with-llama-cpp-dir install junctions/symlinks the canonical llama.cpp dir to the user's own checkout, but two backend paths still treated it as a Studio-owned tree: - The in-app updater (llama_cpp_update) offered and could apply an official prebuilt over the link, writing through it into the user's checkout (or failing) and silently dropping the link the flag created. - Orphan cleanup (LlamaCppBackend._kill_orphaned_servers) resolved the linked root into its kill allowlist, so a llama-server the user launched from the same checkout was classified as ours and killed on startup. Detect the canonical dir being a symlink/junction (reparse point) and treat the install as unmanaged: get_update_status reports unsupported, start_update refuses with reason "local_link", and the linked root is left out of the orphan allowlist. Adds behavioral tests (link vs plain dir, updater refusal, and the spared-vs-killed orphan control). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add behavioral shell test for --with-llama-cpp-dir linking The existing tests/sh/test_with_llama_cpp_dir_flag.sh is a static grep of the scripts. This adds a behavioral test that extracts the real link block from studio/setup.sh (by content anchors, with a self-validating extraction) and runs it against hermetic fake dirs, asserting the outcomes that matter: - an external CMake build links and arms neither the prebuilt download nor the source build - a flat / make tree (root-level llama-server, no build/bin) is accepted too - an unbuilt tree is rejected with a non-zero exit and no link left behind - relinking over a stale link preserves the target's contents (no data loss) - pointing at the canonical path is a no-op reuse, not a self-referential link Symlink-identity checks run only where real symlinks exist (skipped on Windows git-bash copy-mode); the link/skip/no-data-loss checks run everywhere. Wired into studio-backend-ci.yml next to the static test. * Install psutil in backend CI so orphan-cleanup tests run The new orphan-cleanup tests import psutil for the process scan, but the Backend CI deps step installed studio.txt plus a fixed extras list that omits it, so the two tests failed with ModuleNotFoundError. Add psutil to both backend pytest dep steps (kept in shared shape), and guard the import with pytest.importorskip so a minimal env without psutil skips these tests instead of erroring. --------- Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat: add mlx public trainer api * test: cover mlx public trainer api * fix: preserve mlx epoch trainer configs * fix: pass mlx warmup ratio through config * fix: align mlx trainer dataset order * fix: keep mlx chat templates import-light * fix: infer mlx trainer context length * fix: mirror cuda mlx context defaults * fix: align mlx notebook trainer defaults * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: keep mlx public helpers import-light * refactor: reuse mlx optimizer normalization * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: address mlx review feedback * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: tighten mlx training argument parity * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: align mlx trainer eos default * Fix MLX trainer to accept DataCollatorForSeq2Seq and handle TokenizerWrapper in get_chat_template * Trim redundant docstrings on internal MLX helpers * MLX review fixes: Studio optimizer import-safe on non-MLX hosts, preserve explicit max_length, skip MLX tests before import * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * MLX review round 2: defer max_length to model context, optimizer alias fallback for older zoo, skip non-MLX test on missing GPU deps * MLX review round 3: keep chat_templates importable without torch on MLX * fix: preserve MLX trainer notebook shims * fix: ignore CUDA tokenizer moves on MLX * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: harden MLX trainer shims * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: unwrap MLX scheduler enum args * fix: coerce integral MLX epoch counts * fix: spoof CUDA compatibility APIs on MLX * fix: harden MLX notebook compatibility shims * MLX: add torch.cuda.mem_get_info to the compatibility shim Notebook memory cells call torch.cuda.mem_get_info()[0] directly (not gated by is_available), so on MLX it raises without a shim. Return (free, total) bytes from the MLX device stats, consistent with the other torch.cuda compat helpers, and add a matching assertion to the compat-API test. * MLX: use active memory for mem_get_info; fix BatchEncoding.to keyword device Address review on the MLX compatibility shim: - torch.cuda.mem_get_info() now derives free bytes from current active MLX memory instead of the peak high-water mark, so a capacity check stays accurate after a transient spike or a prior run. - BatchEncoding.to(device=...) passed by keyword no longer forwards a positional None alongside the keyword (which raised "multiple values for 'device'"), so non-CUDA keyword moves like .to(device="cpu") delegate correctly. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * MLX: accept preserve_dataset_order; stub RL trainers with a clear error Two fixes so unmigrated notebooks behave predictably on MLX (torch present): - preserve_dataset_order is a real MLXTrainingConfig field but was missing from the extra-argument allowlist, so passing it (as a config or trainer kwarg) could be rejected as unknown on a zoo without the field. Add it to _MLX_IMPLEMENTED_EXTRA_ARGUMENTS so the documented no-shuffle path is reachable. - GRPO/DPO/ORPO (and KTO/PPO/Reward) have no MLX trainer yet. Retarget the ones the installed trl exposes to a stub that raises a clear 'not supported on MLX' error instead of importing the real torch/CUDA trainer and crashing deep inside it. Only existing trainers are retargeted (no invented attributes), idempotent across re-imports. * MLX: make RL-trainer stubbing import-safe; back current-memory APIs with active memory Address review on the MLX shims: - The RL-trainer stub loop probed trl with getattr(_trl, name), which triggers trl's lazy trainer import and pulls torch -- that can crash import unsloth on a torch-free MLX install just to check existence. Decide what to stub from trl.__all__ + already-materialized attrs (vars) instead; never resolve the real trainer. All trl trainer names are in __all__, so they are still stubbed (even torch-free), and the probe no longer imports torch. - torch.cuda.memory_reserved / memory_allocated (the current, non-max APIs) were aliased to peak max_memory_reserved. Back them with current active MLX memory so cleanup / capacity checks see live usage; max_* keep the peak high-water mark. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * MLX: keep TRL's SFTConfig epoch default under the trl.SFTConfig alias Unmigrated notebooks import SFTConfig from trl, which the MLX build aliases to the public training-args class. TRL/HF SFTConfig defaults to num_train_epochs=3 (max_steps=-1); the native MLX config defaults to max_steps=60. So an SFTConfig built without an explicit length silently ran 60 MLX steps instead of TRL's 3 epochs under the alias. Alias trl.SFTConfig to a thin subclass that seeds the TRL epoch default only when neither max_steps nor num_train_epochs is given; explicit lengths pass through untouched, and the native public args class keeps its MLX default. Epoch mode is supported by the MLX trainer. * MLX CI: keep the GGUF reload smoke under the job timeout The RELOAD-GGUF-via-llama-cli step timed out at 300s. BF16 GGUF decode is CPU-bound on the macOS runner (~10s+/token), so generating 24 tokens landed right on the 300s cliff and killed the process. This step is a save/reload integrity smoke (it only needs a few chars of output), so the token count is incidental: generate 8 tokens with explicit threads and a small headroom on the subprocess timeout, all env-tunable (UNSLOTH_GGUF_RELOAD_N / _THREADS / _TIMEOUT). Cuts the reload well under the 25 minute job budget. * MLX: broaden trainer stubs, real peak-memory reset, fix shim tests Address review on the MLX public API: - The SFTConfig identity tests asserted trl.SFTConfig is UnslothTrainingArguments, but the alias now points at the _MLXSFTConfig subclass that preserves TRL's epoch default, so the MLX suite failed before testing the shim. Assert issubclass instead. - torch.cuda.reset_peak_memory_stats was a no-op, so max_memory_reserved kept earlier model-load peaks across a scoped run. Wire it to mx.reset_peak_memory with the same core/metal fallback used for the reads. - The unsupported-trainer stubs were a fixed list, so trainers outside it (a newer RLOOTrainer) still routed to the real torch trainer. Derive the set from trl.__all__ (every non-SFT *Trainer) so all non-SFT surfaces fail with a clear MLX message; names come from __all__ so trl is never resolved. - The non-MLX export smoke skipped only on missing bitsandbytes/triton; other absent GPU deps (numpy/torch/unsloth-zoo, or _gpu_init re-raising ImportError) made it fail on CPU hosts. Skip on any ImportError. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: keep MLX notebook compatibility minimal * MLX CI: force CPU + small context for the GGUF reload smoke The RELOAD-GGUF-via-llama-cli step timed out even at 8 tokens (>420s), so it is a fixed hang, not per-token cost: on the paravirtual macOS runner GPU llama.cpp's Metal backend stalls, and the gemma3 GGUF advertises a 32768 context that llama-cli would otherwise fully allocate. Run llama-cli CPU-only (-ngl 0) with a small context (-c 256); keep generation short. All env-tunable (UNSLOTH_GGUF_RELOAD_NGL / _CTX / _N / _THREADS / _TIMEOUT). Also print llama.cpp's partial stdout/stderr on timeout so a future hang is diagnosable instead of an opaque TimeoutExpired. * MLX CI: export the reload-smoke GGUF as q8_0, not bf16 The GGUF reload via llama-cli timed out on the runner even CPU-only with a tiny context and 8 tokens. Root cause is the format, not the flags: the smoke exported quantization_method='not_quantized', which maps to a bf16 GGUF, and llama.cpp's bf16 CPU decode is unusably slow on the paravirtual macOS runner. Export q8_0 (fast_quantized, the exporter default and what users deploy) instead -- llama.cpp has optimized q8_0 CPU kernels, so the fresh-process reload loads and generates in seconds. The reload stays CPU-only (-ngl 0) with a small context. * test: clear TRL shim before availability check --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* report a complete load once llama-server is healthy load_progress() derived its fraction purely from the llama-server's VmRSS over the GGUF shard total. With layers offloaded to VRAM (-ngl) the process releases the mmap'd weight pages after upload, so VmRSS sinks back well below the shard total: the fraction climbs toward ~1.0 during mmap, then collapses to a small value (~8%) once the weights are on the GPU. A fraction-driven progress bar therefore restarts and sticks there indefinitely even though the model is loaded and serving, which reads as a hang at "Starting model...". Once the server is healthy the load is complete by definition, so report fraction 1.0 (and bytes_loaded == bytes_total) in the ready phase regardless of resident set size. The VmRSS read is factored into _read_rss_bytes() with its original semantics preserved (0 on a missing VmRSS line, None when /proc is unavailable) so it can be unit-tested off Linux. Fixes #5740 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * stub heavy deps in the load-progress test and guard a valueless VmRSS Two review fixes: 1. The new test imported core.inference.llama_cpp at module top, which pulls in loggers/structlog/httpx and fails collection with ModuleNotFoundError in the lightweight backend test env when the file is run on its own. Stub loggers, structlog and httpx via sys.modules.setdefault before the import, mirroring test_llama_cpp_load_progress_matrix.py; setdefault keeps the real modules when installed. Verified the file now collects and passes with only pytest present. 2. Catch IndexError in _read_rss_bytes: a "VmRSS:" line with no value column would make line.split()[1] raise and crash a load-progress poll. Return None instead, with a test for the valueless line. * Hold load-progress high-water mark and explain a never-healthy load (#5740) load_progress() now holds a per-process VmRSS high-water mark, so the bar no longer regresses to ~8% when -ngl offloads the weights and frees the mmap pages mid-load. A live server that never returns 200 on /health now gets a specific error (context/VRAM too large, or a local proxy/VPN intercepting the loopback probe) instead of the generic invalid-GGUF/out-of-memory message. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Hakan Baysal <hakan.baysal@trmix.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
…False (#6786)
* fast_generate: clear error for vLLM-style inputs when fast_inference=False
When fast_inference=False, fast_generate falls back to HuggingFace
generate, and the wrapper already rejects vLLM-only usage (a
sampling_params or lora_request kwarg, or a string prompt). A vLLM prompt
dict ({'prompt':..., 'multi_modal_data':...}) or a SamplingParams passed
positionally slipped through and hit transformers.generate, raising a
cryptic 'SamplingParams object has no attribute update'. Detect both and
raise the same clear 'only supported with fast_inference=True' error.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fast_generate: also reject positional list of SamplingParams and list of vLLM prompt dicts
Address review feedback: the slow-mode guard missed SamplingParams passed inside a
positional list and a list of {"prompt": ...} dicts, both valid vLLM batched shapes
that leaked into transformers.generate. Fold the checks into small predicates and
extend the GPU-free test (now 7 reject + 3 pass).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test_fast_generate_slow_guard: expose assertions via a test_ function so pytest collects them
The assertions lived in run(), only called from __main__, so pytest reported no tests
collected and CI skipped the coverage. Rename to test_fast_generate_slow_guard; the
standalone script entrypoint still works.
* fast_generate: reject vLLM tokenized/embeds prompt dicts in the slow-mode guard
vLLM also accepts prompt dicts keyed by prompt_token_ids or prompt_embeds, not just
prompt/multi_modal_data. Those slipped past the slow-mode guard and fell through to
HuggingFace generate with a cryptic error. Recognize all vLLM prompt-dict keys and
add a TokensPrompt test case (now 8 reject + 3 pass).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fast_generate slow-mode guard: catch vLLM prompts= keyword form
vLLM's generate names its first argument `prompts`, so a slow-mode call
like fast_generate(prompts="hi") or prompts=[{"prompt": ...}] bypassed the
guard and leaked into HuggingFace generate as an unexpected kwarg. Check
kwargs["prompts"] with the same _is_vllm_prompt predicate and add two test
cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fast_generate slow-mode guard: reject vLLM tokenized prompt kwargs
vLLM's legacy call shape passes tokens as prompt_token_ids= (and prompt_embeds=),
which are not HuggingFace generate arguments. In slow mode these bypassed the
guard and leaked into HF generate as unexpected kwargs. Reject their presence
with the same tokenize-first message and add a test case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fast_generate slow-mode guard: treat prompts= as vLLM-only
prompts is a vLLM keyword, not a HuggingFace generate argument, so any value
passed as prompts= (including a bare token-id list, which _is_vllm_prompt
deliberately ignores for positional HF token ids) is a vLLM-style call. Reject
prompts= / prompt_token_ids= / prompt_embeds= on presence, and keep the
conservative _is_vllm_prompt check only for the positional arg.
* fast_generate slow-mode guard: reject vLLM prompt kwargs on presence
prompts / prompt_token_ids / prompt_embeds are vLLM-only keyword names that
HuggingFace generate does not accept, so a defaulted call like prompts=None
should raise the actionable slow-mode error instead of leaking a None kwarg
into HF generate. Check membership in kwargs rather than a non-None value.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
* replaced connect with start * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix * Studio: build the coding-agent command from the selected server The API keys panel showed a hardcoded `unsloth start claude`. `unsloth start` defaults to 127.0.0.1:8888 and only mints a key for a loopback server, so a non-default port or a tunnel/remote base would target the wrong server or fail to mint. Build the command from the panel base/key (and emit a key for non-loopback), matching the other snippets in the panel. * CLI: keep `unsloth connect` as a hidden alias for `unsloth start` Avoids breaking existing scripts and docs that still call `unsloth connect`. * Tests: stub _unstarted_cleanup in same-task disconnect test The test builds _SameTaskStreamingResponse via __new__, so set the attribute that __call__ now reads. * Match coding-agent command loopback check to the CLI 127.0.0.0/8 rule (#6613) * Keep unsloth_cli.commands.connect importable as a deprecated shim (#6613) * Format the new coding-agents panel strings and import per biome (#6613) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Drop the unsloth connect alias and shim; unsloth start is the only command (#6613) * Route unsloth connect to unsloth start as a hidden backward-compatible alias (#6613) * Forward unsloth run model-load flags to unsloth start (gguf-variant, context-length, load-in-4bit, tensor-parallel) (#6613) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Session-scope coding agent config in unsloth start Configure each agent for the current session instead of writing the Studio endpoint, key, and default model into the user's own config. Codex, OpenCode, OpenClaw, and Hermes get a private config relocated through their config-path env vars (CODEX_HOME, OPENCODE_CONFIG overlay, OPENCLAW_CONFIG_PATH plus OPENCLAW_STATE_DIR, HERMES_HOME). Claude Code suppresses the attribution header for the session via the CLAUDE_CODE_ATTRIBUTION_HEADER env var plus a --settings overlay, with no ~/.claude write. --launch uses an ephemeral temp dir removed after the agent exits; --no-launch uses a stable Unsloth-owned dir and prints the matching export lines. * Read relocated agent session config in Local Agent Guides CI The contract crosscheck and the openclaw/hermes patch helpers now read each agent's config from the relocated path printed by unsloth start --no-launch (CODEX_HOME, OPENCODE_CONFIG, OPENCLAW_CONFIG_PATH, HERMES_HOME) instead of fixed home paths. The Claude attribution A/B toggles the header for the session only (shipped-config HIT vs vanilla MISS) instead of editing ~/.claude/settings.json. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Skip the POSIX-only --no-launch parser test on Windows test_no_launch_output_is_parseable mirrors the #6547 bash CI parser, which greps export/unset lines and only runs on Linux/macOS runners. On Windows --no-launch prints PowerShell ($env: / Remove-Item), so the export-line assertion does not apply there. Cross-OS staging CI surfaced this. * Size Claude Code's auto-compact window to the loaded model's context Claude Code auto-compacts against its native (~600k token) window, so against a smaller local model it overflows the server's context (silent truncation) long before it compacts. Set CLAUDE_CODE_AUTO_COMPACT_WINDOW to the loaded model's real context length (the value codex/openclaw already get via model_context_window / contextWindow). Omitted when the model reports no context length. * Pin OpenCode/Hermes context window and set 90% compaction across agents Feed every agent the server-determined sequence length (the value /v1/models reports from runtime_context_length) and a ~90% compaction threshold. OpenCode: a custom-provider model with no limit defaults to context 0, which silently disables auto-compaction, so set limit.context/output and scale the compaction buffer to 10% of the window. Hermes: pin model.context_length (it otherwise falls back to a 256k default when the server's /v1/models omits the field) and set compression.threshold 0.9. Claude: add CLAUDE_AUTOCOMPACT_PCT_OVERRIDE=90 alongside the window. Codex (model_context_window) and OpenClaw (contextWindow) already carried the window and auto-manage off it. * Add `unsloth start pi` recipe Pi was the only agent without a built-in recipe, so the agent-guides CI hand-wrote ~/.pi/agent/models.json. Add a first-class `pi` command mirroring the others: - write_pi_config writes the session-scoped OpenAI-compatible provider config (key in the config, like openclaw/opencode). - pi() launches `pi --provider unsloth --model <id>` (Pi defaults to the google provider, so the provider/model are pinned on the command line) with HOME relocated for the session. Pi has no config-dir env var and resolves ~/.pi off $HOME, so HOME-scoping keeps the user's ~/.pi untouched. Migrate the agent-guides CI off the hand-written config onto the `unsloth start pi --no-launch` path (connection + file-edit), with a crosscheck for the provider api, so the documented recipe is exercised. * Harden unsloth start for Windows and WSL agent launches Address the Codex review on PR 6613: - write_pi_config now pins the loaded contextWindow and a sane maxTokens so Pi compacts instead of overflowing a small Studio context (it otherwise assumes its 128000 default), matching the other agents. - pi() sets USERPROFILE (and HOMEDRIVE/HOMEPATH when present) alongside HOME on native Windows, where Node resolves ~/.pi via USERPROFILE rather than HOME, so the session no longer reads or writes the user's real ~/.pi. - The WSLENV bridge flags path-valued vars with /p so a Windows npm shim under /mnt receives translated paths, while scalar vars (the numeric context window) pass through untranslated. WSLENV is deduped on the bare name. - _print_env prints the launch command with PowerShell-safe quoting so the inline --settings JSON survives copy-paste on native Windows --no-launch. Add tests for the WSLENV path flagging, PowerShell quoting, the Pi context window, and the Pi USERPROFILE relocation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Set CLAUDE_CODE_NO_FLICKER for the Claude session A local server streams in bursts, so Claude Code's full-screen TUI redraw flickers between tokens. Disable it for the session via CLAUDE_CODE_NO_FLICKER, alongside the other CLAUDE_CODE_* session env knobs. * Add a normalized --yolo flag routed to each agent's auto-approve mode It is easy to forget which agent spells "run tools without prompting" which way, so `unsloth start` now accepts all three spellings as one option (--yolo, --dangerously-skip-permissions, --dangerously-bypass-approvals-and-sandbox) and routes to the agent's own mechanism: - claude: --dangerously-skip-permissions - codex: --dangerously-bypass-approvals-and-sandbox - hermes: --yolo - pi: --approve (Pi's only approval gate is project trust) - opencode: a permission allow block in opencode.json (no CLI flag exists) - openclaw: tools.exec security=full / ask=off / host=gateway (no CLI flag exists) Because the option is parsed by `unsloth start`, the "wrong" spelling for an agent still routes correctly instead of leaking through to the agent and erroring. IS_SANDBOX is deliberately left unset for Claude so its root/sandbox safety gate still applies. Adds routing, cross-routing, and per-config tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix review findings: IPv6 loopback command, pi USERPROFILE under WSL, yolo guard From a 10-reviewer pass over the PR: - studio/frontend agent-command.ts: normalize bracketed IPv6 hosts. URL.hostname returns "[::1]" for http://[::1]:8888, which never matched the "::1" loopback checks, so the copied command embedded the placeholder API key for a local IPv6 server instead of the bare auto-minting command. Now [::1] is treated as loopback like the CLI's is_loopback_url, so the command matches the CLI contract. - pi(): also relocate USERPROFILE (and HOMEDRIVE/HOMEPATH) when running under WSL against a /mnt Windows shim, not just on native Windows. Windows Node resolves ~/.pi via USERPROFILE, and the WSLENV bridge translates the path, so pi no longer falls back to the user's real ~/.pi in that case. - _yolo_command_flags: use .get so a config-based agent (or a typo) yields no flag instead of a latent KeyError. Adds tests for the WSL pi USERPROFILE relocation, the yolo unmapped-agent guard, and that opencode/openclaw --yolo stays config-only (no argv flag). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix round-2 review findings: WSLENV /p upgrade, agent help text - _merge_wslenv now upgrades a user's pre-existing unflagged WSLENV entry (e.g. a bare HOME or USERPROFILE) to the path-translated form (HOME/p) instead of leaving it as-is, so a Windows agent shim under WSL receives the translated session path rather than the raw Linux path. - Generalize the `unsloth start` registration help to list all six agents (was only "Claude Code, Codex"). Adds a test for the WSLENV unflagged-entry upgrade. * Fix round-3 review findings: complete openclaw --yolo, refresh stale copy - openclaw --yolo now also writes the host approvals file (exec-approvals.json with defaults security=full / ask=off / askFallback=full) alongside the tools.exec config. OpenClaw gates tool execution on both layers (the stricter wins), so the config alone could still leave it prompting or denying. Mirrors `openclaw exec-policy preset yolo`. ask=off means nothing is ever prompted, so the runtime socket block is unnecessary. - Studio API panel copy: clarify that a local server auto-mints the key while a remote one embeds it in the command, and add pi to the swap hint. - Local Agent Guides CI: drop the stale "pi has no start.py recipe" note now that all six agents are driven via `unsloth start <agent> --no-launch`. Adds the openclaw approvals-file assertions and a no-yolo openclaw test. * start: parse claude --version with a regex so a format change does not drop optimization flags * start: offer to install a missing agent (prompt then run its install command) * start: auto-start a Studio server for --model when none is running, and stop it on exit * inference: surface an actionable message when llama-server cannot compile a tool grammar * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix review findings: kill the auto-started server tree on Windows; apply the tool-grammar message to the OpenAI passthrough too * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * start: split --model org/repo:variant so a running session is not evicted `unsloth start <agent> --model org/repo:QUANT` failed against an already-running Studio server and, worse, killed whatever model another session had loaded. /v1/models lists a loaded GGUF under its bare repo id (e.g. unsloth/Qwen3-1.7B-GGUF), so _resolve_model never matched the `:QUANT`-suffixed request. It then POSTed /api/inference/load with model_path=org/repo:QUANT, which (a) Hugging Face rejects ("Repo id must use alphanumeric chars, '-', '_' or '.'") and (b) evicts the model the other session was using, so a second 'unsloth start' in a new tmux/terminal tore down the first. Re-running the command then attached to the now-empty server, which is why it 'worked the second time'. Mirror the org/repo:QUANT -> org/repo + --gguf-variant QUANT shorthand that 'unsloth run' and llama.cpp already accept, splitting it in _connect before we match or serve. Matching now resolves against the loaded bare repo id (no spurious reload, no eviction), and any real load uses a valid repo id plus gguf_variant. An explicit --gguf-variant still wins; local paths and Windows drive letters pass through untouched. The auto-serve path likewise spawns 'unsloth run --model org/repo --gguf-variant QUANT'. * start: harden auth-key handling, codex teardown, and CI transcript redaction Three review findings: 1. CI could leak a live key. agent-guides-drive.sh printed the raw 'unsloth start --no-launch' transcript (which carries export UNSLOTH_API_KEY / ANTHROPIC_AUTH_TOKEN lines) to the Actions log on both the failure path and the success path before redact() ran. Add cat_redacted() and use it for those two prints, so the key is scrubbed on the way to the log while the on-disk file stays intact for the env parsing that follows. 2. Outages masqueraded as bad keys. _key_accepted caught a broad Exception and returned False, so a 5xx or timeout while checking a cached key looked like a rejection: it discarded a good key and minted extra ones (local) or reported 'no saved key' (remote). Only treat HTTP 401/403 as a rejection; let other errors propagate so a real outage surfaces. 3. Codex preflight could leave the auto-started server up. _require_gguf_for_codex runs after _connect may have auto-started Studio but before _run installs its teardown finally, so a preflight rejection (e.g. a transformers-backend model) left the server holding the port/GPU until the atexit backstop. Tear it down explicitly at the point of failure. Tests: a 5xx on a saved key surfaces without minting; a non-GGUF codex preflight tears down the auto-served server. * start: fix IPv6/portless studio URLs, Pi config-dir isolation, and Pi install recipe Four review findings: 1. Pi ignored the session config when PI_CODING_AGENT_DIR was already set. Pi's getAgentDir() reads process.env.PI_CODING_AGENT_DIR before falling back to $HOME/.pi/agent, so a value inherited from the user's shell sent Pi to their real config and skipped our provider/key (the HOME relocation alone was not enough). Pin PI_CODING_AGENT_DIR at the session's .pi/agent dir; it is path-valued so the WSL bridge translates it automatically. 2. Pi install hint dropped Pi's documented --ignore-scripts. Pi's README installs with 'npm install -g --ignore-scripts @earendil-works/pi-coding-agent' and notes it needs no install scripts, so accepting the prompt now follows that safe recipe. 3. Auto-start ignored a portless UNSLOTH_STUDIO_URL. unsloth run binds to 'parsed.port or 8888', so http://127.0.0.1 launched the child on 8888 but the health poll (and the returned base) still used port 80, stalling until the startup timeout. Normalize the base to host:8888 (IPv6-safe) before starting and polling. 4. API-panel command mistook IPv6 loopback for the bare default. The bare 'unsloth start' only probes 127.0.0.1:8888 on the IPv4 stack, so http://[::1]:8888 must carry an explicit UNSLOTH_STUDIO_URL. Drop ::1 from the bare-default host set while keeping it a loopback host (URL emitted, no key needed). Tests: PI_CODING_AGENT_DIR is set to the session dir; _effective_base normalizes portless/IPv6 bases; a portless UNSLOTH_STUDIO_URL auto-serves on :8888. * start: apply fresh-review findings across CLI, CI, and the API-panel command From a fresh multi-reviewer pass over the merged head plus the latest Codex bot review: 1. Load knobs now always consult the server. _resolve_model matched on model id alone, so --gguf-variant / --context-length / --no-load-in-4bit / --tensor-parallel were silently ignored whenever the id was already loaded (asking for UD-Q4_K_XL kept a Q8_0 serving). With any explicit knob the CLI defers to /api/inference/load, whose already-loaded dedup answers without reloading when variant and settings match, so a second session running the same command still attaches without evicting the first. 2. OpenCode --yolo and the session model pin now ride in OPENCODE_CONFIG_CONTENT. A project's own opencode.json outranks OPENCODE_CONFIG, so a repo config could silently override the session model and the --yolo permission block; OPENCODE_CONFIG_CONTENT outranks project config. The API key stays in the private file, never in printed env. 3. The --no-launch recipe's last line is a self-contained one-liner (inline VAR=value assignments before the command, conflicting vars blanked). People copy just the last line, and a bare codex/claude there ran against the user's real ~/.codex or Anthropic credentials with zero isolation, e.g. inheriting a pre-existing damaged ~/.codex state DB and blaming the recipe. The CI drive script scrubs the key from the one 'invoking:' echo this adds. 4. The auto-serve log is 0600 and the parent handle is closed. It sat world-readable in the shared tempdir under a predictable name while carrying the minted sk-unsloth- key from the unsloth run banner. 5. _key_accepted fails with a clean message on outages. Non-auth errors (5xx, network, timeout) surfaced as a raw traceback; 401/403 still mean a rejected key. 6. _effective_base strips URL paths, and https loopback targets never auto-serve. http://127.0.0.1:8888/studio polled /studio/api/health (404) and https://127.0.0.1 polled the wrong scheme, both spinning until the 15-minute startup timeout. 7. API-panel command: only literal 127.0.0.1:8888 earns the bare command. localhost can resolve to ::1, which the bare CLI never probes, so it keeps UNSLOTH_STUDIO_URL. 8. CI artifact sweep covers redacted-configs/ and agent-workdir/, not just logs/. Tests: 125 CLI tests pass (new coverage for each fix), 156 backend tests pass, ruff clean. Adds an unsloth connect alias regression test. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * start: hand Pi a clean screen at launch Pi paints inline from wherever the cursor sits: its first render assumes a clean screen instead of clearing or entering the alternate screen itself (current Pi never emits a clear at startup). Launched under unsloth start, that left the session starting mid-scroll beneath the connection output. Clear the screen (click.clear, cross-platform, no-op without a TTY) right before the Studio banner so Pi opens exactly one line down on a clean viewport. Launch path only: --no-launch recipes and piped output are never wiped, and alternate-screen agents are left alone. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * start: auto-override hermes' 64K context floor for small model windows Hermes refuses to initialize when the served model's context window is under 64,000 tokens, and a second copy of the same check rejects the compression model mid-session. write_hermes_config previously pinned the real window, so any small local model (e.g. 40,960) failed at startup with manual config.yaml instructions. For windows below the floor the recipe now claims 65,536 in model.context_length, scales compression.threshold so compaction still fires at 90% of the real window, and sets auxiliary.compression.context_length to cover the mid-session check. Windows at or above the floor keep the exact previous behavior. * ci: install pi with --ignore-scripts, matching the start.py hint The pi cell predates the pi recipe in start.py and still installed the package with lifecycle scripts enabled, so CI stopped exercising the exact command users are prompted to run. npm_retry now passes extra flags through, the pi branch mirrors the install hint verbatim, and the stale no-recipe comment is refreshed. * ci: fail loudly when a relocation var is missing from connect output The empty-string guards ran after appending /config.toml or /config.yaml, so they could never fire: crosscheck_contract silently skipped its contract checks and patch_hermes_tools died on the root path with a bare traceback. Check the raw variable first and guide_fail with the real cause. * staging: 6613 round 6 (https elision, no-launch home reuse, auto-start key fallback) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: shimmyshimmer <107991372+shimmyshimmer@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
… passthrough (#6801)
* inference: add passthrough tool-call healing core (heal_gate, heal_openai_message, StreamToolCallHealer, nudge helpers)
Small GGUF models often emit tool calls as text (<tool_call>{...}</tool_call>,
Gemma <|tool_call>, <function=> XML) instead of structured tool_calls. Studio's
enable-tools loop already heals these, but the client-tool passthrough
(unsloth run --disable-tools, unsloth start agents) relays them verbatim, so
the agent sees prose and the turn dies.
This module is the shared response-side repair layer the passthrough routes
will call: promote parsed text-form calls to structured calls, but only for
function names the client actually declared; coerce arguments through the same
canonical-key healing as the tool loop; never touch the upstream request body
(llama-server KV/slot reuse stays byte-identical). StreamToolCallHealer is the
streaming buffer-and-repair state machine: prose forwards immediately, only a
partial-signal tail or a suspected tool block is held, false alarms flush
verbatim, and a 64 KiB bound caps memory. nudge_should_retry/nudge_messages
support an opt-in single-retry nudge for non-streaming routes (wired later).
Kill-switch: UNSLOTH_DISABLE_TOOL_CALL_HEALING=1. Reuses
core/tool_healing.parse_tool_calls_from_text, strip_tool_call_markup, and
tool_loop_controller.coerce_tool_arguments unchanged.
* inference: heal text-form tool calls on the OpenAI and Responses passthrough
Wire the passthrough healing core into /v1/chat/completions and /v1/responses,
default ON whenever the request declares client tools:
Non-streaming: heal_openai_message runs inside the existing response-mutation
loop; a promoted call flips finish_reason to tool_calls and nulls the content,
and the verbatim-bytes fast path still applies when nothing was healed.
/v1/responses non-streaming inherits this through openai_chat_completions.
Streaming: a StreamToolCallHealer per stream. Ordinary prose relays
byte-for-byte (a fast path keeps upstream bytes when the healer passes a chunk
through whole); once a tool signal appears, content is held, and at the
finish/[DONE] boundary either synthetic delta.tool_calls chunks replace the
markup (finish_reason rewritten to tool_calls, including the synthetic-finish
path) or a false alarm flushes the held text verbatim. Structured upstream
deltas put the healer to sleep after flushing anything held, so grammar-mode
responses stay byte-identical. The Responses stream feeds healed calls through
the same per-call state machinery as structured deltas (indexes live in a
disjoint range so a healed call can never merge into a structured call's
state), and the visible/reasoning split runs first so reasoning text is never
promoted. parallel_tool_calls=false caps healed calls on every path.
The upstream request body is never touched and healing issues no extra
generation, so llama-server slot/KV-cache reuse is unchanged. Opt-out per
request with auto_heal_tool_calls=false (Responses reads it from the
extra-body); requests without tools relay verbatim.
* inference: heal text-form tool calls on the Anthropic /v1/messages passthrough
Streaming: AnthropicPassthroughEmitter.enable_healing(allowed_tools) routes
content deltas through the shared StreamToolCallHealer. A promoted call closes
any open text block (only the safe prose prefix ever streamed into it), opens a
synthetic tool_use block with a fresh toolu_* id, carries one input_json_delta,
and closes; finish() then forces stop_reason to tool_use unless a truncation
(max_tokens) wins. Structured upstream deltas flush anything held and put the
healer to sleep, so grammar-mode responses are untouched, as is every stream
where enable_healing is never called (Studio's own loop, no-tools requests).
disable_parallel_tool_use caps healed calls too.
Non-streaming: the OpenAI message dict is healed BEFORE block building, so the
existing tool_use promotion loop and stop_reason line treat promoted calls
exactly like native ones (finish_reason length still maps to max_tokens). The
legacy tool-XML strip still runs on remaining text, so opted-out requests keep
today's cleanup behavior byte-for-byte.
auto_heal_tool_calls is now a typed field on AnthropicMessagesRequest
(default True, mirroring Chat Completions) and threads into both passthrough
calls. Healing never touches the upstream request body.
* inference: opt-in single-retry tool-call nudge on the non-streaming passthrough
When the model clearly tried to call a tool (a tool signal in the text) but
healing produced nothing usable, re-ask once: the retry body is the original
body plus an assistant turn (the model's own failed text) and a short user
nudge naming the declared tools. The prompt prefix stays byte-identical, so
llama-server reuses the slot's KV cache and only the two-message suffix is
prefilled. The retry replaces the original response only when it actually
yields a promotable or structured call; on any error or still-garbage output
the original response is returned unchanged. Exactly one retry, non-streaming
OpenAI and Anthropic passthroughs only (a stream has already emitted bytes).
OPT-IN per user decision: nudge_tool_calls=true per request (typed on both
ChatCompletionRequest and AnthropicMessagesRequest, lifted from the Responses
extra-body), or UNSLOTH_TOOL_CALL_NUDGE=1 to flip the process default.
auto_heal_tool_calls=false disables healing AND the nudge.
Also align the non-streaming heal on allow_incomplete=True: the response is
final, so a trailing unclosed tool block is a model failure worth repairing,
matching the enable-tools loop's drain semantics.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* inference: never assume the upstream response shape in the nudge helpers
llama-server error bodies can carry message: null (or no choices at all), and
_last_assistant_text / response_has_promotable_calls / nudge_should_retry
called .get() on the message without a dict check, so a malformed upstream
response raised an AttributeError the surrounding except tuples did not catch,
failing the request instead of degrading to 'nothing to heal'. Route the shape
probing through one _first_choice_message helper that returns None for any
non-dict message, and add a parametrized test over the malformed shapes.
* inference: constrain healing by tool_choice, preserve length finish_reason, keep healed event order in Responses streams
Three review findings on the passthrough healer:
- heal_gate now honors the request's tool_choice: "none" disables healing
outright and a forced function narrows the promotion allowlist to that
one function, so healing can never contradict the request's tool-choice
constraint. Wired through the OpenAI chat (stream and non-stream),
Responses, and Anthropic (converted shape) passthroughs.
- The OpenAI non-streaming heal only upgrades finish_reason "stop" to
"tool_calls"; a truncated generation keeps "length" (the healed call
stays attached) matching the streaming and Anthropic paths.
- The Responses stream emits healer events in order instead of collapsing
all text ahead of the healed calls, so text after a healed call no longer
jumps ahead of the function_call item and output indexes are claimed in
the order the model produced them.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* inference: all-or-nothing promotion when a response mixes declared and undeclared text-form calls
Promoting a subset used to strip ALL tool markup from the content, which
silently deleted the text of any call naming an undeclared tool. The heal
now declines entirely when any parsed call is unpromotable, so the whole
message relays verbatim (pre-PR behavior) and no bytes are ever lost. In
streaming, a declared call that completed before an undeclared one arrived
is already emitted; the late undeclared markup still flushes as raw text.
The nudge helpers mirror the same contract via a shared predicate.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* tests: wrap long lines in the Responses healing tests to the project style
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* inference: span-exact healing, disjoint healed stream indexes, per-call Responses message items, allowlisted nudge acceptance
Four review findings on the passthrough healer:
- parse_tool_calls_from_text gains an optional with_spans return so healing
removes EXACTLY the promoted calls' markup. This supersedes the previous
all-or-nothing rule: declared calls promote and every unpromoted byte
(undeclared calls, unparseable closed blocks, suppressed alternate
formats such as a <function=...> block after a JSON call) relays as text.
The stream healer also processes one block per pass, so text between two
healed calls keeps its document position instead of trailing them.
- The OpenAI chat stream shifts native tool-call delta indexes past any
already-emitted healed calls; clients merge deltas by index, so a healed
call and a later native call can no longer merge into one.
- A healed call in the Responses stream closes the open message item and
trailing text opens a fresh one with a later output index, matching the
native stream shape; response.completed snapshots every message item
with its own text.
- The nudge retry only replaces the original response when the retry's
structured call names a DECLARED tool; a hallucinated undeclared call is
not an improvement.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: stop the heal path folding trailing prose into a closed function call
parse_tool_calls_from_text(allow_incomplete=True) cut a <function=...> body only
at an end-anchored </function>, so a fully closed call followed by trailing prose
(<function=..>..</parameter></function> words) folded </parameter></function> and
the prose into the tool argument and deleted the prose from visible content. The
strict path (allow_incomplete=False) already cut at the real </function> via rfind.
Do the same in both modes: trim the body at the real </function> when present and
end the removal span there, falling back to the end-anchored strip and body_end
only when the call is genuinely truncated. Add a regression test.
* inference: one shared single-call budget for healed and native calls
Codex round 5: the parallel-call caps counted healed and native calls
separately, so a healed text-form call followed by a native structured
delta double-emitted on all three streaming surfaces when the client
disabled parallel calls.
- OpenAI SSE: once a healed call went out with parallel_tool_calls
false, native tool_call deltas are dropped instead of index-shifted.
- Anthropic emitter: native deltas skip block allocation when the
healed-plus-native count already filled the single slot, and healed
emission counts open native states too.
- Responses stream: native deltas that survived the chunk-level cap are
skipped once a healed call claimed the slot.
Also adds a span assertion for the closed-</function> trailing-prose
parse fixed in the previous commit.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: relay undeclared text-form calls as text on Anthropic non-streaming
heal_openai_message promotes only declared text-form tool calls and
span-trims just their markup, deliberately leaving every unpromoted byte
(undeclared text-form calls included) in the content to relay as text.
The Anthropic non-streaming builder then ran a blanket _TOOL_XML_RE strip
over that content unconditionally, deleting the undeclared block before
building the text part, so Anthropic clients silently lost a call the
OpenAI non-streaming path preserves. The strip was harmless when healing
was all-or-nothing but became data loss once healing turned span-exact.
Gate the legacy strip on whether healing promoted a call, matching the
OpenAI passthrough and the intent already stated in the comment above.
Add a route-level regression test for the mixed declared+undeclared case.
* inference: require fully declared nudge retries; keep unpromoted Anthropic text
Codex round 6, two findings:
- response_has_promotable_calls accepted a nudge retry when any one
structured call named a declared tool, so a mixed retry (hallucinated
undeclared call plus a declared one) replaced the original and the
caller forwarded the undeclared call, or with parallel_tool_calls
false could keep only it. All structured retry calls must be declared.
- The Anthropic non-streaming builder still ran the legacy _TOOL_XML_RE
strip after span-exact healing, deleting undeclared or malformed call
text that healing deliberately preserved. The legacy strip now runs
only when healing is off (no declared tools, or opted out), matching
the OpenAI passthrough.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* inference: keep unpromoted Anthropic text whenever healing is active
The previous commit skipped the legacy strip only when a call was
actually promoted, so an undeclared-only (or malformed-only) response
was still silently emptied: exactly the dead-turn shape this path
exists to fix, and inconsistent with the OpenAI passthrough, which
relays those bytes verbatim. Gate the strip on healing being active
instead; opt-out and no-tools requests keep the legacy strip.
* Fix schema-aware tool healing for PR #6801
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix passthrough healing ordering for PR #6801
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix stream finish ordering for PR #6801
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Co-authored-by: wasimysaid <112766706+wasimysaid@users.noreply.github.com>
…d source parity (#6767) * Studio: expose full compressed-tensors scheme set in an export formats dropdown * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: multi-select export formats, portable torchao FP8/INT8, GGUF LoRA, source parity Export page overhaul on top of the formats dropdown: - Unify merged precision into one sorted multi-select list (16-bit first, then 8-bit, then 4-bit). Drop "vLLM" from labels, add INT8 (W8A8), INT8 (W8A16), INT4 (W4A16), MXFP4, MXFP8. Quick formats render as toggle pills; the rest live in a multi-select "More formats" dropdown, so several formats export in one run. - Add a portable torchao FP8/INT8 save path (Float8WeightOnlyConfig / Int8WeightOnlyConfig) that needs no NVIDIA GPU to produce and loads in vLLM. FP8 serializes to safetensors, INT8 to .bin. Wired into save_pretrained_merged and push_to_hub_merged via a TORCHAO_EXPORT_SCHEMES registry and _unsloth_save_torchao, parallel to the compressed-tensors path. - Hide NVIDIA-only compressed-tensors formats when no NVIDIA GPU is present; keep 16-bit and portable FP8/INT8. The backend also rejects a compressed request on non-NVIDIA hardware so it stays authoritative. - Relax merged export to non-PEFT models so Local Model and Hugging Face sources get the same 16-bit / compressed / portable options. - GGUF: send the whole quant list in one call (merge once, quantize many). - LoRA: add a GGUF adapter option (convert_lora_to_gguf.py) with an outtype select (f16/bf16/f32/q8_0/auto), alongside the safetensors adapter. - Thread the new fields through models, routes, orchestrator, and worker; extend the export tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: gate export by accelerator with a torch-aware reason; fix export save dir naming Export runs through Unsloth, which requires a compute accelerator (NVIDIA/AMD/Intel GPU or Apple MLX) and has no CPU code path, so a bare-CPU host cannot export even with PyTorch installed. Add export_capability() in utils/hardware that reports export_supported plus a precise reason so the UI stops showing a generic "no GPU": - pytorch_not_installed: a --no-torch install (even a physical GPU is unusable) - no_accelerator: PyTorch present but no supported accelerator (bare CPU) - mlx_unavailable: Apple Silicon where the MLX stack is missing or too old Expose the fields on /api/system/hardware and /api/system, and guard the mutating export routes (load-checkpoint, export/merged|base|gguf|lora) with HTTP 400 and the reason, leaving read-only endpoints usable so the Export page still renders. Make core/export/export.py import without PyTorch and without a usable accelerator (the Unsloth import is caught) so the export worker degrades to a clear message instead of crashing at import. Frontend: keep /export reachable on chat-only hosts and gray out the method and format options with the backend reason (Alert plus disabled MethodPicker) instead of silently redirecting to /chat, so users see why export is unavailable. Also fix the export save directory producing "model/null" for Local Model and Hugging Face sources that have no run/checkpoint, naming the folder from the model id. * CI: validate Studio export capability gating on Linux, Windows and macOS Add a small pytest matrix that runs studio/backend/tests/test_export_capability.py on ubuntu-latest, windows-latest and macos-latest. It confirms, on each real OS, that hardware.export_capability() reports the right decision and reason (pytorch_not_installed, no_accelerator, or mlx_unavailable) and that the export backend imports without PyTorch and degrades to a clear message instead of crashing. Hosted runners have no GPU/MLX, so this covers the "export unavailable, here is why" path a Mac/Windows user without an accelerator sees; a real accelerator export is validated separately. The job installs only a CPU PyTorch plus the backend import deps (no unsloth, triton, or llama.cpp), so it runs in seconds with no GPU. * Studio export: address Codex review (source-aware gating, GGUF LoRA token/MLX/guard) Frontend (export-page): - Gate LoRA and quantized-model restrictions on the active source. isAdapter / isQuantized come from the selected checkpoint; in Local Model / Hugging Face ("model") source mode they were stale, so LoRA stayed wrongly enabled for a direct base model (backend then rejects "No adapter to export") and a stale "quantized" flag disabled every method for an unrelated, exportable model. Add effectiveIsAdapter / effectiveIsQuantized (false outside checkpoint mode) and use them in the method-reset effect and the MethodPicker disabled state. - Hide the GGUF LoRA option on a macOS/MLX host (the backend rejects GGUF LoRA on MLX), so users no longer pick it, wait through the load, and always fail. Disable the "GGUF adapter" button on a Mac host and never send loraGguf there. Backend (core/export/export.py): - Pass the HF token into the GGUF LoRA conversion (save_pretrained_gguf), so a gated/private base model's config fetch in convert_lora_to_gguf.py is authenticated; without it the load can succeed but the conversion fails. - Guard the save_pretrained_gguf capability check with getattr so an older Unsloth model that lacks the method returns the clean "not supported" message instead of an AttributeError that surfaces as a generic 500. * Studio export: address 2nd Codex review (CI index, empty merged, test import) - studio-export-capability-ci.yml: add --extra-index-url https://pypi.org/simple to the torch install so torch's transitive deps still resolve; --index-url alone replaces PyPI with only the CPU wheel index, which does not serve all of them. - export-page handleStart: reject an empty merged selection (mirrors canExport), so clicking the panel's Start button with every precision pill deselected no longer submits mergedSelections: [] and launches an unintended default 16-bit export. - test_export_imatrix_compressed: the torchao-registry test now reads unsloth/save.py as text (like the other ast/string checks) instead of `import unsloth.save`, which raised ModuleNotFoundError in the CPU studio-backend suite that has no unsloth installed. * Studio export: make comments succinct across the export changes * Studio export: use load token for local GGUF LoRA export of gated bases * Studio export: harden portable torchao path and gate multi-format Hub push torchao (_unsloth_save_torchao): - merge to an isolated temp staging dir so a co-selected 16-bit output at save_directory is not deleted - narrow VLM detection to vision_config / ForVisionText2Text so T5/BART/Whisper are not misrouted - forward trust_remote_code (from auto_map) to the reload so custom-code models export Export UI: - hide portable torchao formats on macOS/MLX (backend rejects quantized export there) - restrict a Hub merged export to a single format (each writes to the repo root) * Studio export: torchao tokenizer remote-code + XPU offload, scale GGUF timeout torchao (_unsloth_save_torchao): - honor auto_map in the staged tokenizer/processor configs (not just model.config) when deriving trust_remote_code, so custom-code tokenizers reload after the merge - offload single-device XPU models to CPU (and empty the XPU cache) before the reload, matching the CUDA path, so an Intel GPU that fits the model once does not OOM on the second copy Export orchestrator: - scale the GGUF wait timeout by the number of requested quants so a multi-quant list export of a large model does not time out at a flat 3600s * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio export: show portable torchao formats only on non-NVIDIA (CPU) hosts Portable torchao FP8/INT8 is the fallback for hosts without the NVIDIA compressed-tensors path. On an NVIDIA GPU the compressed-tensors FP8/FP4/INT formats are the intended path (llm-compressor auto-installs), so hide the portable duplicates there; keep them on CPU / non-NVIDIA hosts and continue hiding them on macOS/MLX. * Studio export: report all output folders and the exported formats - Multi-format merged export now collects every sibling output directory (one per selected precision) instead of only the last; the success banner lists them all. - Show the selected precision formats in the run summary (a Formats row, like GGUF Quantizations), so the panel says what is being exported rather than just 'Merged Model'. - Persist the selected formats in the run summary and seed them on mount, so navigating away and back (or toggling the export method) restores the selection instead of resetting to 16-bit. * Studio export: list all output formats, add GGUF LoRA target, default Q8_0, auto-select newest checkpoint - Progress/summary panel now shows a Formats row with the selected merged formats, and the success banner lists every output folder a multi-format merged run creates (one line per format) instead of only the last one. - Merged format selection is seeded from the active run, so navigating away and back (or switching method cards) no longer resets it to 16-bit. - GGUF / Llama.cpp now offers an Export target toggle (Full model or LoRA adapter) for adapter checkpoints, reusing the LoRA GGUF export path. - Removed the Auto GGUF LoRA output type and defaulted to Q8_0 in the UI, the request model, and the backend defaults; the outtype list is now Q8_0/F16/BF16/F32. Core save.py still accepts auto for external callers. - When a finetune has no checkpoint selected, auto-select the newest one. * Studio torchao export: robust reload class + optional VLM import Two fixes to the portable torchao FP8/INT8 export reload, from review of the narrowed VLM detection: - Encoder-decoder seq2seq checkpoints (T5/BART/Whisper) are not causal LMs. With the narrowed is_vlm test they now correctly skip the image-text class, but fell through to AutoModelForCausalLM and failed to reload after the merge. Reload them with their own architecture class from the config instead. - AutoModelForImageTextToText was imported unconditionally at the top of the torchao path, so on Transformers builds without that class the import aborted every torchao export (even text-only). Import it lazily only for a VLM, with the AutoModelForVision2Seq fallback used elsewhere in Unsloth. * Studio: enable FP8/FP4 compressed export for newer-transformers models The shipped llm-compressor 0.10.x pins transformers<=4.57.6, so FP8/FP4 export failed for models needing a transformers 5.x sidecar (Qwen3.5, Gemma-4, Qwen3-Next): the quantization subprocess crashed importing the removed TORCH_INIT_FUNCTIONS. Run the quantization against a dedicated llm-compressor-main "shadow": a --target package dir (transformers 5.10.2 + llm-compressor main + compressed-tensors) layered over the existing torch. It installs --no-deps so torch is never touched (works on any Studio torch build), is provisioned lazily and fingerprint-cached, and can be turned off with UNSLOTH_DISABLE_LLMCOMPRESSOR_MAIN. - transformers_version.py: provision + validate .venv_llmcompressor. - export.py: route all compressed exports through the shadow when available; else keep the workspace 0.10.x path and fail fast past its transformers ceiling. - save.py: launch _compressed_quantize.py with a clean PYTHONPATH = shadow. - _compressed_quantize.py: skip linear_attn / vision tower / MTP modules (matches the RedHatAI and NVIDIA reference quants, and is required by the grouped schemes). Verified all four schemes (fp8, w8a8, w4a16, mxfp4) on Qwen3.5-9B and Llama-3.2-1B, and fp8 on Gemma-4, end to end through Studio. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix GGUF LoRA export tests * Fix export CI expectations * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: wasimysaid <112766706+wasimysaid@users.noreply.github.com> Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
…ls (#6835) * Studio: flush passthrough stream headers before upstream prefill stalls * Studio: clean up delayed passthrough send failures * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: close passthrough preheader cleanup gaps * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: retry delayed passthrough overflow truncation * Studio: close completed passthrough send responses * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
…ng (#6829) * Fix TrainingArguments silently disabling unsloth gradient checkpointing * Cover loaded adapters and preserve explicit None in GC restore * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
… fit budget (#6718) --------- Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>
* Fix external drive custom folder selection * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update studio/backend/tests/test_linux_external_media_paths.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep legacy media scan validation strict * Apply sensitive-dir denylist to legacy folder browser for PR #6799 The legacy /api/models browse endpoint gained the new /run/media mount roots in its allowlist but not the credential/config guard that scan-folder registration and the Hub browser already enforce. Filter sensitive names during enumeration and reject them in _resolve_browse_target so .ssh, .aws, .config, etc. under allowlisted roots stay unbrowseable, matching the Hub browser. Add a public contains_sensitive_path_component helper and cover the legacy resolver with a regression test. * Trim redundant comments in PR #6799 changes * Skip sensitive Linux media roots * Reject sensitive dirs at exact browse roots for PR #6799 Both _resolve_browse_target functions only checked contains_sensitive_path_component while walking descendant parts, so requesting an allowlisted root itself (empty relative path) returned it unchecked. A pre-existing scan-folder row under ~/.ssh, ~/.aws, ~/.config, etc. (registerable before the denylist was added) is re-added to the allowlist on upgrade and could then be browsed. Check the resolved target once before returning in both the legacy and Hub browsers, and cover the root case in both test suites. * fix: avoid unused path helper reexports * fix: import sensitive path helpers directly * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: imagineer99 <samleejackson0@gmail.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* feat(studio): expose opt-in MCP control plane * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden Studio MCP tools: byte-safe auth, page clamping, forward export/checkpoint fields Follow-up hardening on the opt-in MCP control plane. All changes are additive and backwards compatible. - BearerTokenMiddleware now compares the Authorization header on raw bytes. A non-ASCII bearer value previously reached str-based hmac.compare_digest, which raises TypeError and surfaced as a 500 instead of a clean 401. The constructor also rejects an empty or whitespace-only token so an empty token can never match an empty "Bearer " header. - MCP tools call the route functions directly, which skips FastAPI Query validation. list_training_runs and get_recipe_job_dataset now clamp limit and offset to the same bounds the HTTP routes enforce (a negative SQLite LIMIT otherwise means "no limit"). - export_gguf forwards hf_token (the backend rejects a Hub upload without it), accepts a list of quantization methods, and exposes imatrix / imatrix_path so the IQ low-bit quants are reachable. - load_checkpoint forwards hf_token and approved_remote_code_fingerprint so gated checkpoints and the remote-code approval retry work. Its docstring is corrected: the export backend coexists with training and inference rather than freeing GPU work. - start_training passes via_api_key=False explicitly instead of relying on the unfilled Depends default. Tests: add coverage for non-ASCII and empty-token auth, the correct-token pass through, non-http scope pass through, pagination clamping, and the forwarded export/checkpoint fields. * Harden Studio MCP: cap /mcp request bodies, reject unusable tokens, fix docs Follow-up hardening from a full review pass. All changes are additive and backwards compatible. - Add "/mcp" to _BODY_PROTECTED_PREFIXES so MaxBodyMiddleware enforces the same request-body cap it already applies to every other write endpoint (/api/train, /api/export, /api/data-recipe, ...). The MCP endpoint accepts authenticated POST tool-call bodies; without this an authenticated client could send an unbounded body. The middleware only buffers the request body (not the SSE response), so streaming is unaffected, and the 500MB default cap never affects a real JSON-RPC tool call (verified live). - Reject a non-ASCII UNSLOTH_STUDIO_MCP_TOKEN at construction. HTTP header values are ASCII, so a non-ASCII token cannot be sent by a standard client and would silently lock out the endpoint; fail fast instead. - MCP.md: document the canonical /mcp/ endpoint and note that /mcp redirects to it, so clients that do not follow redirected POSTs still connect. Tests: add non-ASCII token rejection coverage. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten MCP server comments --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothshared@gmail.com>
… (#7194)
* fix(tokenizer): check for tokenizer.model after saving it, not before
`fix_sentencepiece_tokenizer` creates its temporary directory, then returns
early unless that directory already contains a tokenizer.model:
if not os.path.exists(temporary_location):
os.makedirs(temporary_location) # fresh, empty
if not os.path.isfile(f"{temporary_location}/tokenizer.model"):
return new_tokenizer # always true
old_tokenizer.save_pretrained(temporary_location) # writes that file
The file only appears on the line after the check, so the guard is always
true and the body never runs. Nothing else writes that path either --
`convert_to_fast_tokenizer` saves into a per-name subdirectory, not
`{temporary_location}/tokenizer.model`.
Both call sites are in `get_chat_template` and are commented "Must fix the
sentence piece tokenizer since there's no tokenizer.model file!" -- the
guard defeats the exact intent the caller states. The effect is silent: the
caller still gets a working `new_tokenizer`, but the sentencepiece piece
rename is skipped, so the mapped token (e.g. the eos token remapped to
`<|im_end|>`) is missing from tokenizer.model and GGUF/llama.cpp exports
carry the old piece.
`check_if_sentencepiece_model` in save.py does the same probe in the right
order -- makedirs, save_pretrained, then isfile. Match it.
Tests are added under tests/saving/ next to the existing sentencepiece
coverage, and to the two Bucket-A lists in consolidated-tests-ci.yml, since
Repo tests (CPU) --ignores tests/saving and these need protobuf.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Clear stale tokenizer.model before the sentencepiece guard
The guard now runs after old_tokenizer.save_pretrained, but the default
temporary_location is a fixed reusable directory. A fast-only tokenizer writes
no tokenizer.model, so a stale file from an earlier sentencepiece call could
pass the guard and patch the wrong model (e.g. mixing models in one process,
like a long-running server). Remove any existing tokenizer.model first, and add
a regression test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Empty the reusable sentencepiece scratch directory each call
The final AutoTokenizer.from_pretrained reloads the whole temporary_location, so
removing only a stale tokenizer.model still let other artifacts from a previous
tokenizer (added_tokens.json, chat template, etc.) leak into the reload when the
default reusable directory is used across models in one process. Recreate the
directory instead, and add a regression test for the leaked-artifact case.
* Clear only top-level scratch files, keep subdirectories
Recreating the whole reusable directory deleted the {name} subtree that
convert_to_fast_tokenizer stores a converted tokenizer's source vocab in, so
old_tokenizer.save_pretrained could not copy tokenizer.model and the guard
returned the tokenizer unpatched for those legacy converted tokenizers. Remove
only stale top-level files (all the final reload reads) and leave subdirectories
intact. Add a regression test for the converted-source subdirectory.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep the current tokenizer's own source vocab when clearing
On a repeated get_chat_template(map_eos_token=True) call, the returned tokenizer's
vocab_file points back at the top-level tokenizer.model, and the cleanup deleted
that source before old_tokenizer.save_pretrained could re-emit it, so the guard
returned the tokenizer unpatched. Skip removing the old tokenizer's own source
vocab while still clearing stale files from a different tokenizer, and add a
regression test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use a per-call temporary directory for the sentencepiece fix
The scratch directory defaulted to a single shared path, so concurrent or repeated
get_chat_template(map_eos_token=True) calls could delete or overwrite each other's
tokenizer.model between save and reload (tripping the piece assertion or reloading
the wrong model), and stale files from an earlier tokenizer could leak into the
reload. Work in a unique per-call subdirectory instead: this isolates every call
without deleting anything the caller owns, and replaces the earlier per-file cleanup.
Tests updated to read the patched model from the reloaded directory and to cover
isolation and source-vocab preservation.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pass only the applied token mappings into the sentencepiece fix
get_chat_template mirrors token remaps into tokenizer.model via
fix_sentencepiece_tokenizer, but two caller paths passed a mapping that did not
match what they wrote to the fast tokenizer JSON, so once the sentencepiece patch
runs the model and JSON disagree:
- the mapped-token path skipped entries whose target already existed but still
passed the full mapping, renaming a piece the JSON never changed;
- the EOS-swap path swapped both tokens in the JSON but passed only one direction,
leaving two stop_word pieces and no old EOS piece.
Pass the applied mapping (and both swap directions) instead. Add regression tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten sentencepiece guard comments
* Add SPDX license identifier to sentencepiece guard test
* Reclaim the per-call sentencepiece scratch directory
The per-call tempfile.mkdtemp fixed the shared-directory race but never cleaned
up, so a long-running process leaked one scratch dir per call. The dir cannot be
deleted eagerly for sentencepiece tokenizers because the returned tokenizer's
vocab_file points into it (a later save_pretrained copies the patched
tokenizer.model from there). Reclaim it correctly instead: remove the dir right
away on the fast-only path (the returned tokenizer never references it), and
attach a weakref.finalize so the sentencepiece dir is removed once its tokenizer
is garbage collected. Add regression tests for both.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten the scratch-dir reclaim comment
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: danielhanchen <unslothshared@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* Stabilize Studio regression tests Rebuild on current main. Restore the set-membership sidebar account-block matcher (#6647, which fixed the same order-sensitive regex, was reverted on main, so the guard is failing on main again) and keep the watchdog replacement-race fix, whose blocked-watchdog stub now waits without a timeout so a superseded watchdog stays alive until cleanup regardless of scheduler load. * Tighten the blocked-watchdog stub comment --------- Co-authored-by: Daniel Han <unslothshared@gmail.com>
…(#7195)
* fix(dataprep): skip .jsonl lines that are valid JSON but not objects
`_read_file_by_format` json.loads each line and hands the result to
`_extract_text_from_json`, which assumes a dict:
for field in self._TEXT_FIELDS:
if field in data and isinstance(data[field], str):
A JSON line does not have to be an object -- `"context"`, `["text"]` and
`42` are all valid JSON. For those, `field in data` stops being a key
lookup and becomes a substring/membership test, so `data[field]` raises:
"context" -> "text" in "context" is True (substring!)
-> TypeError: string indices must be integers
["text", "foo"] -> TypeError: list indices must be integers
42 -> TypeError: argument of type 'int' is not iterable
The TypeError escapes past `except json.JSONDecodeError: continue`, so the
whole load dies on one odd line.
That except clause is also the tell: a *malformed* line is already skipped
gracefully. A *well-formed* line that happens not to be an object should be
too -- it carries no text either way. This makes the two agree.
Reachable from `unsloth-cli.py:253` (`--dataset foo.jsonl` auto-detect) and
`RawTextDataLoader` is exported from `unsloth/__init__.py`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Slim the non-object jsonl regression test and shorten the guard comment
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
…7230) * fix(studio): equal padding in dataset source segmented control * fix(studio): scope dataset source pill layoutId per component instance
The Connections form hid the API key field for the Ollama preset, which blocked Ollama cloud (it requires a key). Show the optional field for Ollama; the backend already sends Authorization: Bearer when a key is set and omits the header when empty, so local keyless servers are unaffected. Fixes #7163
…#7231) With 5 or more pills active the composer collapses every pill to an icon, which hid the Bypass permissions label behind a small glyph. Exempt the permission pill via data-keep-label so it always shows its label, with the collapsed icons lining up to its right. Since the pill is never icon-only now, drop the compact-mode fallthrough in the glyph off switch so it works while the other pills are collapsed.
* fix(studio): add MLX adapter state control * fix(studio): honor MLX adapter comparison state * fix(studio): keep enabled MLX adapters permissive * Studio: preserve public error message on MLX compare-mode adapter failures generate_with_adapter_control raised a plain RuntimeError, which the compare route handled with the generic handler that drops the operational message. Raise GenStreamErrorRaised(public=chunk.public) instead and catch it in the streaming and non-streaming consumers, matching the safetensors tool loop, so errors like 'model is being unloaded' surface their real message. * Studio: re-emit VLM think prefill inside the adapter context The compare-mode merge dropped _generate_vlm's upfront yield of the prefilled <think> block. Restore it as the first snapshot inside the lock+adapter context (matching _generate_text) so the UI renders the thinking block during prefill and a cancel/error before the first token does not drop it. Adds a regression test asserting the prefill is emitted first, after entering the adapter context. --------- Co-authored-by: danielhanchen <unslothshared@gmail.com>
…release (#7223) The pip scan-packages studio shard is red on main and on every open PR: the baselined fastapi finding (the benign SSE keepalive `while True:` loop in fastapi/routing.py, reviewed and suppressed long ago) records its evidence at L586 with the span digest of the fastapi release current at baseline time. The latest fastapi shifts that loop to L587 and its span digest with it, so the evidence hash no longer matches and the scanner reports the finding as new, failing the shard with one unsuppressed CRITICAL. Re-reviewed the flagged code in the current release before refreshing: L587 is the same keepalive loop inside the streaming response machinery, not a beacon. Only the one entry's evidence and evidence_hash change. Verified with the scanner itself: `scan_packages.py fastapi --no-baseline` reproduces the exact CI evidence string, and with the updated baseline the same scan exits 0 with the finding suppressed as 1 CRITICAL baselined.
…185) * Studio: enforce 60s minimum on idle auto-unload TTL (0 stays off) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Drop decorative section separator from idle TTL floor tests --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: danielhanchen <unslothshared@gmail.com>
* Replace standalone Studio wording with Unsloth Replace the single word Studio with Unsloth wherever it is used as shorthand for Unsloth Studio in docs, CLI output, UI strings, i18n locales, workflow display names, comments and docstrings. Kept unchanged: the full name Unsloth Studio, third party product names (LM Studio, Visual Studio, Mac Studio), feature names (Recipe Studio, Fine-tuning Studio and its translations), and all identifiers such as env vars, commands, paths and filenames. * Address review feedback on the Studio wording rename Use "an" before Unsloth where the rename left the article as "a". Restore the split brand where Unsloth and Studio render as two halves of the full product name: the onboarding sidebar subtitle and the IPv6 localhost warning. Scope two messages to the full name Unsloth Studio where plain Unsloth was misleading: the AMD README bullet and the CLI studio setup error.
… inventory (#7018) * Studio: hide infra models from the hub cached inventory The hub inventory scans behind /api/hub/cached-gguf and /api/hub/cached-models returned the llama.cpp install validation probe (ggml-org/models) and the RAG embedder (unsloth/bge-small-en-v1.5[-GGUF]) as on-device models. Share the hidden-model check from routes/models.py via utils/models/hidden_models.py and apply it in both scans. A GGUF infra repo stays visible when the user explicitly downloaded a variant through the Hub, since variant manifests only exist for user-initiated downloads. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make On Device trust the hub inventory, match repo ids exactly, lighten the hidden-model import Follow-up on the hub cached-inventory hidden-model change, addressing the review. On Device now trusts the Hub inventory API for cached rows. The backend already hides the RAG embedder and the llama.cpp probe and re-includes a GGUF infra repo once the user downloads a variant through the Hub, but the frontend was re-hiding it by repo id, so the user-downloaded variant never appeared in the On Device list or the count. isVisibleInventoryRow now short-circuits cached rows (kind === "cache") to visible and keeps client-side needle hiding only for local filesystem rows and Discover. is_hidden_model matches Hub repo ids exactly (case-insensitive) against the probe plus the effective embedder and its GGUF companion, instead of substring matching the configured-embedder basename. A custom embedder with a generic basename like org/model no longer hides unrelated cached repos such as user/model-chat or org/model-instruct. The probe filename and local-path embedders keep exact matching. The helper moves to utils/hidden_models.py and is imported at module scope in the hub cache scanner, so it no longer pulls in utils/models/__init__ (the eager model-config/checkpoint stack) and a broken import fails at startup instead of being swallowed per-repo and silently emptying the inventory. routes.models keeps the _is_hidden_model and _safe_resolve aliases and drops the unused _HF_REPO_ID_RE re-export that was failing source lint. Tests: exact repo-id matching with a custom embedder, the cached-models scan keeping an unrelated repo, and a clean-interpreter check that the helper imports without the model-config stack. * Studio: match the llama.cpp probe filename on both path separators The hidden-model check compared the probe's on-disk filename with Path(value).name, which on a POSIX interpreter does not split a Windows-style path ("...\stories260K.gguf") and would let the probe through. Split on both separators so the probe is matched regardless of which OS produced the path, matching the tolerance of the previous substring check. Adds a Windows-path assertion to the probe test. * Studio: harden hidden infra model handling * Fix hidden cache row confirmation * Fix hidden local rows and confirmed hint merges * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Handle snapshot-configured hidden models * Hide basename-only default embedders * Fix dynamic embedder inventory filtering * Studio: hide the configured RAG embedder from Discover and feed rows --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothshared@gmail.com> Co-authored-by: Daniel Han <23090290+danielhanchen@users.noreply.github.com>
On Windows ROCm without a HIP SDK, amd-smi is disabled and the System tab fell back to torch mem_get_info, which reports free==total there (ROCm/legacy-rocm-build#1909), so used VRAM showed as 0. The perf-counter fallback also summed every adapter into a single device with only GPU 0's total, hiding the second GPU. Read per-adapter Dedicated Usage (LUID-instanced) for used and take each GPU's total from torch properties, and treat the free==total case as unknown rather than 0, so every GPU shows real usage. NVIDIA, Linux ROCm, Apple and CPU paths are unchanged. Final validation needs a real Windows AMD box. Fixes #7072
for more information, see https://pre-commit.ci
Two gaps in the Windows ROCm VRAM path. When more adapters are actively using VRAM than are visible to the process (a GPU outside the visibility mask), the per-adapter attribution paired usage by size and fabricated a per-GPU value; report unknown for every device in that case rather than mis-assign. And the System API turned an unknown (None) used value into 0 with ``or 0``, then reported the full card as free, re-hiding the exact case this change surfaces; keep None so the UI shows unknown.
for more information, see https://pre-commit.ci
The backend reports null usage when it is unknown (e.g. the Windows ROCm perf counter is unavailable or localized), but the System tab coerced null to 0 and derived free from it, fabricating a 0-used/full-free total. Preserve null and render the translated Unknown for per-device used, free and utilization, and mark the aggregate VRAM tile unknown when any device is unknown.
The floating VRAM monitor and the aggregate utilization ring both still coerced a null usage to 0, showing a fabricated 0.00 GiB / full free / 0% on the same Windows ROCm no-counter case the resources tab already handles. Guard both on whether every device reports a finite usage and render Unknown (value and percent) instead of a concrete 0.
On Windows/ROCm there is no shared key between LUID performance-counter instances and torch ordinals, so usage was paired to devices purely by capacity ranking. That pairing is only trustworthy when capacity forces it (a usage larger than every smaller device can sit on one card). When a smaller-capacity device could equally hold a strictly larger usage (for example an 8 GiB card near full beside a lightly used 48 GiB card), the two values are swappable without violating any capacity, so the ranking is a guess with no key to break the tie. A wrong guess both mislabels the System tab and feeds routes/training_vram.py a wrong per-index free value, driving a wrong keep-resident decision. Report unknown for every device when the assignment is ambiguous, keeping the attribution only for the capacity-forced case. Returning None is the conservative direction: training_vram treats a missing index as zero free, so it never keeps a chat model into an OOM. Add regression tests for the not-capacity-ordered, same-capacity, single-fits-both, and capacity-forced cases.
…lter When HIP_VISIBLE_DEVICES exposes a subset of the physical adapters, the LUID usage counters cover cards outside the visibility mask too. The sub-64 MiB noise filter could drop a genuinely-idle visible card's real usage while keeping a hidden larger card's high usage, which was then clamped onto the smaller visible device and reported as fully used (for example a hidden 48 GiB card at 40 GiB shown as a visible 8 GiB card fully used, with its true 10 MiB usage filtered out). That fabricated reading also feeds routes/training_vram.py a wrong per-index free value. Flag extra adapters on the raw counter count (before the noise filter, since an idle visible card can itself fall below the floor) and, when a kept usage exceeds its ranked visible capacity, report unknown rather than clamp a hidden card's usage onto a visible device. The genuinely-idle-noise and capacity-forced single-model cases are unchanged. Add a regression test for the hidden high-use-adapter case in both counter orders.
…oise filter When more raw counters than visible devices are present but every counter sits below the 64 MiB noise floor (an idle real GPU alongside a Windows Basic Render Driver placeholder), the non_trivial-or-raw fallback resurrected the raw magnitude-sorted counters and could attribute the placeholder to a real GPU while dropping a real card's reading. With a single visible device the swap-ambiguity check cannot catch it (it needs at least two ranks), so the fabricated value reached the System tab and automatic GPU selection. Return unknown for every device in that case instead of falling back to raw counters. With the earlier guards this completes the invariant: a concrete per-GPU usage is emitted only when the assignment is capacity-forced, and every ambiguous, extra-adapter, placeholder-fallback, or count-mismatch path reports unknown. Add a regression test for the placeholder fallback in both counter orders and the two-idle-GPU case.
for more information, see https://pre-commit.ci
… bijection With more raw adapter counters than visible devices, a survivor that merely fits a visible card was pinned to it by magnitude ranking, fabricating a hidden GPU's usage onto an idle visible card whose true reading was dropped by the sub-threshold noise filter (two visible 48/8 GiB cards using 40 GiB / 10 MiB beside a hidden 6 GiB adapter returned [40, 6]). Emit a concrete per-device value only when the supra-threshold counters number exactly the visible devices (every visible card has one real reading, the extras were sub-threshold placeholders) AND the ranked usage strictly exceeds every smaller visible card's capacity. When a visible card is idle (fewer supra-threshold counters than devices) a survivor could be the hidden GPU's usage, so every device reports unknown; more active counters than visible cards, the smallest card, and any merely-fitting usage stay unknown too. The reporter's loaded-card display is preserved (40 GiB / 0.5 GiB across 48/8 GiB -> [40, None]). Adds a regression test for the reported case plus an exhaustive capacity-forced/bijection matrix.
for more information, see https://pre-commit.ci
_apply_unified_memory_correction gated both the total and the used update on torch_used_gb being known, so on a unified-memory APU (Strix Halo) where torch reports used=None (the Windows-ROCm free==total sentinel) but an authoritative full-GTT total, the device kept amd-smi's small dedicated carve-out and underreported its capacity on the System tab. Adopt torch's larger total independently of used; overwrite used only when torch's is known (otherwise keep amd-smi's dedicated-usage figure) and recompute utilization against the corrected total. Adds regression tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Disposable CI run for unslothai/unsloth#7238. Do not merge; closed after CI.