Conversation
for more information, see https://pre-commit.ci
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for the DFlash block-diffusion speculative decoding drafter (dflash-*.gguf), integrating it similarly to the existing MTP drafter mechanism. The changes generalize companion path checks, sibling detection, and command-line flag generation to support both MTP and DFlash drafters, and include comprehensive unit tests. Feedback on the changes suggests refining the sibling drafter matching logic to prevent false positives where a shorter model name prefix incorrectly matches a longer model name (e.g., matching qwen to qwen3).
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2da0f2878b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
for more information, see https://pre-commit.ci
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
unsloth/studio/backend/utils/models/model_config.py
Lines 1362 to 1363 in d2b20b8
Prefer the most specific DFlash match
In a folder containing multiple related targets, this returns the first sorted drafter whose stem prefixes the weight name. For example, dflash-Qwen3-4B.gguf sorts before dflash-Qwen3-4B-Instruct.gguf, and both prefix Qwen3-4B-Instruct-Q4_K_M.gguf, so loading the Instruct target attaches the base-model DFlash drafter even though the exact drafter is present. Please choose the longest/exact matching stem rather than the first lexicographic match so mixed local folders do not pair a foreign draft model.
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3b46f728f4
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
1 similar comment
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bf9b6cc2fa
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bf9b6cc2fa
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bf9b6cc2fa
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5480517169
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fec2210c81
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6a2d9aab80
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f8870c1c91
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bb853e2d4e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 731dae7759
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
for more information, see https://pre-commit.ci
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 862228108e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 73ce91a846
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if dflash is not None: | ||
| total += getattr(dflash, "size", 0) or 0 |
There was a problem hiding this comment.
When active training is running and an HF GGUF repo is a VLM (include_mmproj=True), this still adds a DFlash sibling to the admission estimate even though the loader suppresses DFlash for effective vision loads (is_dflash_model requires not is_vision). A vision repo that happens to contain a DFlash file can therefore be rejected with a 409 despite /load only mapping the main GGUF and mmproj; skip the DFlash add when the projector is active.
Useful? React with 👍 / 👎.
| for attr in ("gguf_mmproj_file", "gguf_mtp_file", "gguf_dflash_file"): | ||
| f = getattr(config, attr, None) | ||
| if f and Path(f).is_file(): | ||
| total_bytes += Path(f).stat().st_size |
There was a problem hiding this comment.
For local VLM GGUFs loaded with their mmproj enabled, load_model() passes effective_is_vision=True and _build_speculative_flags() will not emit DFlash, but this unconditional companion loop still charges gguf_dflash_file during the active-training guard. A local vision model directory with a DFlash sibling can be blocked as too large even though the drafter would not be launched; gate the DFlash entry on the effective mmproj/--no-mmproj state.
Useful? React with 👍 / 👎.
|
Thanks for this, and sorry it sat open so long. I ended up building the same feature in #8338 before catching up with this PR properly, so I want to be straight about what happened and make sure your work is not just discarded. Why I am landing #8338 rather than thisThis branch is 848 commits behind main and conflicts in 7 backend files. That is not the real problem though. The real problem is that it branched before the DSpark drafter stack existed at all -- The one behavioural difference I think matters: detection here is filename only. Upstream hard-fails on a drafter whose architecture is not exactly What I carried across from here into #8338, with credit
What I deliberately did not carry across
Closing this as superseded by #8338. The five items above are in that PR with this one linked from the commits. If you think I have got the vision call wrong, or anything else here, say so on #8338 and I will re-open the question -- the benchmark script is in the PR discussion so it is easy to re-run. |
…got right Five changes on top of the DFlash drafter work. Lint gate. The compatibility shim re-exporting the moved drafter helpers from model_config tripped scripts/verify_import_hoist.py, whose __all__ exemption is scoped to package __init__.py and which ships a reexport_in_ordinary_module_is_still_blocked self-test. The shim is gone: the module imports only what it still calls, and every other call site imports from utils.models.drafters directly. dspark_preference_key stays reachable from model_config as a delegating def, because repointing routes/inference.py's pre-existing function-local import is the verifier's TARGET-CHANGED case and its relocation exemption only covers module-level imports. Download plan. preferred_dflash_sibling in hub/utils/gguf_plan.py, and the sidecar as an expected file on every GgufVariantPlan. The sidecar was fetched but the hub manifest never knew about it, so download progress under-counted by ~1.5 GiB. Ranked with dflash_repo_preference_key, so the plan and the loader cannot disagree, and per variant, so a multi-family repo does not hand variant B the drafter named after variant A. Capability-regained retry. A load that stood down because llama-server could not run the drafter told the user to update, then deduped the reload the update was meant to repair. spec_binary_fallback_can_retry re-reads the binary, asking about the capability the drafter kind actually needs rather than the reason code, since every kind records the same binary_no_mtp. Transient fetch retry. _download_companion_gguf gained on_transient_failure, so a listing that never answered or a download that dropped is worth one more Apply. Permanent Hub errors, a full or unwritable cache, offline mode and cancellation are unaffected, and a header rejection still falls through to the next candidate rather than counting as transient. The probe cache key moved from (path, int(mtime)) to (path, st_mtime_ns, st_size), so an update landing in the same second as the probe is not answered with the old build's capabilities. CLI. unsloth_cli/_inference.py passes gguf_dflash_file into the GGUF load, so the managed CLI path engages DFlash instead of silently running without it. No vision gate, now measured rather than argued. Muse-Glimmer-30B UD-Q4_K_XL with mmproj-kquant and dflash-kquant, llama.cpp b10342, one B200, n_max=2, greedy, on a prompt carrying ~545 image tokens: 92.1 to 114.2 tok/s at 0.646 acceptance, greedy output byte-identical to the drafter-free run, no load failure. The comment at the Auto promotion site cited llama.cpp #22673, which is an MTP result, for a DFlash decision; it now cites the measurement. Also fixes test_from_identifier_never_reads_a_sidecar_outside_the_boundary, which patched is_dflash_architecture on the re-exporting module rather than the one detect_dflash_file resolves it in, so its reads == [] assertion held whether or not the lease boundary worked.
* Studio: launch a DFlash drafter automatically Studio has recognised dflash-*.gguf since #7811, but only to hide it from the quant picker. Nothing ever launched it, so a model that ships a DFlash sidecar fell through to no speculative decoding at all. Add DFlash as the third launchable drafter kind beside MTP and DSpark: a _is_dflash_drafter_path predicate, local and Hub discovery, a supports_dflash capability parsed from llama-server --help, and the --model-draft / --spec-type draft-dflash emission. Unlike DSpark it is on under Auto, since the published sidecar is 1.52 GiB and ships in the model's own GGUF repo rather than being an ~11 GB opt-in fetch. DSpark keeps first refusal when a repo somehow ships both, matching llama.cpp's own downloader. Discovery confirms general.architecture = dflash in the header rather than pairing on the filename: the published sidecar is dflash-kquant.gguf, which names no model family, so the DSpark pairing rule would reject the one file this exists to find. The dflash/ directory is still not a drafter marker, and DFlash is still excluded from companion reclaim, both because the name doubles as a family a publisher puts on real weights. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Harden the DFlash drafter fallback, pairing and dedupe Four fixes from review of the auto-launch path. Strip user-supplied DFlash args on the drafterless retry. The gate that enters the retry counts a DFlash request, but the cleanup only recognised MTP and DSpark. llama.cpp accumulates speculative types, so prepending --spec-default while the DFlash group survived relaunched the drafter that had just failed, and a main model that loads fine without it was lost instead of recovered. Skip a DFlash sidecar that names another weight in the same folder. _drafter_matches_weight is False both for a sidecar naming no family and for one naming a different family, so ranking put them in one bucket and precision could float the foreign one to the top: loading model B beside dflash-model-A-Q8_0.gguf and dflash-kquant.gguf launched model A's drafter. Both files carry a real dflash header, so the architecture check behind the ranking cannot catch it. The decision is made against the weights actually present in the folder rather than by guessing which stems are precision tokens, which keeps the published unpaired sidecar eligible. Stand the Auto DFlash fetch down once DSpark has resolved. DSpark takes first refusal in the promotion, so for a repo shipping both kinds the DFlash sidecar could never launch and the fetch spent bandwidth and cache on a file that would not be used. An explicit dflash request still fetches. Keep Auto deduplicated after a failed DFlash drafter. _speculative_type is reset to "default" by a successful drafterless retry while the launch still records the resolved sidecar, so the next Apply compared the intent's empty MTP path against it and reloaded a healthy server. _spec_drafter_kind survives the fallback and now decides the comparison. test_mtp_drafter_companion.py, test_native_gguf_companion.py, test_llama_cpp_mtp_detection.py and test_resolve_quant_gguf.py: 489 passed, including two new tests for the foreign-sidecar case and for the paired sidecar still winning. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep DFlash discovery, the training guard and the hints in step Discovery now accepts the dflash- prefix only. The shared companion predicates recognise DFlash by that prefix, so a <model>-dflash.gguf accepted by discovery was also a selectable Q8_0 main model in the quant picker, and choosing that variant handed llama-server the drafter as the target. Teaching the predicate the suffix instead would hide a real model whose name merely ends in DFlash, which is the case #7811 exists to protect, so detection gives the form up rather than the picker giving up a model. No published sidecar uses it; the shipped one is dflash-kquant.gguf. The same mismatch exists for MTP on main and is left alone here. The training VRAM guard now sizes a drafter named through llama_extra_args. Discovery never fills gguf_dflash_file for a file outside the model directory, but load_model still passes that path to llama-server, so a load could be admitted beside a training run while nothing was charged for the sidecar it makes resident. The Speculative Decoding hint said Auto picks DSpark or else MTP / ngram and that everything but DSpark leaves output unchanged. Auto now picks DFlash too, and like DSpark it is not bit-identical on quantized targets. The Draft Tokens hint gained the DFlash default, which shares the MTP branch at 2 on GPU and 3 on CPU/Mac. 514 passed across the drafter, companion, detection, quant-resolution and picker suites, including two new tests pinning the suffix form out of discovery and the prefix form still in. * Pair the remote DFlash sidecars with the selected weight detect_dflash_file already refuses a sidecar named after a NEIGHBOURING weight, so a folder holding two families cannot attach a foreign drafter locally. The download picker and the offline cache reuse still ranked every dflash-*.gguf by precision and name alone, never comparing a candidate against the weight being loaded, so in a repo hosting more than one family dflash-model-A-Q8_0.gguf outranked the generic dflash-kquant.gguf and model B downloaded and launched model A's drafter. The pairing rule now lives in one place, dflash_repo_preference_key, built on the same _drafter_names_other_weight predicate the local scan uses: a sidecar naming this weight's family first (most specific stem first, as detect_mtp_file does), then one naming no weight present here, then one naming a neighbour. The last is demoted rather than dropped, so a repo whose only sidecar looks foreign still has a fallback. Deciding against the weights actually present is what keeps the published unpaired sidecar eligible: dflash-kquant.gguf has a precision token for a stem, not a family name, so "the stem is non-empty" cannot stand in for "this names another model". Nothing changes for a repo with one sidecar, and with no weight in hand the order is precision only, as before. Tests cover a multi-family repo picking the generic sidecar, the same repo picking the specific one for its own weight, the shipped Muse-Glimmer layout still resolving, and the cached path agreeing with the download path. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Validate DFlash candidates and size the extras drafter once Three fixes found in review of the DFlash drafter work. detect_dflash_file read a candidate's GGUF header before asking the caller's accept callback about it, so a dflash-*.gguf symlink in a directory reached through a native grant had its out-of-lease target opened before the grant check ran, and no later rejection takes a read back. The loop now resolves the launch path, runs accept, and only then parses the header and applies the architecture check. accept still receives the resolved launch path, and callers that pass no accept see the same candidates in the same order as before. The training admission guard charged the llama_extra_args --model-draft sidecar on top of the local one discovery had already found, so a 1.5 GiB drafter was billed as 3 GiB and the guard could refuse an inference load that fits. The effective draft path is now sized exactly once, with identity taken from the resolved path so a symlink or another spelling of the same file dedupes too. That same charge also satisfied the local-weights early return on its own. Loading a remote GGUF repo has no local main weight, so a local --model-draft made the guard return the drafter alone and skip the listing that prices the target model, which could admit a load that then exhausts VRAM next to a running training job. The local branch now fires only when a local weight is actually present, and the drafter is added to whichever branch produces the estimate, including the remote one. Regression tests for all three. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Validate remote DFlash files by header and stop charging unused DFlash bytes Two fixes to the DFlash sidecar paths. Remote and cached DFlash candidates are now confirmed by their GGUF header, not by their filename. _pick_dflash and _cached_repo_dflash_drafter selected with _is_dflash_drafter_path, a dflash- prefix test, while the local scan in detect_dflash_file also required general.architecture == dflash. A remote repo holding an ordinary weight whose basename starts with dflash- therefore had that full weight downloaded and handed to llama-server as --model-draft, which falls back at startup after the bytes are already spent. The architecture rule moves into is_dflash_architecture in model_config, beside the naming rules and shared by every path, the way dflash_repo_preference_key already is. The header is only readable once the file is on disk, so the download validates after the fetch and falls through to the next candidate instead of returning None; the prefix-only naming rule is unchanged. The training coexistence guard no longer charges DFlash bytes a load under Auto will never fetch. _remote_gguf_companion_bytes added the preferred DSpark and the preferred DFlash sidecar whenever the repo listed both, but the loader stands down on the DFlash fetch once DSpark resolves under Auto, so those bytes are never resident and the guard could 409 a load that fits. The new dspark_first flag mirrors that selection. Where the choice is genuinely unknown the deliberate over-estimate stands, and an explicitly forced DFlash still pays for its sidecar. Regression tests cover the fetch falling through an impostor to the real sidecar, an all-impostor repo recording a permanent absence, the snapshot reuse and offline cache lookups applying the same rule, and the Auto guard charging DSpark only when a repo publishes both kinds. * Gate the DFlash stand-down and the guard's sizing on what the load actually does The Auto DFlash fetch stood down whenever _download_dspark answered with a path, but that call deliberately reports an already-cached DSpark sidecar even on a binary with no usable --spec-type draft-dspark (so the route's reuse check does not reload the same server on every Apply). The promotion refuses such a path, so on a DFlash-capable binary a repo shipping both companions suppressed the DFlash fetch for a sidecar that can never launch and the load ended up with no drafter at all. The capability gate now lives in _dspark_wins_auto, shared by the fetch and the promotion so the two cannot disagree. _remote_gguf_companion_bytes still ranked DFlash candidates with the name-only dflash_preference_key while the loader moved to the family-aware dflash_repo_preference_key, so in a multi-family repo the guard could price a different, smaller sidecar than the one that lands. The selected weight name is threaded down and the guard now sorts with the downloader's key over the neighbouring weights from the same listing. * Apply the load's boundaries to drafter discovery, and size Auto's one drafter ModelConfig.from_identifier ran the local companion scan with no way for the caller to say what was in bounds, so a native-grant load read the header of a dflash-*.gguf symlinked out of the granted directory. The validated rescan on the load route rejected it afterwards, which does not take a read back. The boundary now travels into the scan, for all three drafter kinds, so the two passes cannot disagree about what is in bounds. Remote DFlash discovery matched the basename in any nested directory, but the local contract is root level only: a quants/dflash-*.gguf is an ordinary weight detect_dflash_file would never offer, and the header can only be read once the bytes are here, so the whole weight downloaded before the rejection. Checked through a separate predicate so the prefix-only naming rule the other callers share stays exactly as it is. A split companion is only usable as a whole set, since llama-server resolves the sibling shards from the first one's directory. Fetching just the picked shard left a drafter whose header reads fine and which the server cannot open, so the load fell back to no speculation with nothing to show for the download. The companion download now resolves its shards with the same helper the main-model download uses, and neither reuse path reports a half set as a cache hit. The remote sizing charged the first-ranked DFlash candidate, but a rejected candidate falls through to the next name in the ranking, which can be a larger file; headers are unreadable from a listing, so the bound now covers every candidate the fallback can reach. And under Auto the guard charged the MTP drafter on top of the DFlash sidecar that replaces it. Auto launches exactly one drafter, in a fixed order, so dspark_first now expresses the whole promotion: DSpark alone when the repo publishes one, otherwise the larger of the DFlash bound and the MTP drafter, since every DFlash candidate can still be turned away on its header and the load then keeps the MTP one. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Budget a split DFlash sidecar as a set, and reject half a cached one The remote sizing bounded the DFlash fetch with the largest candidate the post-fetch fallback could land on, but each entry is one shard, while _download_companion_gguf fetches the whole shard set the picked file belongs to and llama-server keeps every shard resident. A sidecar published as two 1 GiB shards was budgeted at 1 GiB, and under-charging is the direction that waves a load through and then exhausts VRAM beside a running training job. The candidates are grouped into their sets with _gguf_extra_shards, the same helper the download resolves shards with, and the bound is the largest set total. _cached_repo_dflash_drafter's offline fallback accepted a candidate on is_file plus its header, so a snapshot holding shard 1 alone was handed back as the drafter with no fetch left to complete the set. The header reads fine, then llama-server cannot open the siblings it resolves from that directory and the load falls back to no speculation. Same _drafter_split_is_complete rule the snapshot reuse already applies, and skipped rather than fatal like the header check, since another snapshot may hold the complete copy. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Move drafter naming, ranking and DFlash discovery into a drafters package Pure structural move, no behaviour change. The shared primitives, the ranking keys and the DFlash detector were spread through model_config alongside unrelated model handling, and the same rules are reached from four different paths, so they now live in one package. model_config re-exports every moved name, so callers and tests that import them from there keep working. The package deliberately does not import model_config at module import time. The GGUF split and quant naming helpers stay where they are, since non-drafter code shares them, and are imported per call instead. * Give the guard's DFlash bound a name and a home The bound was fifteen lines of generator plus the comment explaining why it is a max over shard sets rather than the best-ranked candidate, inline in the middle of a function that also sizes mmproj, MTP and DSpark. It is pure arithmetic over a listing, so it moves to drafters.budget with the reasoning attached to it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: fix the DFlash lint gate, and carry over what #6747 got right Five changes on top of the DFlash drafter work. Lint gate. The compatibility shim re-exporting the moved drafter helpers from model_config tripped scripts/verify_import_hoist.py, whose __all__ exemption is scoped to package __init__.py and which ships a reexport_in_ordinary_module_is_still_blocked self-test. The shim is gone: the module imports only what it still calls, and every other call site imports from utils.models.drafters directly. dspark_preference_key stays reachable from model_config as a delegating def, because repointing routes/inference.py's pre-existing function-local import is the verifier's TARGET-CHANGED case and its relocation exemption only covers module-level imports. Download plan. preferred_dflash_sibling in hub/utils/gguf_plan.py, and the sidecar as an expected file on every GgufVariantPlan. The sidecar was fetched but the hub manifest never knew about it, so download progress under-counted by ~1.5 GiB. Ranked with dflash_repo_preference_key, so the plan and the loader cannot disagree, and per variant, so a multi-family repo does not hand variant B the drafter named after variant A. Capability-regained retry. A load that stood down because llama-server could not run the drafter told the user to update, then deduped the reload the update was meant to repair. spec_binary_fallback_can_retry re-reads the binary, asking about the capability the drafter kind actually needs rather than the reason code, since every kind records the same binary_no_mtp. Transient fetch retry. _download_companion_gguf gained on_transient_failure, so a listing that never answered or a download that dropped is worth one more Apply. Permanent Hub errors, a full or unwritable cache, offline mode and cancellation are unaffected, and a header rejection still falls through to the next candidate rather than counting as transient. The probe cache key moved from (path, int(mtime)) to (path, st_mtime_ns, st_size), so an update landing in the same second as the probe is not answered with the old build's capabilities. CLI. unsloth_cli/_inference.py passes gguf_dflash_file into the GGUF load, so the managed CLI path engages DFlash instead of silently running without it. No vision gate, now measured rather than argued. Muse-Glimmer-30B UD-Q4_K_XL with mmproj-kquant and dflash-kquant, llama.cpp b10342, one B200, n_max=2, greedy, on a prompt carrying ~545 image tokens: 92.1 to 114.2 tok/s at 0.646 acceptance, greedy output byte-identical to the drafter-free run, no load failure. The comment at the Auto promotion site cited llama.cpp #22673, which is an MTP result, for a DFlash decision; it now cites the measurement. Also fixes test_from_identifier_never_reads_a_sidecar_outside_the_boundary, which patched is_dflash_architecture on the re-exporting module rather than the one detect_dflash_file resolves it in, so its reads == [] assertion held whether or not the lease boundary worked. * Tighten the DFlash comments for PR #8338 * Apply ruff-format kwarg spacing for PR #8338 * Fix the DFlash download plan and two stale-state reloads for PR #8338 Five review items, all reproduced first. The download plan promised the wrong files. A split sidecar contributed only its first shard, so the variant read complete while the loader's completeness check then refused the companion; it now carries the whole shard family. The pairing weight came from the listing's first sibling while plan_from_expected_files keeps the lexicographically first family, so a two-family variant key planned the discarded family's sidecar; both now use the kept family. And a root-level dflash- prefix is one real weights carry, which a listing cannot tell apart from a drafter, so a 54 GB model was planned as a companion to a 15 GB variant; a candidate is now bounded by the weights it would draft for, since a drafter is a few layers of its target and cannot outweigh it. The training coexistence guard charged the Auto DFlash sidecar even when extra args owned --spec-type, which stops the loader's promotion, so a chat load could be refused with 409 for bytes nothing would open. Extra args asking for draft-dflash keep the charge. The diffusion early-return cleared the speculative fallback state but not the DFlash retry flag, and discovery runs before the metadata read that classifies the model, so a transient sidecar failure tore down a healthy diffusion server on every Apply. Each fix has a regression test that fails without it. * Carry the DFlash plan bounds into the runtime paths for PR #8338 Four review items from the second round, each reproduced first. The budget still charged a forced dflash mode when extra args owned --spec-type. _build_speculative_flags returns before any mode branch in that case, so neither the forced mode nor the Auto promotion reaches the sidecar; only extra args asking for draft-dflash themselves still pay. The runtime picker had no size bound, so a root-level ordinary weight carrying the dflash- prefix downloaded in full before its header could be read, which is exactly what the download plan now refuses. It applies the same bound, sized from the repo listing, and an unavailable size leaves the candidate eligible as before. A permanent listing error records no answer at all, so _dflash_sidecar_absent stayed False and the drafter_not_found arm relaunched a healthy drafter-free server on every Apply. DFlash asks through _dflash_retry_needed instead, which is set only for the failures worth another attempt. A listing holding part of a split companion returned its first shard as usable, contradicting the complete-set checks on snapshot and cache reuse and handing llama-server a set it cannot open. The filename carries the set size, so the listing is now checked before the download. Each fix has a regression test that fails without it. * Make the DFlash size and split rules agree across plan, fetch and guard for PR #8338 Six review items from the third round, each reproduced first. Five are places the previous round's rules had not reached. dflash_plan_files now filters candidate families before ranking rather than after, so a half-published split set or an oversized ordinary weight at the top of the order steps aside for a usable sidecar behind it instead of taking the plan down with it. It also applies the split-completeness rule the runtime got last round, since planning a set the listing only half carries reports the download complete and then loses DFlash. The runtime size bound compared the picked shard rather than its whole set, so a split ordinary weight whose halves each sit under the target still downloaded in full. It sums the family now, through a shared helper. The training coexistence guard took the maximum over every root candidate with no size bound at all, charging gigabytes for files the fetch itself refuses. dflash_budget_bytes takes the target size and drops them. The incomplete-split rejection added last round lands after outcome["listed"] is set, so DSpark read a settled answer as retryable and relaunched a healthy server on every Apply. It records absence explicitly. SpeculativeType omitted dflash, so Typer rejected --speculative-type dflash before any of the new loading code ran and the mode was reachable only through Auto. Each fix has a regression test that fails without it. * Price DSpark by shard set and share one split-listing rule for PR #8338 Four review items from the fourth round, two of them under-charges that could admit a load beside a running training job and then exhaust VRAM. The guard priced a remote DSpark sidecar as the single file the ranking picked, while llama-server maps every shard of a split set, so a two-shard sidecar was budgeted at roughly half its resident weight. DSpark candidates are grouped into shard families now and the selected family's total is charged, matching what DFlash already did. Auto granted DSpark first refusal on the strength of the listing alone. Since the fetch now refuses an incomplete split set, the load falls through to DFlash, which can be the larger of the two, and the guard had already returned the DSpark figure. Only a complete set settles it. The runtime DFlash picker filtered incomplete families after ranking rather than before, so a half-published set at the top of the order returned a shard, _download_companion_gguf refused it, and the loop ended instead of reaching the complete sidecar behind it. Extras owning --spec-type with their own --model-draft charged the discovered sidecar as well, though _build_speculative_flags returns before that one is emitted. Only the drafter that launches is charged now. Extras without --spec-type still charge both, since Studio emits its own and which lands is genuinely unknown. The listing completeness rule was about to have three copies, so it moved into utils.models.drafters as split_listing_is_complete and the plan, the fetch and the guard all call it. Each fix has a regression test that fails without it. * Tighten the DFlash review-round comments for PR #8338 Comments and docstrings only, no code change: verified with comment_tools.py check and the prepush gate's comment-only mode. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2 participants
Adds Studio support for DFlash draft models. DFlash is a block-diffusion drafter (a separate
--model-draftGGUF, mechanically the same as Gemma's separate MTP drafter), so this reuses that lane: detect adflash-*.ggufsibling next to the weights and auto-engage--spec-type draft-dflash.Usage
Drop a
dflash-<model>.ggufnext to the target GGUF and load in Auto mode. DFlash currently applies only to text models. Studio detects it and launches:Default draft depth is 4 (vs MTP's 2/3). llama-server clamps it to the drafter's trained block size.
Speed
Direct llama.cpp
b9840, Qwen3-4BQ4_K_Mtarget + DFlashq8_0drafter, single RTX 6000 Ada, code prompt, thinking off, greedy:End-to-end through Studio (default n-max 4, same model and GPU): baseline ~200 tok/s, DFlash ~270-357 tok/s, 1.3x-1.7x.
The gain is workload-dependent. It shows on low-entropy work (code, structured output) with thinking off, and shrinks on chat or creative work, or with thinking on, where acceptance drops. The drafter is small (~575 MB at q8), so the cost when it does not help is low.
Making the drafter GGUF
The public community DFlash GGUFs are built for forks (arch
dflash-draft) and fail on upstream llama.cpp, which expects archdflashfrom llama.cpp#22105. So the drafter used here was converted from the z-lab checkpoint with theb9840converter:--target-model-diris required: the drafter carries no tokenizer of its own and inherits the target's tokenizer and embeddings. The target GGUF in the speed test below isunsloth/Qwen3-4B-GGUFatQ4_K_M.What changed
detect_dflash_filefinds adflash-*.ggufsibling (stem-paired to the weight, like the MTP drafter). It is excluded from main-model selection in the three companion-predicate mirrors._emit_dflashemits--model-draft / --spec-type draft-dflash / --spec-draft-n-max, gated on adraft-dflashcapability probe. It falls back to--spec-defaulton an older binary.-hfloads and included in non-vision Hub download plans.dflash-*.ggufappearing or disappearing next to a loaded local model changes the launch state, mirroring the MTP drafter. Transient HF companion download failures are retried without maintaining DFlash-only file-revision state.Tests
New
test_dflash_drafter.pycovers companion predicates, sibling detection and pairing, flag emission, HF download precedence, fallback behavior, and reload deduplication. The focused DFlash, MTP, route, budget, context-fit, and tensor suites pass (661 tests).Verified live on the b9840 Vulkan build: Auto load engages DFlash, identical reload dedups, dropping the drafter reloads to spec-default, and a broken drafter falls back to no speculative decoding.
Scope boundary
DFlash intentionally reuses the MTP companion lane instead of adding a parallel policy layer. HF Auto loads resolve DFlash first and skip the unused MTP download when DFlash succeeds. Training admission uses the same conservative companion accounting already used for MTP. DFlash does not add capability-dependent validation fields or binary/drafter revision fingerprints that MTP does not maintain.