QVAC-22629 feat: run an advisory llama.cpp fit check before loadModel by simon-iribarren · Pull Request #4010 · tetherto/qvac

simon-iribarren · 2026-08-23T19:01:46Z

🎯 What problem does this PR solve?

loadModel has no idea whether the llama.cpp load it is about to run fits in memory. A model that doesn't fit either fails late or loads fine and then fails at the first decode.

📝 How does it solve it?

Before each completion/embedding llama.cpp load, run @qvac/model-fit in one disposable Bare child and log the verdict (fit / does-not-fit / no evidence).
Advisory and fail-open: no verdict, crash, timeout, or refusal ever blocks or alters a load. Denial stays gated behind QVAC-22634.
On by default; QVAC_ADVISORY_MODEL_FIT=0 opts out. Cost: one child (~0.5 s) per llama.cpp load.
Builds the fit request from the same config transforms the real load uses; forwards only memory-relevant keys, and refuses to guess (mobile, sharded, multimodal, LoRA, CPU, unclassified keys → no verdict).
The outcome is stored on the registry entry, internal only — it lives beside the assessModelFit estimators (resources/model-fit/native-probe/, basis/estimatorVersion); exposing it publicly is a follow-up decision.
Runnable demo: packages/sdk/examples/advisory-model-fit.ts.

🧪 How was it tested?

71 tests / 204 asserts in test/native-probe-*: supervisor against real disposable children (crash, hang, kill), request partitioning, fail-open on every failure mode. typecheck/lint clean.
End-to-end on 24 GiB and 48 GiB M4 Pros: verdicts logged, loads never blocked; a does-not-fit load proceeded to a working completion. Verdicts are machine-dependent by design, so no e2e pins a specific one.
Bundling: the exact bare-pack --linked invocation qvac bundle uses, all hosts — bundles clean, no --defer needed.

github-actions · 2026-08-23T19:02:08Z

License compliance — clean

No new dependency license findings in this PR.

Warn-only (shadow) mode — this check does not block merges yet.

Updated automatically by the canonical license compliance workflow.

NOTICE presence (advisory)

Missing NOTICE (advisory, does not block):

./docs/website
./packages/fabric/test/integration
./packages/llm-llamacpp/benchmarks/server
./packages/llm-llamacpp/benchmarks/performance
./packages/inference-addon-cpp/mobile
./packages/asr-ggml/benchmarks/server
./packages/embed-llamacpp/benchmarks/server
./packages/embed-llamacpp/benchmarks/performance
./packages/sdk/e2e
./packages/vla-ggml/sim/server
./.github/actions/release-merge-guard

socket-security · 2026-08-26T10:37:22Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	npm/@qvac/model-fit@0.8.0

View full report

github-actions · 2026-08-26T10:37:34Z

Review Status

Current Status: ✅ APPROVED
Approvals so far: Team Lead: 1, Member: 1

…adModel Squash of the branch history (supervisor port to protocol v2, request partitioning, advisory orchestration, load-model call site, env gate, tests, example, and the packages/inference relocation after #3595) into one commit for a clean rebase onto current main.

- Raise the `@qvac/model-fit` floor to ^0.8.0, the first release built against the qvac-fabric#214 memory-reporting fix. Verified on the M4 Pro regression fixtures: the honest budget (total - wired - compressor) now drives the verdicts, the idle-machine false negative stays conservative, and the true positives hold. - Classify `load_mode` in place of the removed `no_mmap` (renamed on main by #4078): without it, any load setting the mode would be refused as an unclassifiable key and produce no verdict. - Remove the QVAC_ADVISORY_MODEL_FIT gate: the check now runs on every completion/embedding llama.cpp load. The env schema entry, the enabled seam, and the inertness test go with it. - Reword the resident-reserve and CPU-refusal rationale for 0.8.0: the fit child now sees system-wide wired memory, so the reserve covers only the unwired remainder and deliberately errs conservative.

Review finding from a local verification run: 'flash-attn' is a completion schema key (since split-mode 'tensor' landed) but was in none of the request builder's partitions, so any completion load setting it refused the check and produced no verdict — the safe direction, but silently switching the check off for a memory-load-bearing setting, and with it every 'tensor' split load, which requires flash attention. model-fit's allowlist already accepts the key, so forward it as evidence.

opaninakuffo

advisory llama.cpp fit check on every completion/embedding load, fail-open, no public API. partitioning + supervisor look careful; the flash-attn classify follow-up was the right catch.

title still says "opt-in" after 2d70d39 dropped QVAC_ADVISORY_MODEL_FIT — changelog will ship that wording. rest is nits on the example.

- Match the examples/ log style: '▸' for the reprinted advisory lines and a quickstart-style 'console.error(✖', ...)' catch. - Correct the stale unload rationale: since the resident reserve and the 0.8.0 system-wide wired budget, leaving the first model loaded WOULD shift the second verdict; the unload keeps the demo comparable to the fixture tables.

lauripiisang

Implementation is good; placement seems wrong — the SDK already has a public API for this question and this PR doesn't use it:

assessModelFit exists: client packages/sdk/src/client/api/assess-model-fit.ts, handler packages/inference/src/registry.ts:78, engine packages/inference/src/resources/model-fit/ (vs. this PR's
src/model-fit/).
It already returns verdict/basis/budget/estimate/estimatorVersion with a pluggable estimator registry (estimators/{llm,whisper}.ts) — where a native probe would slot in.
This PR's verdict has no return path at all: examples/advisory-model-fit.ts:83 reads it via log.message.includes('[advisory-fit:').
And it invents a second margin policy — 1024 MiB + Σ(resident bytes) (advisory-fit.ts:220) vs interactive-v1's 20%/2 GiB — so the two can disagree on the same model with nothing saying which wins.

Other two blockers: NOTICE missing bare-runtime + transitives (qv-notice-generate); probably inert in bare-pack bundles (build one and confirm).

Rest is minor (opt-out flag, log level, margin constant, tail trim).

…ail, NOTICE - Restore QVAC_ADVISORY_MODEL_FIT as an operator opt-out, default on (0/false/off/no disable). Removing the flag fixed the accuracy story but left no switch for a load-heavy startup path or a runtime where the disposable child cannot spawn. - Always send marginMiB (base + reserve) instead of omitting it at zero reserve: the omitted case relied on the addon default matching ADVISORY_FIT_BASE_MARGIN_MIB — two sources of truth that would diverge silently. - Log unsupported-load refusals at debug: every non-llama and mobile load takes that path, and at info it tagged every whisper/tts/ocr load. Child failures stay at info. - Keep the stderr tail from opening mid-codepoint: after the byte trim, walk forward past UTF-8 continuation bytes. - Regenerate packages/inference/NOTICE: bare-runtime is a new third-party prod dependency and pulls bare-subprocess plus a platform binary package; the regen also restores missing model attributions.

…fit check Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…elInfo The verdict was only reachable by substring-matching an engine log line, so nothing could consume or assert it. `loadModel` now keeps the outcome, stores it on the registry entry, and `getLoadedModelInfo` returns it as `fitProbe`. Also collapses the two near-identical fit namespaces: the native probe moves from `src/model-fit/` to `src/resources/model-fit/native-probe/`, beside the `assessModelFit` estimators, and its outcome now carries `basis` and `estimatorVersion` like the pre-download API. The headroom policy it applies is named `native-probe-v1` with the precedence against `interactive-v1` written down. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Only the new `LoadedModelInfo.fitProbe` block. A full `contract:export` on this machine also rewrites three unrelated `type: [...]` unions as `anyOf`, which is local toolchain drift, not this change: `contract:check` reports the same staleness at the parent commit with a clean tree, where CI passes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Generated from the committed contract by scripts/generate.py. Additive only: the NativeProbeFit models and the LoadedModelInfo field. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Back out the public exposure of the probe verdict: getLoadedModelInfo no longer returns fitProbe, the contract schema entry is removed, and the example reads the log stream again. The internal scoping stays — the probe lives under resources/model-fit/native-probe/ beside the assessModelFit estimators, its outcome carries basis/estimatorVersion, and loadModel still stores it on the registry entry for internal use. Wiring it into a public API is a separate decision once the shape is confirmed.

9196c5d removed the fitProbe schema entry but left the generated NativeProbeFit models behind, so the sdk-python contract check failed. Regenerated from the committed contract; deletions only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

simon-iribarren mentioned this pull request Aug 24, 2026

QVAC-22630 chore[skiplog]: backmerge release model-fit 0.6.0 — changelog #4016

Merged

simon-iribarren force-pushed the feat/qvac-22629-advisory-fit-supervisor branch from 801f318 to be4a88b Compare August 31, 2026 11:46

simon-iribarren mentioned this pull request Aug 31, 2026

QVAC-24114 fix: fail the load when the model cannot decode #4039

Closed

simon-iribarren marked this pull request as ready for review September 1, 2026 14:03

simon-iribarren requested review from a team as code owners September 1, 2026 14:03

simon-iribarren added 2 commits September 2, 2026 12:43

simon-iribarren force-pushed the feat/qvac-22629-advisory-fit-supervisor branch from fff2b07 to 2d70d39 Compare September 3, 2026 09:22

simon-iribarren added 5 commits September 3, 2026 11:25

Merge branch 'main' into feat/qvac-22629-advisory-fit-supervisor

300ba6b

Merge branch 'main' into feat/qvac-22629-advisory-fit-supervisor

468aef1

Merge branch 'main' into feat/qvac-22629-advisory-fit-supervisor

1704bce

Merge branch 'main' into feat/qvac-22629-advisory-fit-supervisor

df306ef

opaninakuffo reviewed Sep 4, 2026

View reviewed changes

Comment thread packages/sdk/examples/advisory-model-fit.ts Outdated

Comment thread packages/sdk/examples/advisory-model-fit.ts Outdated

simon-iribarren changed the title ~~QVAC-22629 feat: run an opt-in advisory llama.cpp fit check before loadModel~~ QVAC-22629 feat: run an advisory llama.cpp fit check before loadModel Sep 4, 2026

simon-iribarren added 2 commits September 4, 2026 16:39

Merge branch 'main' into feat/qvac-22629-advisory-fit-supervisor

eb2aa7c

Merge branch 'main' into feat/qvac-22629-advisory-fit-supervisor

c6bccf6

opaninakuffo previously approved these changes Sep 4, 2026

View reviewed changes

lauripiisang requested changes Sep 4, 2026

View reviewed changes

simon-iribarren added 2 commits September 4, 2026 19:24

Merge branch 'main' into feat/qvac-22629-advisory-fit-supervisor

adfa94f

simon-iribarren dismissed opaninakuffo’s stale review via e0fa806 September 4, 2026 18:43

simon-iribarren added 3 commits September 6, 2026 11:45

Merge branch 'main' into feat/qvac-22629-advisory-fit-supervisor

e507376

Merge branch 'main' into feat/qvac-22629-advisory-fit-supervisor

23d57c9

Merge branch 'main' into feat/qvac-22629-advisory-fit-supervisor

bea852b

Merge branch 'main' into feat/qvac-22629-advisory-fit-supervisor

d782a9e

arun-mani-j requested changes Sep 8, 2026

View reviewed changes

Comment thread packages/inference/src/resources/model-fit/native-probe/create-llama-fit-request.ts

Comment thread packages/inference/test/model-fit-run-isolated-fit.test.ts Outdated

QVAC-22629 fix: classify projection_model_src as unsupported for the …

0f047a6

…fit check Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

arun-mani-j previously approved these changes Sep 8, 2026

View reviewed changes

opaninakuffo previously approved these changes Sep 8, 2026

View reviewed changes

Merge branch 'main' into feat/qvac-22629-advisory-fit-supervisor

7f7ecc1

simon-iribarren requested a review from lauripiisang September 9, 2026 10:18

simon-iribarren dismissed stale reviews from opaninakuffo and arun-mani-j via 7c42a61 September 9, 2026 11:32

simon-iribarren changed the title ~~QVAC-22629 feat: run an advisory llama.cpp fit check before loadModel~~ QVAC-22629 feat[api]: run an advisory llama.cpp fit check before loadModel Sep 9, 2026

simon-iribarren and others added 3 commits September 9, 2026 13:44

QVAC-22629 chore: regenerate the Python client for fitProbe

12c5cd5

Generated from the committed contract by scripts/generate.py. Additive only: the NativeProbeFit models and the LoadedModelInfo field. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

simon-iribarren changed the title ~~QVAC-22629 feat[api]: run an advisory llama.cpp fit check before loadModel~~ QVAC-22629 feat: run an advisory llama.cpp fit check before loadModel Sep 9, 2026

lauripiisang previously approved these changes Sep 9, 2026

View reviewed changes

simon-iribarren and others added 2 commits September 9, 2026 15:04

Merge branch 'main' into feat/qvac-22629-advisory-fit-supervisor

319f7a0

simon-iribarren dismissed lauripiisang’s stale review via 1905a08 September 9, 2026 13:06

opaninakuffo approved these changes Sep 9, 2026

View reviewed changes

Merge branch 'main' into feat/qvac-22629-advisory-fit-supervisor

15c535a

lauripiisang approved these changes Sep 9, 2026

View reviewed changes

simon-iribarren added 4 commits September 9, 2026 17:11

Merge branch 'main' into feat/qvac-22629-advisory-fit-supervisor

1bfc933

Merge branch 'main' into feat/qvac-22629-advisory-fit-supervisor

34e10c9

Merge branch 'main' into feat/qvac-22629-advisory-fit-supervisor

51c6914

Merge branch 'main' into feat/qvac-22629-advisory-fit-supervisor

b4cb5f6

simon-iribarren merged commit 4fa007c into main Sep 9, 2026
33 checks passed

simon-iribarren deleted the feat/qvac-22629-advisory-fit-supervisor branch September 9, 2026 16:00

Victor-Rodzko mentioned this pull request Sep 10, 2026

QVAC-24907 fix: keep the desktop-only fit subprocess out of mobile bundles #4392

Merged

opaninakuffo mentioned this pull request Sep 10, 2026

chore[skiplog|notask]: backmerge release-sdk-0.19.1 — version bump, changelog, NOTICE #4397

Merged

Navigation Menu

QVAC-22629 feat: run an advisory llama.cpp fit check before loadModel - #4010

QVAC-22629 feat: run an advisory llama.cpp fit check before loadModel#4010
simon-iribarren merged 30 commits into
mainfrom
feat/qvac-22629-advisory-fit-supervisor

simon-iribarren commented Aug 23, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 23, 2026 •

edited

Loading

Uh oh!

socket-security Bot commented Aug 26, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 26, 2026 •

edited

Loading

Uh oh!

opaninakuffo left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

lauripiisang left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

simon-iribarren commented Aug 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎯 What problem does this PR solve?

📝 How does it solve it?

🧪 How was it tested?

Uh oh!

github-actions Bot commented Aug 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

License compliance — clean

NOTICE presence (advisory)

Uh oh!

socket-security Bot commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review Status

Uh oh!

opaninakuffo left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

lauripiisang left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

simon-iribarren commented Aug 23, 2026 •

edited

Loading

github-actions Bot commented Aug 23, 2026 •

edited

Loading

socket-security Bot commented Aug 26, 2026 •

edited

Loading

github-actions Bot commented Aug 26, 2026 •

edited

Loading

opaninakuffo left a comment •

edited

Loading

lauripiisang left a comment •

edited

Loading