hostfit: size ollama loads the way llama.cpp's fit does, and recommend only what stays resident (#1347, #1337) - #1378
Merged
Conversation
…d only what stays resident (#1347, #1337) The recommendation and the serve tuning priced an ollama load as weights + KV at half of f16 + a one-point overhead (1024 MiB + 40 MiB per GB), with a ×3.0 calibration standing in for everything the sum left out. On a 24 GB card that predicted 11 % spill for qwen3.8-27b MTP-Q4 at 200k tokens; llama.cpp's fit put 13 of 66 layers in system RAM and decode ran 28× slower at 176k (waired#1357). Estimate (proto/hostfit/estimate.go): the same terms the fit adds, from the GGUF header facts the catalog now carries (catalog.GGUFLayout, derived by `catalog-tool layout`, checked by `--check`): - weights llama.cpp loads (tensor table, minus the next-token blocks a tag does not draft with, plus the projector and the device copy of a tied embedding), less the input layer it keeps in system RAM - KV at ggml's block factors (q8_0 34/64, q4_0 18/64) - the recurrent state of hybrid models, × (1 + draft tokens) - the compute buffer: 80 MiB per 512 of ubatch, the attention mask, and the f16 copy of one layer when the cache is quantized - the MTP draft head's KV, compute and setup - the per-process device context and llama.cpp's fit target (1 GiB, or projector + 1 GiB) Against 24 logged loads (CUDA 19, Metal 3, Vulkan 2, ollama 0.33.3) the estimate is within −40..+348 MiB of the fit's projection; the 27B and 35B builds within 81 MiB. OllamaPredictPlacement walks the layers the way the fit fills them (experts first on a mixture of experts) and matches the logged 53/63/61/66/66. Rung rule 2 reads that placement's CPU weight-byte share (decision 3 of the record). The recommendation needs the whole ~200k window resident (decision 10 of docs/decisions/20260913/2355-catalog-variant-kv-and-residency-rulings.md); the KV type is priced through OllamaDefaultKVCacheType (q8_0 until #1348). Capacity also refuses when the host-resident weights alone exceed the RAM budget. Presentation itemises the window figure (device_weights_mb, kv_cache_mb) so a surface prints weights, KV cache and engine overhead separately, and carries gpu_layers / total_layers. vLLM: the gpt-oss KV figures were 3× the derivation (73728 → 24576, 98304 → 36864), and max-model-len reserves the measured 1.28 GiB activation peak, which the old ×1.15 + 1 GiB sum over-promised by 0.2–1.2 GiB on a 24 GB card. The bands that move, the tests this inverts and the contract table are in the PR body. Signatures are unchanged; the recalibrated terms are unexported so a later recalibration is a tag, not a const-value change. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013dwDgbCKP8ac4MvyVTCoB7 Signed-off-by: gen16k <gen16k@users.noreply.github.com>
gen16k
force-pushed
the
fix/1347-vram-sizing-matches-fit-proto
branch
from
September 13, 2026 19:29
57b355e to
502f2f0
Compare
This was referenced Sep 13, 2026
gen16k
added a commit
that referenced
this pull request
Sep 15, 2026
…as not used is its own verdict, and the spill suffix says what is in system RAM (#1337, #1330, #71) After a load the agent judged spill from /api/ps, which is ollama's own parse of the buffer lines and leaves out the MTP draft context and the CPU_Mapped weights: an MTP build using 21.2 GB read as 14.97 GB, and a size_vram of 0 — a backend that did not engage at all — read as a spill to shrink the window for (#71). And the decision reason said "fully GPU-resident" for a placement nothing could falsify (#1330). The witness is now the engine log (internal/runtime.ParseLlamaPlacement): the last runner's `offloaded N/M layers`, its device / host model buffers, the KV cache type it allocated, and the fit's projected / shortfall lines — accepted only when the runner's --model and n_ctx match the tuning being verified. Spill is measured in bytes against the placement hostfit predicted (#1378), with one layer of tolerance; on a mixture of experts the fit moves expert tensors while still reporting every layer offloaded and CPU_Mapped reports the whole file, so the measure is the device buffer's drop from the planned figure. `offloaded 0/M` (or size_vram == 0 when the log says nothing) is tuningGPUNotEngaged: Degraded with a warning, no shrink and no restart, the window still declared. Without a witness the check abstains. Tuning plans the rung and the placement through the calibrated estimator and says what it predicted: "predicted to hold N of M layers in VRAM; X GB of input embedding weights stay in system RAM". The CLI and tray suffix becomes "· 2.5 GB in system RAM" — the fit moves layers, not a KV cache — with the legend, the download note, the docs-site pages and the TRANSLATION.md rulings changed together (owner decision 2026-09-14, docs/decisions/20260914/1200 decision 9). scoring.KVFactorQ4_0 (0.25) is gone; the tuning prices through hostfit.OllamaKVCacheFactor. The four producedInProto exemptions #1378 left for these writers are deleted. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013dwDgbCKP8ac4MvyVTCoB7 Signed-off-by: gen16k <gen16k@users.noreply.github.com>
gen16k
added a commit
that referenced
this pull request
Sep 15, 2026
…as not used is its own verdict, and the spill suffix says what is in system RAM (#1337, #1330, #71) (#1384) * inference: the placement witness is llama.cpp's own log, a GPU that was not used is its own verdict, and the spill suffix says what is in system RAM (#1337, #1330, #71) After a load the agent judged spill from /api/ps, which is ollama's own parse of the buffer lines and leaves out the MTP draft context and the CPU_Mapped weights: an MTP build using 21.2 GB read as 14.97 GB, and a size_vram of 0 — a backend that did not engage at all — read as a spill to shrink the window for (#71). And the decision reason said "fully GPU-resident" for a placement nothing could falsify (#1330). The witness is now the engine log (internal/runtime.ParseLlamaPlacement): the last runner's `offloaded N/M layers`, its device / host model buffers, the KV cache type it allocated, and the fit's projected / shortfall lines — accepted only when the runner's --model and n_ctx match the tuning being verified. Spill is measured in bytes against the placement hostfit predicted (#1378), with one layer of tolerance; on a mixture of experts the fit moves expert tensors while still reporting every layer offloaded and CPU_Mapped reports the whole file, so the measure is the device buffer's drop from the planned figure. `offloaded 0/M` (or size_vram == 0 when the log says nothing) is tuningGPUNotEngaged: Degraded with a warning, no shrink and no restart, the window still declared. Without a witness the check abstains. Tuning plans the rung and the placement through the calibrated estimator and says what it predicted: "predicted to hold N of M layers in VRAM; X GB of input embedding weights stay in system RAM". The CLI and tray suffix becomes "· 2.5 GB in system RAM" — the fit moves layers, not a KV cache — with the legend, the download note, the docs-site pages and the TRANSLATION.md rulings changed together (owner decision 2026-09-14, docs/decisions/20260914/1200 decision 9). scoring.KVFactorQ4_0 (0.25) is gone; the tuning prices through hostfit.OllamaKVCacheFactor. The four producedInProto exemptions #1378 left for these writers are deleted. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013dwDgbCKP8ac4MvyVTCoB7 Signed-off-by: gen16k <gen16k@users.noreply.github.com> * runtime: the placement parser and its fixtures read CRLF logs (Windows) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013dwDgbCKP8ac4MvyVTCoB7 Signed-off-by: gen16k <gen16k@users.noreply.github.com> --------- Signed-off-by: gen16k <gen16k@users.noreply.github.com> Co-authored-by: gen16k <gen16k@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1347
Refs #1337, #1346, #1348, #1375, waired-ai/waired#1357
L113 / L119 の proto 側。ollama のロードを llama.cpp の
--fitと同じ項で見積り、推奨は「~200k のコンテキストウィンドウごと VRAM に載る」ものだけにする（決定記録 2355 の決定 10）。agent 側（ロード後の証拠をエンジンログで読む、tuning、文言）は次の PR。何が変わるか
proto/hostfit/estimate.go: 重み（GGUF のテンソル表。draft しないタグは nextn を除く。projector と tied embedding の device 側コピーを足す）− 入力層 + KV（ggml のブロック係数 q8_0 34/64、q4_0 18/64）+ 再帰状態 ×(1 + draft) + compute（ubatch 512 あたり 80 MiB + マスク + 量子化 KV の f16 1 層分）+ MTP draft + プロセスのコンテキスト + fit の余裕（1 GiB / projector + 1 GiB）。実測 24 本（CUDA 19、Metal 3、Vulkan 2、ollama 0.33.3）に対し −40〜+348 MiB、27B / 35B は 81 MiB 以内（estimate_test.go）。OllamaPredictPlacement: fit と同じ順（MoE は expert から）で層を歩く。実測の 53/63/61/66/66 と一致。rung 規則 2 は CPU に置く重みのバイト比を読む（OllamaSpillCalibration×3.0 は不使用に）。OllamaRecommendModelFor(m, v, h, kvType): 第 3 条件が「200k の必要量 ≤ VRAM 予算」。KV 型はOllamaDefaultKVCacheType（q8_0。agent: KV cache defaults to q4_0 with a per-model ladder, the chosen variant and KV type are honoured, tray shows the default variant (L107) #1348 が反転）。insufficient_ram）。gguf配置（18 variant、catalog-tool layout --bundled --checkで登録値と突合）、host_resident_weight_gb、gpt-oss のkv_bytes_per_token_fp16を導出値に（73728 → 24576、98304 → 36864）。qwen3.8-flash-nextは catalog: a community source tag with no digest pin grew from 55 GB to 79 GB upstream; every size, fit and ETA figure is stale #1305 のサイズ訂正まで配置なし（旧項で見積る）。VLLMMaxModelLenが実測のアクティベーション 1.28 GiB を予約（旧式は 24 GB カードの KV プールを 0.2〜1.2 GiB 多く見積っていた）。行の KV 型はVLLMKVCacheType(gpus, requested)（対応カードのときだけ fp8）。device_weights_mb/kv_cache_mb（残りがエンジンのオーバーヘッド）、gpu_layers/total_layers。契約表（additive、
proto-additive-guardOK vs proto/v0.2.70）catalog.Variantgguf*GGUFLayout（block_count,full_attention_layers,recurrent_state_bytes,draft_max_tokens,tensor_bytes,projector_bytes,inline_projector_bytes,nextn_layers,nextn_bytes,repeating_bytes,expert_bytes,tied_output_bytes）catalog-tool layoutが導くhostfit.Presentationdevice_weights_mb,kv_cache_mbhostfit.HostGPUVendor(json:"-")hostfitOllamaEstimateMemory,OllamaMemory,OllamaPredictPlacement,OllamaPlacement,OllamaDeviceCapacityTokens,OllamaKVCacheFactor,OllamaKVBlockFactorQ8_0/Q4_0,OllamaFitTargetMB,OllamaPlannedRungFor,OllamaDeclaresWindowFor,OllamaRecommendModelFor,VLLMRecommendModelOnHostFor,VLLMServesContextFloorFor,VLLMKVCacheTypeOllamaSpillCalibration,OllamaKVFactorQ8_0,OllamaVRAMOverhead*,OllamaMaxContextAtSpill推奨が動く帯（KV q8_0、200k）
#1348（既定 q4_0）で NVIDIA 16 / 20 GB と Mac 16 GB は main と同じ推奨に戻る。
反転したテスト（PR 本文で先に宣言）
proto/hostfit/capacity_test.go8 GB RAM + 2 GB card: tied embedding の device 側コピーを数えるので 4b（8,486 MiB > 8,192）でなく 2b を保持。internal/router/coding_floor_test.go: 期待スピルは ×3.0 の較正値（0.117）でなく fit が CPU に置く重みの比（0.047）。anchor（24 GB）の 1 位は mtp-q4 でなく mtp-q3（mtp-q4 は 13/66 層が CPU）。UMA の roomy は 25 GiB（q8_0 の係数 34/64 で 24 GiB は 103 MiB 不足）。internal/router/uma_tiers_estimated_test.go: 12 GB 4b → 2b、16 GB 9b → 4b、32 GB mtp-q4 → mtp-q3（Metal の fit ログの値）。internal/router/model_picker_test.go16 GB: 推奨できる 35B ビルドは KV 型に依存（q8_0 は無し、q4_0 は mtp-q2）。internal/router/vllm_tuning_test.go/cmd/waired-agent/inference_vllm_tuning_test.go: 1×L4 のクランプ 45,056 → 26,624（fp8 90,112 → 54,272）。internal/setup/modelselect_lightest_test.gounified: 最軽量は 2b でなく 0.8b（6 GB 機は 200k を保持できず最善努力）。cmd/waired-agent/inference_ollama_tuning_test.go/inference_ollama_verify_test.go: 較正値前提の 2 件を新しい比で再固定（agent の判定ロジック自体は次の PR）。HostWeightsMB,TotalLayers×3）は agent 側の書き手が次の PR で入り、そこで削除する。確認
go test ./...（root / proto）、go vet、make verify-cross、golangci-lint 0 件（proto のmodel_size_test.goQF1001 は catalog: add Qwen3.8-Flash-Next, and stamp its renderer on pull (#1192) #1259 からの既存）、protoconsumerOK、proto-additive-guardOK、catalog-tool layout --bundled --checkOK、catalog-tool docs --checkOK、decision-log-guard OK。docs/knowledges/20260914/1230-llamacpp-fit-memory-terms.md）。qwen3.8-flash-next（カタログのサイズ誤り）を拾う。catalog: a community source tag with no digest pin grew from 55 GB to 79 GB upstream; every size, fit and ETA figure is stale #1305 のサイズ訂正で消える。docs-not-needed: ユーザー向けの文言と docs-site は agent 側の次の PR（#1337）で一緒に変える。
translation-not-needed: docs-site に変更なし。
🤖 Generated with Claude Code
https://claude.ai/code/session_013dwDgbCKP8ac4MvyVTCoB7