Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions 00-specs.sh
Original file line number Diff line number Diff line change
Expand Up @@ -64,10 +64,14 @@ du -sh "$HOME/ollama-models" 2>/dev/null || true
# 30-models.sh uses. Previously this section read an undefined $TIER, which
# under `set -u` aborted the report right here -- and its hardcoded copy had
# drifted into recommending models that are never downloaded.
if (( BUDGET_OK )) && plan_for_budget "$FIT_TOTAL_MB" "$MOE_OFFLOAD_MB"; then
if (( BUDGET_OK )) && plan_for_budget "$FIT_TOTAL_MB" "$MOE_OFFLOAD_MB" "$GPU_CC"; then
c_info "Recommended plan for tier: $PLAN_TIER (usable budget ${FIT_TOTAL_MB} MB)"
printf ' %s\n\n' "$PLAN_NOTE"
printf ' Runtime: %s\n\n' "$PLAN_RUNTIME"
printf ' Runtime: %s\n' "$PLAN_RUNTIME"
# Printed as its own line rather than folded into the runtime sentence: it is
# a judgement about an alternative this rig does not use, and it is the line
# most likely to be wrong first, as new hardware ships.
printf ' %s\n\n' "$PLAN_VLLM"
echo " Models ./30-models.sh will fetch for this machine:"
echo
printf ' %-2s %-32s %-14s %8s %-6s %-7s %-11s %s\n' \
Expand Down
2 changes: 1 addition & 1 deletion 30-models.sh
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ c_info "Weight budget: ${FIT_TOTAL_MB} MB split / ${FIT_SINGLE_MB} MB single-GPU
# A tier that names a model missing from lib/catalog.sh is a bug in the tier
# table, and it stops the run here rather than resolving to nothing after a
# long search.
plan_for_budget "$FIT_TOTAL_MB" "$MOE_OFFLOAD_MB" \
plan_for_budget "$FIT_TOTAL_MB" "$MOE_OFFLOAD_MB" "$GPU_CC" \
|| die "${PLAN_ERROR:-could not resolve a plan for this budget}"
c_info "Tier: $PLAN_TIER"

Expand Down
7 changes: 7 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,13 @@ If you see LiteLLM in your service list, the probe failed — see
Ollama was evaluated and rejected; the reasoning is in
[TUNING.md § Rejected](TUNING.md#rejected-with-reasons).

vLLM was too, and `00-specs.sh` re-decides it for **your** GPU rather than repeating
the verdict for this one. Whether vLLM is worth your time turns on compute capability
— native FP8 starts at Ada, NVFP4 at Blackwell — and on the fact that it keeps the
whole model in VRAM, with no `--n-cpu-moe` equivalent, so a machine with far more RAM
than VRAM gives up its largest models by switching. The report names the capability it
read, so you can check the claim instead of believing it.

## Run order

```bash
Expand Down
22 changes: 21 additions & 1 deletion TUNING.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,27 @@ reaches equilibrium temperature.
- **LiteLLM** — unnecessary once llama-server proved to speak Anthropic natively.
Retained only as an automatic fallback.
- **vLLM** — its wins are prefix caching (already have 90×) and continuous batching
(single user). Tensor-parallel without NVLink would run over PCIe.
(single user). Tensor-parallel without NVLink would run over PCIe. Re-examined
2026-08-12 against the hardware rather than the workload, and the hardware is the
stronger argument of the two:
- **No expert offload.** vLLM holds the whole model in VRAM; there is no
`--n-cpu-moe` equivalent. A 118B MoE that llama.cpp runs out of the 109 GB of
system RAM here does not run slower under vLLM, it does not run. On this box that
is the difference between having a frontier-class model and not.
- **sm_86 has no native FP8 and no NVFP4.** W8A8 FP8 starts at Ada (sm_89); NVFP4 is
Blackwell (sm_100+). An FP8 checkpoint still *loads* on Ampere, dequantized
weight-only through FP8 Marlin — the memory saving without the arithmetic. INT4 and
INT8 Marlin do work, and that is the only vLLM path here.
- **The two cards are not symmetric.** ~16376 and ~15352 MiB free; whichever drives
the desktop loses ~1 GB. Tensor parallel sizes to the smaller and assumes they
match.

What would change the answer: one Blackwell card with ≥48 GB. Then NVFP4 plus
Poolside's DFlash speculative decoding — which is *not* in mainline llama.cpp — makes
vLLM the better stack and llama.cpp the compromise. Until then the recommendation is
computed rather than assumed: `lib/runtime.sh` derives it from the detected compute
capability, and `00-specs.sh` prints the verdict with the capability named so the
claim can be checked rather than believed.
- **Speculative decoding** — at 120 t/s the MoE is bandwidth-saturated; a draft model
would consume VRAM better spent on KV cache.
- **`--split-mode row`** — fails to load without P2P.
Expand Down
32 changes: 25 additions & 7 deletions lib/models.sh
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,9 @@ _LLMRIG_MODELS_SH=1
# shellcheck source=lib/catalog.sh
source "$(dirname "${BASH_SOURCE[0]}")/catalog.sh"

# shellcheck source=lib/runtime.sh
source "$(dirname "${BASH_SOURCE[0]}")/runtime.sh"

# A quant preference is an ORDERED alternation: "IQ3_XXS|Q3_K_S" means "take
# IQ3_XXS if the repo has it, otherwise Q3_K_S".
#
Expand Down Expand Up @@ -115,10 +118,18 @@ quant_include_pattern() {
# report still advertised models that were never downloaded, and in two cases
# never existed.
#
# plan_for_budget <fit_total_mb> [moe_offload_mb]
# plan_for_budget <fit_total_mb> [moe_offload_mb] [gpu_cc]
#
# Sets PLAN_TIER, PLAN_SEARCH_{1,2,3}, PLAN_Q_{1,2,3}, PLAN_RUNTIME, PLAN_NOTE,
# and PLAN_MOE_NOTE.
# PLAN_MOE_NOTE and PLAN_VLLM.
#
# gpu_cc is the GPU compute capability ("8.6"), and it is a separate argument
# from the budget because it answers a separate question. The tier decides
# WHICH MODELS fit; the capability decides WHICH RUNTIME can exploit them. The
# two used to be conflated -- every tier carried a hardcoded opinion about
# vLLM, sized on VRAM alone -- and on a 31 GB Ampere pair that produced advice
# that was confidently wrong. See lib/runtime.sh. Omitted, PLAN_VLLM says the
# capability was not assessed rather than guessing at one.
#
# Thresholds are on FIT_TOTAL_MB -- usable VRAM after the KV reserve -- not on
# installed VRAM. Sizing off installed memory is what once recommended a 49 GB
Expand All @@ -129,7 +140,7 @@ quant_include_pattern() {
# budget decision rather than a property of the model: the same 30B MoE wants
# IQ3 on a 8 GB card and Q6_K on a 48 GB one.
plan_for_budget() {
local fit_total="$1" moe_offload="${2:-0}"
local fit_total="$1" moe_offload="${2:-0}" gpu_cc="${3:-}"
# Cleared up front so a failure two calls ago cannot be read back as this
# call's diagnosis. Read by 00-specs.sh and 30-models.sh, which shellcheck
# cannot see from inside this file.
Expand All @@ -141,7 +152,7 @@ plan_for_budget() {
PLAN_ID_1="qwen3-coder-30b"; PLAN_Q_1="IQ3_XXS|Q3_K_S"
PLAN_ID_2="qwen3-4b"; PLAN_Q_2="Q5_K_M"
PLAN_ID_3="qwen3-1.7b"; PLAN_Q_3="Q8_0"
PLAN_RUNTIME="llama.cpp + llama-swap. vLLM is not viable -- it needs the whole model resident."
PLAN_RUNTIME="llama.cpp + llama-swap."
PLAN_NOTE="Claude Code needs 32k context minimum, and the KV cache at that length
will take a large share of this card. The primary pick is a MoE precisely
because only ~3B params are active per token, so CPU expert offload stays
Expand Down Expand Up @@ -170,7 +181,7 @@ plan_for_budget() {
PLAN_ID_1="qwen3-coder-30b"; PLAN_Q_1="Q6_K|Q5_K_M"
PLAN_ID_2="qwen3-32b"; PLAN_Q_2="Q5_K_M"
PLAN_ID_3="devstral-small"; PLAN_Q_3="Q5_K_M"
PLAN_RUNTIME="llama.cpp + llama-swap now; vLLM is worth measuring once you settle on one model."
PLAN_RUNTIME="llama.cpp + llama-swap."
PLAN_NOTE="Enough headroom to spend it on quantization quality rather than more
parameters -- a higher quant of a right-sized model beats a squeezed larger one."

Expand All @@ -179,10 +190,17 @@ plan_for_budget() {
PLAN_ID_1="llama-3.3-70b"; PLAN_Q_1="IQ4_XS|Q4_K_S"
PLAN_ID_2="qwen3-coder-30b"; PLAN_Q_2="Q6_K"
PLAN_ID_3="qwen3-32b"; PLAN_Q_3="Q5_K_M"
PLAN_RUNTIME="vLLM with prefix caching is worth evaluating at this scale; llama.cpp + llama-swap still works."
PLAN_RUNTIME="llama.cpp + llama-swap."
PLAN_NOTE="Large enough to run a frontier-class open model resident."
fi

# The runtime advice, computed rather than hardcoded per tier. Kept out of
# PLAN_RUNTIME so a caller can print or suppress it independently: it is a
# judgement about an alternative, where PLAN_RUNTIME states what this rig
# actually uses.
# shellcheck disable=SC2034 # documented return channel, read by callers
PLAN_VLLM="$(vllm_advice "$gpu_cc" "$fit_total" "$moe_offload")"

# With plenty of system RAM, a MoE far larger than VRAM is viable via
# --n-cpu-moe, because only the active experts must be resident.
if (( moe_offload > 60000 )); then
Expand Down Expand Up @@ -223,7 +241,7 @@ plan_for_budget() {

# These are the function's return values -- callers read them. Exporting
# states that contract (and is what detect_hw already does for its outputs).
export PLAN_TIER PLAN_RUNTIME PLAN_NOTE PLAN_MOE_NOTE \
export PLAN_TIER PLAN_RUNTIME PLAN_NOTE PLAN_MOE_NOTE PLAN_VLLM \
PLAN_ID_1 PLAN_ID_2 PLAN_ID_3 \
PLAN_SEARCH_1 PLAN_SEARCH_2 PLAN_SEARCH_3 \
PLAN_REPO_1 PLAN_REPO_2 PLAN_REPO_3 \
Expand Down
130 changes: 130 additions & 0 deletions lib/runtime.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
#!/usr/bin/env bash
# Which serving runtime to recommend, and on what evidence.
#
# THE BUG THIS FIXES
#
# plan_for_budget decided whether to suggest vLLM from VRAM alone. At 45 GB and
# up it said "worth evaluating at this scale"; below 9 GB it said "not viable".
# Memory is the wrong axis to decide it on by itself.
#
# vLLM's advantage over llama.cpp is throughput from resident weights and fast
# low-precision kernels. WHICH kernels exist is a property of the GPU's compute
# capability, not of how much memory is attached to it. A pair of RTX A4000s is
# 31 GB -- enough to read as "worth evaluating" -- and sm_86, which means no
# native FP8 and no NVFP4. The advice was confidently wrong on this very rig.
#
# The second thing memory alone cannot see: vLLM keeps the whole model in VRAM.
# It has no equivalent of --n-cpu-moe, so a 118B MoE that llama.cpp runs
# comfortably with its experts in system RAM does not run slower under vLLM, it
# does not run. On a machine with far more RAM than VRAM that is the difference
# between a usable model and none.
#
# Everything here is pure: a compute capability and two numbers in, a sentence
# out. No nvidia-smi, no network.
#
# shellcheck shell=bash

[[ -z "${_LLMRIG_RUNTIME_SH:-}" ]] || return 0
_LLMRIG_RUNTIME_SH=1

# --- compute capability -----------------------------------------------------

# runtime_cc_x10 <compute_cap> -- "8.6" as 86, so bash can compare it.
#
# Status 1 for anything unparseable, INCLUDING the empty string. Detection can
# fail, and a missing capability must not silently compare as zero and read as
# an ancient card -- "we could not tell" and "it is too old" want different
# sentences.
runtime_cc_x10() {
local cc="${1:-}" whole frac
[[ "$cc" =~ ^[0-9]+(\.[0-9]+)?$ ]] || return 1
whole="${cc%.*}"
frac=0
[[ "$cc" == *.* ]] && frac="${cc#*.}"
printf '%d' "$(( whole * 10 + ${frac:0:1} ))"
}

# vllm_kernels <compute_cap> -- what low-precision arithmetic this GPU can
# actually do under vLLM. One of:
#
# unknown capability could not be determined
# none below Volta; vLLM does not support the card at all
# int integer quantization only (Volta / Turing)
# int-fp8-weight Ampere: INT4/INT8 Marlin, and FP8 checkpoints LOAD but are
# dequantized weight-only (W8A8 becomes W8A16) -- you get the
# memory saving, not the arithmetic
# fp8 Ada / Hopper: native FP8 W8A8
# fp4 Blackwell: NVFP4
#
# Boundaries, so they can be checked rather than trusted:
# sm_70 Volta, sm_75 Turing, sm_80/86 Ampere, sm_89 Ada, sm_90 Hopper,
# sm_100+ Blackwell.
# FP8 W8A8 is Ada and up; on Ampere an FP8 checkpoint runs through FP8 Marlin
# as weight-only. https://docs.vllm.ai/en/stable/quantization/fp8.html
# NVFP4 is Blackwell (SM100+); below that it is weight-only at best.
vllm_kernels() {
local cc10
cc10="$(runtime_cc_x10 "${1:-}")" || { printf 'unknown'; return 0; }
if (( cc10 >= 100 )); then printf 'fp4'
elif (( cc10 >= 89 )); then printf 'fp8'
elif (( cc10 >= 80 )); then printf 'int-fp8-weight'
elif (( cc10 >= 70 )); then printf 'int'
else printf 'none'
fi
}

# --- the recommendation -----------------------------------------------------

# vllm_advice <compute_cap> <fit_total_mb> [moe_offload_mb]
#
# One sentence for the specs report. Deliberately says what is true of THIS
# card rather than of vLLM in general, and names the capability so a reader can
# check the claim against NVIDIA's table instead of believing it.
vllm_advice() {
local cc="${1:-}" fit="${2:-0}" offload="${3:-0}" kernels
[[ "$fit" =~ ^[0-9]+$ ]] || fit=0
[[ "$offload" =~ ^[0-9]+$ ]] || offload=0
kernels="$(vllm_kernels "$cc")"

# Emitted clause by clause rather than as one long format string, so each
# sentence stays readable at the width the rest of this repo is written to.
case "$kernels" in
unknown)
printf 'vLLM: not assessed -- the GPU compute capability could not be read,'
printf ' and it is what decides whether the low-precision kernels vLLM'
printf ' wins with exist here.'
return 0 ;;
none)
printf 'vLLM: not supported on compute capability %s' "${cc:-unknown}"
printf ' (it needs 7.0 or newer).'
return 0 ;;
esac

# Weights must be resident. Where system RAM dwarfs VRAM that is the whole
# argument, and it is worth making before any kernel talk.
if (( fit > 0 && offload > fit * 2 )); then
printf 'vLLM: it holds the whole model in VRAM, with no --n-cpu-moe'
printf ' equivalent, so the %s GB of system RAM here buys it nothing --' "$(( offload / 1024 ))"
printf ' a MoE larger than %s GB runs under llama.cpp and not at all' "$(( fit / 1024 ))"
printf ' under vLLM. '
fi

case "$kernels" in
int)
printf 'On compute capability %s there is no FP8 path at all, so the' "$cc"
printf ' gain over llama.cpp would be batching this rig does not do.' ;;
int-fp8-weight)
printf 'On compute capability %s (Ampere) there is no native FP8 and no' "$cc"
printf ' NVFP4: INT4/INT8 Marlin work, and an FP8 checkpoint loads only'
printf ' as weight-only W8A16, which saves memory without speeding'
printf ' arithmetic up. Worth revisiting on Ada or newer.' ;;
fp8)
printf 'Compute capability %s has native FP8, so vLLM is worth measuring' "$cc"
printf ' once you have settled on one model -- its remaining costs here'
printf ' are a cold start per model and one model per process.' ;;
fp4)
printf 'Compute capability %s has NVFP4, which is where vLLM pulls' "$cc"
printf ' clearly ahead. Worth measuring against llama.cpp on your'
printf ' actual model.' ;;
esac
}
Loading