Decide vLLM on compute capability, not on VRAM

00-specs.sh

@@ -64,10 +64,14 @@ du -sh "$HOME/ollama-models" 2>/dev/null || true
 # 30-models.sh uses. Previously this section read an undefined $TIER, which
 # under `set -u` aborted the report right here -- and its hardcoded copy had
 # drifted into recommending models that are never downloaded.
-if (( BUDGET_OK )) && plan_for_budget "$FIT_TOTAL_MB" "$MOE_OFFLOAD_MB"; then
+if (( BUDGET_OK )) && plan_for_budget "$FIT_TOTAL_MB" "$MOE_OFFLOAD_MB" "$GPU_CC"; then
   c_info "Recommended plan for tier: $PLAN_TIER  (usable budget ${FIT_TOTAL_MB} MB)"
   printf '  %s\n\n' "$PLAN_NOTE"
-  printf '  Runtime: %s\n\n' "$PLAN_RUNTIME"
+  printf '  Runtime: %s\n' "$PLAN_RUNTIME"
+  # Printed as its own line rather than folded into the runtime sentence: it is
+  # a judgement about an alternative this rig does not use, and it is the line
+  # most likely to be wrong first, as new hardware ships.
+  printf '  %s\n\n' "$PLAN_VLLM"
   echo "  Models ./30-models.sh will fetch for this machine:"
   echo
   printf '    %-2s %-32s %-14s %8s %-6s %-7s %-11s %s\n' \

30-models.sh

@@ -127,7 +127,7 @@ c_info "Weight budget: ${FIT_TOTAL_MB} MB split / ${FIT_SINGLE_MB} MB single-GPU
 # A tier that names a model missing from lib/catalog.sh is a bug in the tier
 # table, and it stops the run here rather than resolving to nothing after a
 # long search.
-plan_for_budget "$FIT_TOTAL_MB" "$MOE_OFFLOAD_MB" \
+plan_for_budget "$FIT_TOTAL_MB" "$MOE_OFFLOAD_MB" "$GPU_CC" \
   || die "${PLAN_ERROR:-could not resolve a plan for this budget}"
 c_info "Tier: $PLAN_TIER"
 

README.md

@@ -29,6 +29,15 @@ If you see LiteLLM in your service list, the probe failed — see
 Ollama was evaluated and rejected; the reasoning is in
 [TUNING.md § Rejected](TUNING.md#rejected-with-reasons).
 
+vLLM was too, and `00-specs.sh` re-decides it for **your** GPU rather than repeating
+the verdict for this one. Whether vLLM is worth your time turns on compute capability
+— loading an FP8 checkpoint weight-only starts at Turing, computing in FP8 starts at
+Ada, NVFP4 at Blackwell — and on the fact that it keeps the whole model in VRAM, with
+no `--n-cpu-moe` equivalent, so a machine with far more RAM than VRAM gives up its
+largest models by switching. Capacity gates the enthusiastic verdict too: even NVFP4
+is not a reason to switch on a card that cannot hold the model you want. The report
+names the capability it read, so you can check the claim instead of believing it.
+
 ## Run order
 
 ```bash

TUNING.md

@@ -145,7 +145,37 @@ reaches equilibrium temperature.
 - **LiteLLM** — unnecessary once llama-server proved to speak Anthropic natively.
   Retained only as an automatic fallback.
 - **vLLM** — its wins are prefix caching (already have 90×) and continuous batching
-  (single user). Tensor-parallel without NVLink would run over PCIe.
+  (single user). Tensor-parallel without NVLink would run over PCIe. Re-examined
+  2026-08-12 against the hardware rather than the workload, and the hardware is the
+  stronger argument of the two:
+  - **No expert offload.** vLLM holds the whole model in VRAM; there is no
+    `--n-cpu-moe` equivalent. A 118B MoE that llama.cpp runs out of the 109 GB of
+    system RAM here does not run slower under vLLM, it does not run. On this box that
+    is the difference between having a frontier-class model and not.
+  - **sm_86 has no native FP8 and no NVFP4.** FP8 has two floors, not one, and they
+    are easy to conflate. Native W8A8 arithmetic starts at Ada (sm_89); *loading* an
+    FP8 checkpoint works from sm_75 (Turing) upward, dequantized weight-only through
+    FP8 Marlin as W8A16 — the memory saving without the arithmetic. Ampere sits above
+    the loading floor and below the compute one, so INT4/INT8 Marlin plus weight-only
+    FP8 is the whole vLLM menu here. NVFP4 is Blackwell (sm_100+).
+    ([vLLM FP8 docs](https://docs.vllm.ai/en/stable/features/quantization/fp8/))
+  - **The two cards are not symmetric.** ~16376 and ~15352 MiB free; whichever drives
+    the desktop loses ~1 GB. Tensor parallel sizes to the smaller and assumes they
+    match.
+
+  What would change the answer: one Blackwell card with ≥48 GB. Both halves are load
+  bearing. NVFP4 without the capacity is a faster way to run a model small enough to
+  be the wrong model — vLLM must hold the whole thing resident, so kernels alone never
+  carry the argument. Then NVFP4 plus Poolside's DFlash speculative decoding — which is
+  *not* in mainline llama.cpp — makes vLLM the better stack and llama.cpp the
+  compromise.
+
+  Until then the recommendation is computed rather than assumed: `lib/runtime.sh`
+  derives it from the detected compute capability *and* the weight-resident budget,
+  and `00-specs.sh` prints the verdict with the capability named so the claim can be
+  checked rather than believed. The 48 GB gate there is measured on the budget left
+  for weights after the KV reserve, which makes it stricter than a 48 GB sticker
+  price — deliberate, since nobody here has run a Blackwell box.
 - **Speculative decoding** — at 120 t/s the MoE is bandwidth-saturated; a draft model
   would consume VRAM better spent on KV cache.
 - **`--split-mode row`** — fails to load without P2P.

lib/models.sh

@@ -15,6 +15,9 @@ _LLMRIG_MODELS_SH=1
 # shellcheck source=lib/catalog.sh
 source "$(dirname "${BASH_SOURCE[0]}")/catalog.sh"
 
+# shellcheck source=lib/runtime.sh
+source "$(dirname "${BASH_SOURCE[0]}")/runtime.sh"
+
 # A quant preference is an ORDERED alternation: "IQ3_XXS|Q3_K_S" means "take
 # IQ3_XXS if the repo has it, otherwise Q3_K_S".
 #
@@ -115,10 +118,18 @@ quant_include_pattern() {
 # report still advertised models that were never downloaded, and in two cases
 # never existed.
 #
-#   plan_for_budget <fit_total_mb> [moe_offload_mb]
+#   plan_for_budget <fit_total_mb> [moe_offload_mb] [gpu_cc]
 #
 # Sets PLAN_TIER, PLAN_SEARCH_{1,2,3}, PLAN_Q_{1,2,3}, PLAN_RUNTIME, PLAN_NOTE,
-# and PLAN_MOE_NOTE.
+# PLAN_MOE_NOTE and PLAN_VLLM.
+#
+# gpu_cc is the GPU compute capability ("8.6"), and it is a separate argument
+# from the budget because it answers a separate question. The tier decides
+# WHICH MODELS fit; the capability decides WHICH RUNTIME can exploit them. The
+# two used to be conflated -- every tier carried a hardcoded opinion about
+# vLLM, sized on VRAM alone -- and on a 31 GB Ampere pair that produced advice
+# that was confidently wrong. See lib/runtime.sh. Omitted, PLAN_VLLM says the
+# capability was not assessed rather than guessing at one.
 #
 # Thresholds are on FIT_TOTAL_MB -- usable VRAM after the KV reserve -- not on
 # installed VRAM. Sizing off installed memory is what once recommended a 49 GB
@@ -129,7 +140,7 @@ quant_include_pattern() {
 # budget decision rather than a property of the model: the same 30B MoE wants
 # IQ3 on a 8 GB card and Q6_K on a 48 GB one.
 plan_for_budget() {
-  local fit_total="$1" moe_offload="${2:-0}"
+  local fit_total="$1" moe_offload="${2:-0}" gpu_cc="${3:-}"
   # Cleared up front so a failure two calls ago cannot be read back as this
   # call's diagnosis. Read by 00-specs.sh and 30-models.sh, which shellcheck
   # cannot see from inside this file.
@@ -141,7 +152,7 @@ plan_for_budget() {
     PLAN_ID_1="qwen3-coder-30b"; PLAN_Q_1="IQ3_XXS|Q3_K_S"
     PLAN_ID_2="qwen3-4b";        PLAN_Q_2="Q5_K_M"
     PLAN_ID_3="qwen3-1.7b";      PLAN_Q_3="Q8_0"
-    PLAN_RUNTIME="llama.cpp + llama-swap. vLLM is not viable -- it needs the whole model resident."
+    PLAN_RUNTIME="llama.cpp + llama-swap."
     PLAN_NOTE="Claude Code needs 32k context minimum, and the KV cache at that length
   will take a large share of this card. The primary pick is a MoE precisely
   because only ~3B params are active per token, so CPU expert offload stays
@@ -170,7 +181,7 @@ plan_for_budget() {
     PLAN_ID_1="qwen3-coder-30b"; PLAN_Q_1="Q6_K|Q5_K_M"
     PLAN_ID_2="qwen3-32b";       PLAN_Q_2="Q5_K_M"
     PLAN_ID_3="devstral-small";  PLAN_Q_3="Q5_K_M"
-    PLAN_RUNTIME="llama.cpp + llama-swap now; vLLM is worth measuring once you settle on one model."
+    PLAN_RUNTIME="llama.cpp + llama-swap."
     PLAN_NOTE="Enough headroom to spend it on quantization quality rather than more
   parameters -- a higher quant of a right-sized model beats a squeezed larger one."
 
@@ -179,10 +190,17 @@ plan_for_budget() {
     PLAN_ID_1="llama-3.3-70b";   PLAN_Q_1="IQ4_XS|Q4_K_S"
     PLAN_ID_2="qwen3-coder-30b"; PLAN_Q_2="Q6_K"
     PLAN_ID_3="qwen3-32b";       PLAN_Q_3="Q5_K_M"
-    PLAN_RUNTIME="vLLM with prefix caching is worth evaluating at this scale; llama.cpp + llama-swap still works."
+    PLAN_RUNTIME="llama.cpp + llama-swap."
     PLAN_NOTE="Large enough to run a frontier-class open model resident."
   fi
 
+  # The runtime advice, computed rather than hardcoded per tier. Kept out of
+  # PLAN_RUNTIME so a caller can print or suppress it independently: it is a
+  # judgement about an alternative, where PLAN_RUNTIME states what this rig
+  # actually uses.
+  # shellcheck disable=SC2034  # documented return channel, read by callers
+  PLAN_VLLM="$(vllm_advice "$gpu_cc" "$fit_total" "$moe_offload")"
+
   # With plenty of system RAM, a MoE far larger than VRAM is viable via
   # --n-cpu-moe, because only the active experts must be resident.
   if (( moe_offload > 60000 )); then
@@ -223,7 +241,7 @@ plan_for_budget() {
 
   # These are the function's return values -- callers read them. Exporting
   # states that contract (and is what detect_hw already does for its outputs).
-  export PLAN_TIER PLAN_RUNTIME PLAN_NOTE PLAN_MOE_NOTE \
+  export PLAN_TIER PLAN_RUNTIME PLAN_NOTE PLAN_MOE_NOTE PLAN_VLLM \
          PLAN_ID_1 PLAN_ID_2 PLAN_ID_3 \
          PLAN_SEARCH_1 PLAN_SEARCH_2 PLAN_SEARCH_3 \
          PLAN_REPO_1 PLAN_REPO_2 PLAN_REPO_3 \

lib/runtime.sh

@@ -0,0 +1,197 @@
+#!/usr/bin/env bash
+# Which serving runtime to recommend, and on what evidence.
+#
+# THE BUG THIS FIXES
+#
+# plan_for_budget decided whether to suggest vLLM from VRAM alone. At 45 GB and
+# up it said "worth evaluating at this scale"; below 9 GB it said "not viable".
+# Memory is the wrong axis to decide it on by itself.
+#
+# vLLM's advantage over llama.cpp is throughput from resident weights and fast
+# low-precision kernels. WHICH kernels exist is a property of the GPU's compute
+# capability, not of how much memory is attached to it. A pair of RTX A4000s is
+# 31 GB -- enough to read as "worth evaluating" -- and sm_86, which means no
+# native FP8 and no NVFP4. The advice was confidently wrong on this very rig.
+#
+# The second thing memory alone cannot see: vLLM keeps the whole model in VRAM.
+# It has no equivalent of --n-cpu-moe, so a 118B MoE that llama.cpp runs
+# comfortably with its experts in system RAM does not run slower under vLLM, it
+# does not run. On a machine with far more RAM than VRAM that is the difference
+# between a usable model and none.
+#
+# Everything here is pure: a compute capability and two numbers in, a sentence
+# out. No nvidia-smi, no network.
+#
+# shellcheck shell=bash
+
+[[ -z "${_LLMRIG_RUNTIME_SH:-}" ]] || return 0
+_LLMRIG_RUNTIME_SH=1
+
+# --- compute capability -----------------------------------------------------
+
+# runtime_cc_x10 <compute_cap> -- "8.6" as 86, so bash can compare it.
+#
+# Status 1 for anything unparseable, INCLUDING the empty string. Detection can
+# fail, and a missing capability must not silently compare as zero and read as
+# an ancient card -- "we could not tell" and "it is too old" want different
+# sentences.
+runtime_cc_x10() {
+  local cc="${1:-}" whole frac
+  [[ "$cc" =~ ^[0-9]+(\.[0-9]+)?$ ]] || return 1
+  whole="${cc%.*}"
+  frac=0
+  [[ "$cc" == *.* ]] && frac="${cc#*.}"
+  printf '%d' "$(( whole * 10 + ${frac:0:1} ))"
+}
+
+# runtime_arch <compute_cap> -- the marketing name for a capability, so the
+# advice can say "Turing" or "Ampere" without a branch hardcoding one.
+#
+# This exists because it was wrong once: the FP8-weight-only class used to be
+# spelled "(Ampere)" in its sentence, which became a lie the moment Turing was
+# correctly moved into the same class.
+runtime_arch() {
+  local cc10
+  cc10="$(runtime_cc_x10 "${1:-}")" || { printf 'unknown'; return 0; }
+  if   (( cc10 >= 100 )); then printf 'Blackwell'
+  elif (( cc10 >=  90 )); then printf 'Hopper'
+  elif (( cc10 >=  89 )); then printf 'Ada'
+  elif (( cc10 >=  80 )); then printf 'Ampere'
+  elif (( cc10 >=  75 )); then printf 'Turing'
+  elif (( cc10 >=  70 )); then printf 'Volta'
+  else                         printf 'pre-Volta'
+  fi
+}
+
+# vllm_kernels <compute_cap> -- what low-precision arithmetic this GPU can
+# actually do under vLLM. One of:
+#
+#   unknown        capability could not be determined
+#   none           below Volta; vLLM does not support the card at all
+#   int            Volta: integer quantization only, no FP8 path of any kind
+#   int-fp8-weight Turing and Ampere: INT4/INT8 Marlin, and FP8 checkpoints
+#                  LOAD but are dequantized weight-only (W8A8 becomes W8A16)
+#                  -- you get the memory saving, not the arithmetic
+#   fp8            Ada / Hopper: native FP8 W8A8
+#   fp4            Blackwell: NVFP4
+#
+# Boundaries, so they can be checked rather than trusted:
+#   sm_70 Volta, sm_75 Turing, sm_80/86 Ampere, sm_89 Ada, sm_90 Hopper,
+#   sm_100+ Blackwell.
+#
+#   FP8 splits at two different capabilities, and conflating them is the error
+#   this table has already made once. vLLM's FP8 documentation:
+#
+#     "FP8 computation is supported on NVIDIA GPUs with compute capability
+#      >= 8.9 (Ada Lovelace, Hopper)."
+#     "FP8 models will run on compute capability >= 7.5 (Turing) as weight-only
+#      W8A16, utilizing FP8 Marlin."
+#
+#   So the weight-only floor is 7.5, not 8.0 -- Turing loads an FP8 checkpoint
+#   exactly as Ampere does. Native W8A8 arithmetic still starts at 8.9, and
+#   nothing below that should be described as having FP8 hardware.
+#   https://docs.vllm.ai/en/stable/features/quantization/fp8/
+#
+#   NVFP4 is Blackwell (SM100+); below that it is weight-only at best.
+vllm_kernels() {
+  local cc10
+  cc10="$(runtime_cc_x10 "${1:-}")" || { printf 'unknown'; return 0; }
+  if   (( cc10 >= 100 )); then printf 'fp4'
+  elif (( cc10 >=  89 )); then printf 'fp8'
+  elif (( cc10 >=  75 )); then printf 'int-fp8-weight'
+  elif (( cc10 >=  70 )); then printf 'int'
+  else                         printf 'none'
+  fi
+}
+
+# --- the recommendation -----------------------------------------------------
+
+# The weight-resident budget at or above which NVFP4 stops being a spec sheet
+# and starts being a reason to switch. TUNING.md's reversal condition is "one
+# Blackwell card with >=48 GB" and this is that number -- but measured on the
+# budget left for weights after the KV reserve, which is the only memory figure
+# this function is handed.
+#
+# That makes the gate harder to clear than the sticker capacity: a 48 GB card
+# whose KV reserve takes 10 GB lands near 38 GB and gets the hedged sentence
+# rather than the enthusiastic one. Deliberate. Nobody here has measured a
+# Blackwell box, and under-promising about hardware you do not own is the safe
+# direction to be wrong in. Lower it when a real one disagrees.
+VLLM_RESIDENT_WIN_MB=49152
+
+# vllm_advice <compute_cap> <fit_total_mb> [moe_offload_mb]
+#
+# One sentence for the specs report. Deliberately says what is true of THIS
+# card rather than of vLLM in general, and names the capability so a reader can
+# check the claim against NVIDIA's table instead of believing it.
+#
+# Two axes, and a positive verdict needs both: kernels the card has, and enough
+# resident room to put a model in front of them. Either one alone has already
+# produced advice that was confidently wrong.
+vllm_advice() {
+  local cc="${1:-}" fit="${2:-0}" offload="${3:-0}" kernels arch
+  [[ "$fit"     =~ ^[0-9]+$ ]] || fit=0
+  [[ "$offload" =~ ^[0-9]+$ ]] || offload=0
+  kernels="$(vllm_kernels "$cc")"
+  arch="$(runtime_arch "$cc")"
+
+  # 00-specs.sh prints this line bare, with nothing around it to say what it is
+  # about, so the sentence has to introduce itself. Emitted clause by clause
+  # rather than as one long format string, so each stays readable at the width
+  # the rest of this repo is written to.
+  printf 'vLLM: '
+
+  case "$kernels" in
+    unknown)
+      printf 'not assessed -- the GPU compute capability could not be read,'
+      printf ' and it is what decides whether the low-precision kernels vLLM'
+      printf ' wins with exist here.'
+      return 0 ;;
+    none)
+      printf 'not supported on compute capability %s' "${cc:-unknown}"
+      printf ' (it needs 7.0 or newer).'
+      return 0 ;;
+  esac
+
+  # Weights must be resident. Where system RAM dwarfs VRAM that is the whole
+  # argument, and it is worth making before any kernel talk.
+  if (( fit > 0 && offload > fit * 2 )); then
+    printf 'it holds the whole model in VRAM, with no --n-cpu-moe'
+    printf ' equivalent, so the %s GB of system RAM here buys it nothing --' "$(( offload / 1024 ))"
+    printf ' a MoE larger than %s GB runs under llama.cpp and not at all' "$(( fit / 1024 ))"
+    printf ' under vLLM. '
+  fi
+
+  case "$kernels" in
+    int)
+      printf 'On compute capability %s (%s) there is no FP8 path at all --' "$cc" "$arch"
+      printf ' not even the weight-only one Turing gets -- so the gain over'
+      printf ' llama.cpp would be batching this rig does not do.' ;;
+    int-fp8-weight)
+      printf 'Compute capability %s (%s) has no native FP8 and no NVFP4:' "$cc" "$arch"
+      printf ' INT4/INT8 Marlin work, and an FP8 checkpoint loads only as'
+      printf ' weight-only W8A16 through FP8 Marlin -- the memory saving'
+      printf ' without the arithmetic. Worth revisiting on Ada or newer.' ;;
+    fp8)
+      printf 'Compute capability %s (%s) has native FP8, so vLLM is worth' "$cc" "$arch"
+      printf ' measuring once you have settled on one model -- its remaining'
+      printf ' costs here are a cold start per model and one model per'
+      printf ' process.' ;;
+    fp4)
+      # Kernels alone do not make the case. vLLM has to hold the whole model
+      # resident, so NVFP4 on a card too small for the model you want is a
+      # faster way to run something else.
+      printf 'Compute capability %s (%s) has NVFP4' "$cc" "$arch"
+      if (( fit >= VLLM_RESIDENT_WIN_MB )); then
+        printf ', and %s GB of resident budget to spend it on --' "$(( fit / 1024 ))"
+        printf ' that combination is where vLLM pulls clearly ahead.'
+        printf ' Worth measuring against llama.cpp on your actual model.'
+      else
+        printf ', but %s GB of resident budget is under the %s GB' \
+          "$(( fit / 1024 ))" "$(( VLLM_RESIDENT_WIN_MB / 1024 ))"
+        printf ' this repo treats as the switching point. The kernels are'
+        printf ' the fast part; capacity is the binding one, because vLLM'
+        printf ' must hold the whole model. Measure it on a model that fits.'
+      fi ;;
+  esac
+}