Decide vLLM on compute capability, not on VRAM

[rammsguns]

Closes #28.

The advice was keyed on VRAM, which cannot see which low-precision kernels a card has. On this rig that produced a confidently wrong recommendation, so the fix is to derive the verdict from GPU_CC — already detected and exported by lib/detect.sh — and to name the capability in the output so the claim is checkable.

A positive verdict now needs two things: kernels the card has, and enough resident room to put a model in front of them. Each axis alone has already produced advice that was wrong.

Before and after, same machine

Two A4000s, 21,453 MB usable after the KV reserve, ~109 GB system RAM:

Before — llama.cpp + llama-swap now; vLLM is worth measuring once you settle on one model.

After — vLLM: it holds the whole model in VRAM, with no --n-cpu-moe equivalent, so the 109 GB of system RAM here buys it nothing -- a MoE larger than 20 GB runs under llama.cpp and not at all under vLLM. Compute capability 8.6 (Ampere) has no native FP8 and no NVFP4: INT4/INT8 Marlin work, and an FP8 checkpoint loads only as weight-only W8A16 through FP8 Marlin -- the memory saving without the arithmetic. Worth revisiting on Ada or newer.

Change nothing but the capability to 10.0 and the same budget gets a different sentence. That divergence is the property that was missing, and it is what the central test asserts.

Boundaries

Capability	Class
< 7.0	none	vLLM does not support the card
7.0–7.4	int	Volta — no FP8 path of any kind
7.5–8.8	int-fp8-weight	Turing / Ampere — this rig. INT4/INT8 Marlin, plus FP8 checkpoints loaded weight-only as W8A16
8.9–9.x	fp8	Ada / Hopper, native W8A8
≥ 10.0	fp4	Blackwell, NVFP4
unparseable	unknown	declines to advise

unknown is deliberately not the same answer as none. An empty string in an arithmetic comparison is zero, which would classify an undetected card as pre-Volta — the same species of confident wrongness this PR exists to remove.

Two corrections from review, and one found while making them

FP8 has two floors, not one. The table put weight-only FP8 at 8.0. It is 7.5. vLLM's FP8 documentation: "FP8 computation is supported on NVIDIA GPUs with compute capability >= 8.9" and "FP8 models will run on compute capability >= 7.5 (Turing) as weight-only W8A16, utilizing FP8 Marlin." So a Turing owner was being told there was "no FP8 path at all" when they have exactly the path Ampere has. 7.5 now joins int-fp8-weight; 7.0 keeps the older verdict; both sides of 7.4/7.5 are asserted, and a test states plainly that Turing must never be described as having FP8 hardware. (vLLM FP8 docs)

NVFP4 alone did not earn "pulls clearly ahead." TUNING.md's reversal condition is one Blackwell card with ≥48 GB, and the code only checked the kernels — so a 12 GB Blackwell laptop chip was told vLLM pulls clearly ahead, on a card that cannot hold the model resident at all, which is the single thing vLLM requires. The positive verdict now needs both halves. Below the gate the kernels are still reported honestly and capacity is named as the binding constraint. Both sides of 49,151 / 49,152 MB are asserted, and a further test checks the gate does not leak into the other classes — memory cannot conjure kernels for Ampere, and Ada does not lose native FP8 for being small.

Found while fixing the first one: the int-fp8-weight sentence hardcoded the string (Ampere), which became a lie for every Turing card the moment 7.5 moved into that class. The architecture name is derived from the capability now, and a test asserts a Turing card is never called Ampere.

Judgement calls

1. The tiers no longer say anything about vLLM at all, rather than saying something capability-adjusted. Five hardcoded strings were what made this unfixable in one place; a test now asserts none of them mentions vLLM again. PLAN_RUNTIME states what the rig uses, PLAN_VLLM judges the alternative.
2. 00-specs.sh prints the verdict unconditionally, as its own line. It is an opinion about something you are not running, so hiding it when negative would make the report only ever advertise vLLM. Because it prints bare, with no label around it, the sentence now names vLLM in every branch — previously only the two early returns did, so on a box whose RAM does not dwarf its VRAM the report emitted an unattributed sentence about compute capability.
3. gpu_cc is an optional third argument rather than read from the environment, to keep plan_for_budget pure and testable. Omitted, you get "not assessed".
4. The 48 GB gate is measured on the weight budget after the KV reserve, because that is the only memory figure vllm_advice is handed. That makes it stricter than a 48 GB sticker price — a 48 GB card with a 10 GB reserve lands near 38 GB and gets the hedged sentence. Deliberate, and written down at the constant: nobody here has run a Blackwell box, and under-promising about hardware you do not own is the safe direction to be wrong in. Easy to lower if you would rather it track card capacity; that would mean threading total VRAM through as a fourth argument.

Docs

TUNING.md's existing vLLM bullet under Rejected, with reasons is expanded rather than duplicated — it previously rejected vLLM on workload grounds only (prefix caching, single user, no NVLink), which is true but the weaker half of the argument. It now carries the hardware reasoning, both FP8 floors, and the reversal condition with both halves marked load-bearing. README says the same in one paragraph.

Not done

vLLM was not installed or benchmarked here. The claims are about which kernels the hardware has, which is a specification, not a measurement — and the one measurement that would settle it (INT4 Laguna XS under vLLM with TP=2 vs llama.cpp) needs a 19 GB download and a working vLLM install to produce a number that only describes this box. Happy to do it if you want the empirical version.

No Turing or Blackwell card was tested either. Those boundaries are read off vLLM's documentation, which is why the citation is in the source next to the numbers.

Conflict note

main has been merged down after #24 and #25 landed. No conflict — those two both inserted new top-level sections before Known rough edges, while this PR edits a bullet inside Rejected, with reasons, so git took all three. The resulting section order is Rejected → Rolling the OS tuning back → Reproducibility of the stack itself → Known rough edges. (The #24/#25 collision was real and was resolved on #25.)

Tests

480 pass across 15 suites, both runners green. tests/cases/runtime_test.sh is 21 tests, 7 of them added by this review pass: the 7.4/7.5 boundary, Turing's weight-only path, Volta keeping the no-FP8-at-all verdict, derived architecture names, NVFP4-without-capacity, both sides of the 48 GB gate, and the gate not leaking into the other classes.

[rammsguns]

PM review: the Ampere correction is accepted, but two precision fixes are required before merge.

1. sm_75 is currently classified as int, and vllm_advice 7.5 ... therefore says there is “no FP8 path at all.” Current vLLM FP8 documentation says FP8 checkpoints run on compute capability >= 7.5 as weight-only W8A16 through FP8 Marlin. Split or adjust the 7.5 boundary, update the prose/table, and add boundary tests at 7.4/7.5. Do not imply that Turing has native FP8.

2. The documented reversal condition is one Blackwell card with >=48 GB, but the fp4 branch says vLLM “pulls clearly ahead” for every sm_100+ device regardless of capacity; fit_total_mb does not gate that conclusion. Make the positive verdict reflect both kernel capability and resident-memory viability (or soften it to a capability statement when capacity is below/unknown), and test both sides of the chosen capacity boundary.

The A4000/sm_86 conclusion remains correct: FP8 can load as W8A16, yielding weight-memory savings without native FP8 arithmetic, and Laguna XS FP8 still exceeds this rig's usable VRAM.

After these changes, merge current main into this branch because #24 and #25 also touch TUNING.md, rerun CI, and report any conflict resolution explicitly.

[rammsguns]

Both fixes are in 3fd5d56. PR body rewritten to match.

1. The 7.5 boundary. You are right, and the doc says so plainly: FP8 W8A8 needs ≥8.9, but "FP8 models will run on compute capability >= 7.5 (Turing) as weight-only W8A16, utilizing FP8 Marlin." Turing has exactly the path Ampere has, and the table was telling those owners they had none.

• int-fp8-weight now starts at 7.5; int is Volta only.
• Boundary tests at 7.4 and 7.5, plus one asserting 7.0 keeps the no FP8 path at all sentence rather than being dragged up with it.
• A test named for the risk you flagged: Turing is never described as having FP8 hardware. It has to contain W8A16 and no native FP8, and must not contain the Volta sentence.
• The quoted doc text is now in lib/runtime.sh next to the numbers, with the URL, so the next person can check the boundary instead of trusting the table.

Correcting the boundary exposed a second defect you did not ask about: the class sentence hardcoded the literal string (Ampere), which turned into a lie for every Turing card the moment 7.5 joined the class. Architecture names are derived from the capability now, and a test asserts a Turing card is never called Ampere.

2. The capacity gate. Also right, and worse than a hedge — a 12 GB Blackwell laptop chip was being told vLLM "pulls clearly ahead" on a card that cannot hold the model resident at all, which is the one thing vLLM requires. fp4 now needs both halves. Below the gate it still reports NVFP4 honestly and names capacity as the binding constraint. Tests assert 49,152 clears it and 49,151 does not, that the threshold constant equals the number the prose quotes, and that the gate does not leak sideways — Ampere with 64 GB is still told it lacks the kernels, Ada with 12 GB still gets its native-FP8 verdict.

One thing to overrule if you disagree: the 48 GB is measured on the weight budget after the KV reserve, since that is the only memory figure vllm_advice receives. That is stricter than a 48 GB sticker price — a 48 GB card with a 10 GB reserve lands near 38 GB and gets the hedged sentence, so the documented reversal condition in practice wants a larger card. I chose to under-promise about hardware nobody here has run, and wrote that reasoning at the constant. If you would rather it track card capacity literally, that means threading total VRAM through as a fourth argument; say so and I will.

3. Found while in there. 00-specs.sh prints this line bare, and only the two early-return branches carried the vLLM: label. On a box whose RAM does not dwarf its VRAM, the report emitted an unattributed sentence about compute capability. Every branch names the runtime now, asserted for all ten capability classes.

Merge-down and conflicts. main merged down after #24 and #25 landed: no conflict on this branch. Those two both inserted new top-level sections immediately before Known rough edges, whereas this PR expands a bullet inside Rejected, with reasons, so git took all three cleanly. Section order is now Rejected → Rolling the OS tuning back → Reproducibility of the stack itself → Known rough edges.

The #24/#25 collision was real and was resolved on #25 by keeping both sections — rollback first, since it closes out the tuning content the page is about, then reproducibility — plus one sentence tying them together, that pinning the software is only half a reproducible measurement if the machine underneath has drifted. Nothing was dropped from either side.

480 tests pass across 15 suites, both runners.

Still not measured: no Turing card, no Blackwell card, and no vLLM install. Every boundary here is read off vLLM's documentation, and the 48 GB figure is a product decision rather than an observation. That is why the citation sits in the source.