Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions docs/adr/0012-gguf-type-mapping.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,48 @@
overrides, placed before the group overrides — the quantizer
applies the first matching pattern. The backend still rejects
tensor-level *groups*: the boundary moved for protections only.
- **Amendment (2026-08-12, issue #180):** decision 2 gains a second
group shape, drops a fixed prefix, and gains one refusal.

The backend derives the layer index from the group name. Any
layer group ending in `.layers.<n>`, `.h.<n>`, or `.blocks.<n>`
becomes `blk\.<n>\.`. GGUF numbers every layer `blk.<n>.`
whatever the checkpoint calls it. Matching `model.layers.<n>`
alone refused the Nemotron 3.5 Lightning target at
`backbone.layers.<n>` (#160).

The embedding group gains the same treatment. Decision 2 fixed
it at `model.embed_tokens`, and the target names it
`backbone.embeddings`. The backend now carries both names. The
output head stays the literal `lm_head`, which the target
carries verbatim.

A routed-expert stack group becomes its fused tensor:
`blk\.<n>\.ffn_up_exps\.`, `blk\.<n>\.ffn_down_exps\.`, or
`blk\.<n>\.ffn_gate_exps\.`. llama.cpp fuses one layer's routed
experts into a single 3D tensor. That tensor carries one
quantization type, so the expert stack is the unit a pack
addresses (#159, ADR-0001 as amended by #161).

Expert-stack overrides go before layer overrides, and both go
after the protection overrides. The quantizer applies the first
matching pattern, and `blk\.1\.` also matches
`blk.1.ffn_up_exps.weight`. Order is priority: per-tensor, then
expert stack, then layer.

**Every mapped group must hang from one parameter-tree root.**
A free prefix cannot tell a decoder layer from any other layer
stack. The target carries `mtp.layers.<n>` beside
`backbone.layers.<n>`, and a multimodal checkpoint carries a
vision tower that GGUF names `v.blk.<n>.`. Both would map onto
`blk.<n>.` and lose one assignment to the first-match rule. The
backend refuses a recipe naming two roots, and names both.

The backend still refuses every other group, and it names the
group. It refuses tensor-level groups. It refuses any tensor
class outside this mapping — the Mamba `in_proj`, `out_proj`,
and `conv1d`, the attention projections, the router, the shared
experts. Issue #183 carries those.

## Context

Expand Down
27 changes: 27 additions & 0 deletions docs/adr/0022-within-layer-protections.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,33 @@
map. `stack` exists because it is *coarser* than per-tensor and
matches the pack, not because per-tensor scanning came back.

- **Amendment (2026-08-12, issue #180):** this amendment supersedes
the amendment above in part. The GGUF backend now maps a
routed-expert stack group to its fused tensor. It derives the
layer index from the group name instead of matching
`model.layers.<n>`, and it carries the target's
`backbone.embeddings` name
([ADR-0012](0012-gguf-type-mapping.md) decision 2, amended the
same day). A `layer`-keyed recipe for the Nemotron 3.5 Lightning
target packs. So does a recipe whose only stack groups are
routed-expert stacks.

Decision 1's refusal narrows. It does not lift. A whole-model
`stack`-keyed recipe still carries groups outside any GGUF class
mapping — the Mamba `in_proj`, `out_proj`, and `conv1d`, the
attention projections, the router, the shared experts. The
backend refuses each by name. Issue #183 carries that table, and
it must first rule what a recipe does with a group
`llama-quantize` never quantizes.

The protection path is unchanged.
[ADR-0012](0012-gguf-type-mapping.md) decision 2's class table
still matches `model.layers.<n>.<class>.weight` only. The target
does carry four of the seven classes, at
`backbone.layers.<n>.mixer.{q,k,v,o}_proj`. Generalizing the
prefix alone would still miss them, because the table keys read
`self_attn.q_proj` and this checkpoint says `mixer.q_proj`.

## Context

The twelfth data point (in
Expand Down
33 changes: 22 additions & 11 deletions docs/reference/sensitivity-map.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,14 +146,25 @@ below remain, the sub-4-bit pricing claims do not.
`tensor`-keyed map of that model prices 5888 distinctions no pack
can express.

!!! warning "A `stack` scan does not pack yet"

The GGUF v1 backend maps only `model.layers.<n>` groups to
`blk.<n>.` patterns. It refuses every other group name with a
`PackError` (ADR-0022 decision 1, amended 2026-08-11). A
`stack`-keyed recipe therefore reaches `vramfit pack` and
stops there. The same refusal catches any model whose layers
are not named `model.layers.<n>`, including the Nemotron 3.5
Lightning target at `backbone.layers.<n>`. Issue #180 carries
the backend work. Scan and plan with `stack` today. Do not
spend a multi-day scan expecting to pack the result.
!!! warning "A `stack` scan packs its expert stacks, not every group"

The GGUF backend maps two group shapes (ADR-0012 decision 2,
amended 2026-08-12). A layer group becomes `blk.<n>.` across
the three naming families above — both `model.layers.<n>`
and the Nemotron 3.5 Lightning target's
`backbone.layers.<n>`. A routed-expert stack becomes its
fused tensor: `blk.<n>.ffn_up_exps.`,
`blk.<n>.ffn_down_exps.`, or `blk.<n>.ffn_gate_exps.`.

Every other `stack` group still raises a `PackError` that
names it. On the Nemotron target that covers the Mamba
`in_proj`, `out_proj`, and `conv1d`, the attention
projections, the router, and the shared experts. So a
`layer`-keyed recipe packs today and a whole-model
`stack`-keyed recipe does not. Issue #183 carries the
remaining classes.

The backend also refuses a recipe naming two layer stacks.
GGUF numbers one stack `blk.<n>.`, so the target's
`mtp.layers.<n>` and a multimodal checkpoint's vision tower
each collide with the backbone. Scan one stack at a time.
178 changes: 151 additions & 27 deletions src/vramfit/adapters/outbound/gguf/types.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,10 @@
The decision core of the GGUF backend, kept free of IO so the mapping
is testable and the verified fake can share it. Nominal precisions
map to K-quant types (the full llama.cpp capability set since
ADR-0013), layer groups map to escaped `blk.<n>.` regex patterns,
protected tensors map through the fixed HF-to-GGUF class table to
ADR-0013), layer groups map to escaped `blk.<n>.` regex patterns
across the three naming families the scan produces, routed-expert
stack groups map to their fused `blk.<n>.ffn_<proj>_exps.` tensor
(#159, #161), protected tensors map through the fixed HF-to-GGUF class table to
per-tensor patterns (ADR-0022), excluded pairs map to the full GGUF
tensor names ``--exclude-weights`` deletes by substring (ADR-0023),
and the embedding and `lm_head` groups map to the quantizer's
Expand Down Expand Up @@ -65,11 +67,51 @@
# another runtime must not silently become a GGUF (ADR-0013).
GGUF_RUNTIME: Final[str] = LLAMA_CPP

EMBEDDING_GROUP: Final[str] = "model.embed_tokens"
# The embedding group names the scan produces, across naming
# families. llama-family checkpoints say `model.embed_tokens`.
# Nemotron-H says `backbone.embeddings`. Both drive the one
# `--token-embedding-type` flag, so the backend needs the names, not
# a pattern.
EMBEDDING_GROUPS: Final[frozenset[str]] = frozenset(
{"model.embed_tokens", "backbone.embeddings"}
)

OUTPUT_GROUP: Final[str] = "lm_head"

_LAYER_GROUP: Final[re.Pattern[str]] = re.compile(r"^model\.layers\.(\d+)$")
# A layer group, under the three naming families the scan produces
# (`domain.scan` names them the same way). The prefix is free, so
# `model.layers.4` and Nemotron-H's `backbone.layers.4` both yield 4
# (#160). GGUF numbers every layer `blk.<n>.`, whatever the
# checkpoint calls it.
_LAYER_GROUP: Final[re.Pattern[str]] = re.compile(r"^.+\.(?:layers|h|blocks)\.(\d+)$")
Comment on lines +81 to +86

# A routed-expert stack group: a layer prefix, then `.experts.` with
# the expert index already collapsed by `group_key`, then the
# projection (#161). The dot before `experts` matters — it refuses
# `shared_experts`, which GGUF names `ffn_up_shexp` and this table
# does not carry (#183).
_EXPERT_STACK: Final[re.Pattern[str]] = re.compile(
r"^.+\.(?:layers|h|blocks)\.(\d+)\.(?:.*\.)?experts\.([A-Za-z0-9_]+)$"
)

# The parameter-tree root a layer or expert-stack group hangs from.
# `blk.<n>.` addresses exactly one layer stack, so a recipe that
# names two of them cannot pack. The target carries `backbone` and
# `mtp`, and a multimodal checkpoint carries a vision tower that
# GGUF names `v.blk.<n>.` instead.
_STACK_ROOT: Final[re.Pattern[str]] = re.compile(
r"^(.+?)\.(?:layers|h|blocks)\.\d+(?:\.|$)"
)

# llama.cpp fuses one layer's routed experts into a single 3D tensor
# that carries one quantization type, so the pack addresses the
# stack and never one expert inside it (#159). HF projection name to
# fused GGUF tensor.
GGUF_EXPERT_STACK_BY_HF: Final[dict[str, str]] = {
"up_proj": "ffn_up_exps",
"down_proj": "ffn_down_exps",
"gate_proj": "ffn_gate_exps",
}

# The fixed class table (ADR-0022): HF tensor suffix to GGUF tensor
# suffix, for the seven quantized projections of a llama-family layer.
Expand Down Expand Up @@ -202,7 +244,8 @@ def token_embedding_type(recipe: Recipe) -> str | None:
``--token-embedding-type`` binds the embedding tensor before any
pattern override, so the embedding group never becomes a pattern.
When the model ties embeddings, this assignment also governs the
output head (ADR-0012).
output head (ADR-0012). The group carries one of the names in
`EMBEDDING_GROUPS`, which differ by naming family.

Args:
recipe: The recipe to pack.
Expand All @@ -223,7 +266,7 @@ def token_embedding_type(recipe: Recipe) -> str | None:
```
"""
for assignment in recipe.assignments:
if assignment.group == EMBEDDING_GROUP:
if assignment.group in EMBEDDING_GROUPS:
return ggml_type_for(assignment.bits)
return None

Expand Down Expand Up @@ -367,24 +410,101 @@ def imatrix_exclusion_names(recipe: Recipe) -> tuple[str, ...]:
)


def tensor_overrides(recipe: Recipe) -> tuple[TypeOverride, ...]:
r"""Translate layer groups into quantizer tensor-type overrides.
def gguf_stack_prefix(group: str) -> str | None:
"""Map one routed-expert stack group to its GGUF tensor prefix.

Args:
group: Recipe group name, e.g.
``backbone.layers.3.mixer.experts.down_proj``.

Returns:
The GGUF tensor prefix, e.g. ``blk.3.ffn_down_exps.``, or
None when the group is not a routed-expert stack.

Raises:
PackError: If the group is a routed-expert stack whose
projection has no entry in the fused-stack table.

Examples:
The Nemotron 3.5 Lightning down projection:

One override per layer group: ``model.layers.<n>`` becomes the
escaped pattern ``blk\.<n>\.``. Escaping matters — an unescaped
``blk.1.`` would also match ``blk.11.``. The embedding and
``lm_head`` groups map to dedicated flags and are skipped here.
```python
group = "backbone.layers.3.mixer.experts.down_proj"
assert gguf_stack_prefix(group) == "blk.3.ffn_down_exps."
```
"""
match = _EXPERT_STACK.match(group)
if match is None:
return None
suffix = GGUF_EXPERT_STACK_BY_HF.get(match.group(2))
if suffix is None:
raise PackError(
f'expert stack "{group}" has no GGUF mapping — llama.cpp fuses '
f"the projections {sorted(GGUF_EXPERT_STACK_BY_HF)} (#159)"
)
return f"blk.{match.group(1)}.{suffix}."


def _claim_root(group: str, roots: dict[str, str]) -> None:
"""Hold every mapped group to one parameter-tree root.

Args:
group: Recipe group name.
roots: Roots claimed so far, mapped to the group that
claimed each. Updated in place.

Raises:
PackError: If ``group`` hangs from a second root.
"""
match = _STACK_ROOT.match(group)
if match is None:
return
root = match.group(1)
roots.setdefault(root, group)
if len(roots) > 1:
first = next(iter(roots))
raise PackError(
f'groups "{roots[first]}" and "{group}" name two layer stacks — '
f'a GGUF pack numbers one stack "blk.<n>." and would silently '
f"drop the other (#183)"
)


def tensor_overrides(recipe: Recipe) -> tuple[TypeOverride, ...]:
r"""Translate recipe groups into quantizer tensor-type overrides.

Two group shapes map. A layer group under any of the three
naming families — ``model.layers.<n>``, ``backbone.layers.<n>``
— becomes the escaped pattern ``blk\.<n>\.``. A routed-expert
stack group becomes the escaped pattern for its fused tensor,
e.g. ``blk\.<n>\.ffn_up_exps\.`` (#159, #161). Escaping matters
— an unescaped ``blk.1.`` would also match ``blk.11.``. The
embedding and ``lm_head`` groups map to dedicated flags and are
skipped here.

Expert-stack overrides come first, ahead of the layer overrides.
The quantizer applies the first matching pattern, and
``blk\.1\.`` also matches ``blk.1.ffn_up_exps.weight``. Callers
place the protection overrides ahead of both — a per-tensor
pattern is the most specific of the three (ADR-0022).

Every mapped group must hang from one parameter-tree root.
``blk.<n>.`` addresses a single layer stack, so a recipe naming
two of them would map both onto it and silently drop one.

Args:
recipe: The recipe to pack.

Returns:
Overrides in recipe order. The quantizer applies the first
match, and the patterns are mutually exclusive.
Expert-stack overrides in recipe order, then layer overrides
in recipe order.

Raises:
PackError: If a group is not a layer group, the embedding,
or the output head, or its precision has no table entry.
PackError: If a group is not a layer group, a routed-expert
stack, the embedding, or the output head. Also if a
routed-expert stack names a projection outside the
fused-stack table, if the groups hang from two roots, or
if a precision has no table entry.

Examples:
The group ``model.layers.7`` at 4-bit becomes an escaped
Expand All @@ -394,21 +514,25 @@ def tensor_overrides(recipe: Recipe) -> tuple[TypeOverride, ...]:
assert TypeOverride(r"blk\.7\.", "q4_k") in tensor_overrides(recipe)
```
"""
overrides: list[TypeOverride] = []
stacks: list[TypeOverride] = []
layers: list[TypeOverride] = []
roots: dict[str, str] = {}
for assignment in recipe.assignments:
if assignment.group in (EMBEDDING_GROUP, OUTPUT_GROUP):
if assignment.group in EMBEDDING_GROUPS or assignment.group == OUTPUT_GROUP:
continue
_claim_root(assignment.group, roots)
prefix = gguf_stack_prefix(assignment.group)
if prefix is not None:
bits = ggml_type_for(assignment.bits)
stacks.append(TypeOverride(re.escape(prefix), bits))
continue
match = _LAYER_GROUP.match(assignment.group)
if match is None:
raise PackError(
f'group "{assignment.group}" has no GGUF tensor mapping — the '
"v1 backend maps layer groups, the embedding, and the output "
"head (ADR-0012)"
"backend maps layer groups, routed-expert stacks, the "
"embedding, and the output head (ADR-0012, ADR-0022)"
)
overrides.append(
TypeOverride(
pattern=rf"blk\.{match.group(1)}\.",
quant_type=ggml_type_for(assignment.bits),
)
)
return tuple(overrides)
bits = ggml_type_for(assignment.bits)
layers.append(TypeOverride(rf"blk\.{match.group(1)}\.", bits))
return tuple(stacks) + tuple(layers)
7 changes: 4 additions & 3 deletions src/vramfit/domain/pack.py
Original file line number Diff line number Diff line change
Expand Up @@ -86,9 +86,10 @@ class PackResult:
scan measured one, the embedding assignment otherwise
(ADR-0012). None when the recipe has neither group.
overrides (tuple[TypeOverride, ...]): Ordered per-tensor
overrides, in recipe order. Patterns are unique — the
quantizer applies the first match, so a duplicate would
silently shadow its successor.
overrides: protections, then expert stacks, then layer
groups, each in recipe order. Order carries priority.
The quantizer applies the first match, so a broader
pattern placed first would shadow a narrower one.
imatrix_path (str | None): Importance matrix file driven into
the quantizer (ADR-0016). None when the pack ran without
one.
Expand Down
Loading
Loading