Emit GGUF zero points explicitly

[justinchuby]

Summary

This is the clean-main rebuild of the urgent correctness fix from #458. It contains only the GGUF zero-point fix, based on current origin/main.

GGUF Q4_0 and Q8_0 are symmetric formats, but their dequantization formulas still require non-zero zero points:

• Q4_0: (q - 8) * scale
• Q8_0: (q - 128) * scale

Mobius previously marked those GGUF imports as symmetric, so QuantizedLinear and QuantizedEmbedding omitted the optional zero_points input for MatMulNBits and GatherBlockQuantized. That is not portable: GatherBlockQuantized defaults diverge between ORT CPU and ORT CUDA when zero_points is omitted, corrupting CUDA embeddings before the first decoder layer runs.

This PR makes GGUF Q4_0/Q8_0 emit explicit zero-point initializers, so both GatherBlockQuantized and MatMulNBits receive the intended values instead of relying on EP defaults.

Regression test

Added a single-node GatherBlockQuantized runtime regression:

• qweight nibbles are all 10
• scales are known constants
• explicit zero point is 8
• output is asserted against hand-computed (q - 8) * scale
• rerunning with zero point 0 is asserted not to match

I also temporarily flipped the test's zero point to 0; it failed with a 100% output mismatch, so the test checks the value and not just the presence of the input.

Synthetic GGUF tests now also assert:

• Q4_0 MatMulNBits nodes have the explicit fourth input
• Q4_0 GatherBlockQuantized has the explicit fourth input
• Q4_0 embedding zero points are packed as 0x88

Quantization types checked

Direct GGUF repackers already produce explicit zero-points for supported direct formats:

• Q4_0: zp=8
• Q4_1: per-block affine zp
• Q4_K: requantized per-block affine zp
• Q8_0: zp=128
• Q1_0: zp=1; Tencent Q1_0 uses its custom zp path

Q5_0/Q5_1/Q5_K/Q6_K are not direct MatMulNBits repack targets here; when they appear in mixed presets they go through dequantize+requantize/native-block fallback paths, which produce explicit zero-points where needed.

Validation

• python -m pytest src\mobius\integrations\gguf\_builder_test.py -q -> 33 passed
• python -m pytest src\mobius\integrations\gguf\ src\mobius\_configs\ src\mobius\_model_package_test.py -> 304 passed
• python -m ruff check src\mobius\integrations\gguf\_builder.py src\mobius\integrations\gguf\_builder_test.py -> passed
• python -m ruff format --check src\mobius\integrations\gguf\_builder.py src\mobius\integrations\gguf\_builder_test.py -> passed

Fresh conversion validated with C:\Users\justinchu\dev\models-gguf\qwen2.5-0.5b-instruct-q4_0.gguf:

python -m mobius build-gguf C:\Users\justinchu\dev\models-gguf\qwen2.5-0.5b-instruct-q4_0.gguf --output C:\Users\justinchu\dev\models\qwen2.5-0.5b-q4_0-mobius --keep-quantized --dtype f16 --ep cuda

Converted graph verification:

• GatherBlockQuantized input count: 4
• all MatMulNBits input counts: 4
• zero-point initializers: 170

Generation checks on the freshly converted model, no post-hoc graph patching:

• ORT CPU via onnx-genai CLI: The capital of France is Paris. It is the largest city in
• ORT CUDA via onnx-genai CLI (ONNX_GENAI_ORT_LIB_DIR pointed at the installed onnxruntime-gpu package, CUDA/cuDNN DLL dirs on PATH): The capital of France is Paris. It is the largest city in
• native CUDA via onnx-genai CLI (--features native-cuda, CUDA/cuDNN DLL dirs on PATH): The capital of France is Paris. It is the largest city in

Upstream ORT issue

Reported the provider default divergence here: microsoft/onnxruntime#31692

Relationship to #458

#458 was accidentally based on another in-flight feature branch and includes unrelated commits. This PR is the clean-main replacement for the urgent zero-point correctness fix only.

[Copilot]

Pull request overview

This PR fixes GGUF Q4_0/Q8_0 correctness by ensuring the ONNX graph always supplies explicit zero_points inputs to MatMulNBits and GatherBlockQuantized, avoiding provider-dependent defaults (notably CPU vs CUDA divergence) that can corrupt embeddings.

Changes:

• Update GGUF quant-param detection so Q4_0/Q8_0 no longer allow omitting zero_points (forcing explicit zp initializers).
• Add/extend GGUF build tests to assert explicit .zero_points initializers and 4-input contrib ops, plus a runtime regression for GatherBlockQuantized dequantization semantics.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
src/mobius/integrations/gguf/_builder.py	Changes quant-param detection logic to force explicit zero-points for Q4_0/Q8_0 (and updates related naming/semantics).
src/mobius/integrations/gguf/_builder_test.py	Adds assertions + a runtime regression to verify correct zero-point handling for MatMulNBits / GatherBlockQuantized.

Suppressed comments (1)

src/mobius/integrations/gguf/_builder.py:535

• The return contract is still documented as is_symmetric, but the implementation now treats the third return value as “can omit zero_points” (and Q4_0/Q8_0 now return False despite being symmetric formats). Updating the docstring helps prevent future callers from misusing this boolean.

    # Whether the graph can omit zero_points for each supported GGUF type.
    #
    # Mainline Q1_0 (1-bit binary) is repacked into 2-bit MatMulNBits
    # with zp=1 — see _repack_q1_0. Tencent's custom Q1_0 (2-bit SEQ,
    # 512-elt blocks) is inflated to 4-bit MatMulNBits with zp=3 — see

[justinchuby]

@copilot do not use onnx or helper module, use onnx-ir instead for building the test model. Also fix lint and unit tests

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ justinchuby
❌ Copilot
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Copilot]

@copilot do not use onnx or helper module, use onnx-ir instead for building the test model. Also fix lint and unit tests

Addressed in commits f77907f and d5182bc. The GatherBlockQuantized runtime regression now builds the model with onnx_ir (no onnx/helper usage), and the MatMulNBits assertion now validates all MatMulNBits nodes have explicit zero_points inputs. I also ran lint/format checks and unit tests for src/mobius/integrations/gguf/_builder_test.py (all passing).