Benchmark: Gemma4 ORT GenAI inference — f16 vs Q4_K_M vs NF4 (CUDA + CPU)

[justinchuby]

Summary

Benchmark of Gemma4 inference through onnxruntime-genai on ONNX models exported with mobius, comparing full-precision vs INT4-quantized decoders across CUDA and CPU. Quantization is done with Olive (OnnxKQuantQuantization / OnnxBnb4Quantization).

TL;DR: Q4_K_M is the recommended format — it is the fastest on CPU (+80% tok/s) and slightly faster than f16 on CUDA (+4% tok/s), at ~3.1x smaller decoder. NF4 should be avoided — it is slower than f16 on both EPs.

Environment

GPU	NVIDIA H200 (143 GB)
onnxruntime	1.27.0
onnxruntime-genai	0.14.0-dev (commit 9b875c3, + per-layer-KV patch)
mobius	44fdbd0
Model	google/gemma-4-E2B-it (text-only decode path)
Quantizer	Olive — Q4_K_M (OnnxKQuantQuantization, bits=4, block_size=32), NF4 (OnnxBnb4Quantization)

Methodology: 92-token prompt, forced 64-token decode (min_length==max_length, greedy), 2 warmup runs + best of 3. TTFT = prefill + first token. Decode tok/s = steady-state after first token.

Results — gemma-4-E2B-it

CUDA EP (H200)

Dtype	Decode tok/s	TTFT (ms)	Decoder size	vs f16 (decode)
f16	148.0	7.5	4.73 GB	baseline
Q4_K_M	153.5	10.4	1.51 GB	+3.7%
NF4	127.8	11.6	1.41 GB	−13.6%

CPU EP

Dtype	Decode tok/s	TTFT (ms)	vs f16 (decode)
f16	15.8	327	baseline
Q4_K_M	28.5	494	+80.4%
NF4	8.3	443	−47.5%

Key findings

1. Q4_K_M wins overall. Fastest decode on CPU (+80%) and slightly faster than f16 on CUDA (+4%), with a ~3.1x smaller decoder (4.73 GB → 1.51 GB). No quality regression observed in spot checks (consistent with the L4/L5 golden tests).
2. NF4 is not worth it for inference. Slower than f16 on both CUDA (−14%) and CPU (−47%) despite similar size to Q4_K_M.
3. Quantization raises TTFT but lowers decode latency. Prefill is compute-bound and pays a dequant cost (TTFT f16 7.5 ms → Q4_K_M 10.4 ms on CUDA), while decode is memory-bandwidth-bound and benefits from the smaller weights. For typical chat workloads (decode-dominated) the net effect is a win.
4. Single-token batch=1 decode underutilizes the H200 — throughput is memory-bound, so the absolute CUDA tok/s would not improve much on a larger GPU. Batching would.

gemma-4-12B (unified multimodal) — status

12B was exported (bf16 + Q4_K_M, decoder 23 GB → 6.8 GB) and the files load, but end-to-end genai generation is not yet runnable here: the global-attention layers (head_dim=512) emit the standard Attention op, which has no CUDA kernel and falls back to the CPU EP, where it fails on the mixed-KV-cache total_sequence_length check. Tracking separately; not included in the throughput numbers above. (A bf16 loadability fix for the unified vision/audio Compress op landed in 44fdbd0.)

Reproduction

# Export
mobius build --model google/gemma-4-E2B-it --dtype f16 --ep cuda \
  --runtime ort-genai /tmp/g4e2b/f16/cuda/
# Quantize (Olive, decoder only) — see examples/gemma4_unified_ort_genai.py:quantize_decoder
# Benchmark: 92-tok prompt, 64-tok forced greedy decode, 2 warmup + best-of-3

[justinchuby]

Why is NF4 slower than f16 (and Q4_K_M)? — and the "NF4 hardware acceleration" question

Short answer: there is no NVIDIA hardware acceleration for NF4. NF4 makes the model smaller, but on every current GPU it is dequantized to fp16/bf16 before the matmul, so it adds dequant work without ever reaching a faster compute path. That's exactly what the benchmark shows.

NF4 is a software codebook, not a hardware numeric type

NF4 ("NormalFloat-4") comes from QLoRA / bitsandbytes. It is a non-uniform 4-bit codebook: the 16 representable values are the quantiles of a normal distribution, stored per block with an absmax scale (often double-quantized). It was designed to make fine-tuning memory-cheap (keep frozen weights in 4 bits, dequant on the fly), not to make inference faster.

NVIDIA Tensor Cores cannot multiply NF4 operands. There is no NF4 datapath — the values are decompressed to fp16 immediately before the GEMM. The only 4-bit formats with a native tensor-core path are:

4-bit format	Native tensor-core matmul?	Notes
INT4	Yes (Turing→Hopper; uniform integer)	being de-emphasized in newer arch
NVFP4 / MXFP4	Yes, only on Blackwell (5th-gen TC microscaling)	hardware FP4 with per-block scales
NF4	No — on any GPU	software codebook, always dequant→fp16

So NF4 is not one of the hardware-accelerated 4-bit formats. It is unrelated to NVFP4/MXFP4 despite all being "4-bit."

Why it's slower than Q4_K_M specifically

Both Q4_K_M and NF4 dequantize to fp16 for the GEMM (LLM decode at batch=1 is memory-bandwidth bound, so neither gets a compute speedup from being 4-bit — the win is only smaller weight reads). The difference is the dequant cost and kernel maturity:

• Q4_K_M → ORT MatMulNBits: uniform int4 + per-block scale/zero-point. Cheap, vectorizable dequant with a heavily-optimized CUDA and AVX/CPU kernel.
• NF4 → ORT MatMulBnb4: a lookup-table dequant of the non-uniform 16-level codebook plus blockwise (double-quantized) absmax scaling, on a less-optimized kernel.

Net: NF4 pays a heavier per-weight dequant than Q4_K_M and lacks the well-tuned MatMulNBits path, while getting zero compute benefit over fp16. That's why it lands below f16 on both EPs (−14% CUDA, −47% CPU). The CPU gap is much worse because the table-lookup dequant doesn't SIMD-vectorize the way MatMulNBits' block dequant does.

Takeaway

NF4 is the wrong tool for inference — it's a fine-tuning memory format. For deployment, Q4_K_M is the right 4-bit choice here (same ~3x size win, faster decode, mature kernels). If we ever want actual hardware-accelerated 4-bit inference, the path is INT4 today or NVFP4/MXFP4 on Blackwell — not NF4.