Skip to content

IQ4_K: SOTA 4-bit quantization #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 7 commits into from
Jul 28, 2024
Merged

IQ4_K: SOTA 4-bit quantization #6

merged 7 commits into from
Jul 28, 2024

Conversation

@ikawrakow
Copy link
Owner

@ikawrakow ikawrakow commented Jul 28, 2024

  • Same 4.5 bpw as Q4_K.
  • Significantly reduces quantization error of LLaMA-3.1 (and also 3.0). E.g., 1.77% vs 2.9% for Q4_K_S for LLaMA-3.1-8B (with quantization error defined as PPL(Q)/PPL(fp16)-1)
  • Non-linear quantization similar to IQ4_XS and IQ4_NL with the following differences
    • Blocks of 16 instead of blocks of 32
    • Non-linear values in each block of 16 can be on the original non-linear grid, or can be on a shifted grid. This is indicated by one bit, so we need 16 extra bits per block of 256
    • So, we need 256 * 4 bits for the quants, 16 * 6 bits for the 6-bit block scales, 16 bits for the super-block float scale, and 16 bits for the shift bits, ending up with exactly 4.5 bpw
  • Performance is on par with Q4_K on AVX2 and CUDA, and slightly lower on ARM_NEON and Metal

Iwan Kawrakow added 7 commits July 27, 2024 17:05
iq4_k: basics
8a2d438 
* quantize/dequantize works
* CUDA dequantize works and one can run PPL calcs. I get
  PPL = 6.5258 for LlaMA-3.1-8B, which is 1.77% above fp16.
  In comparison, q4_K_S (same size) is 2.88% above fp16.
* TG on CUDA does not work. Johannes has changed the way i-quant dot
  products are done, so need to sort out what he had in mind
* iqk_mul_mat is not implemented.
iq4_k: TG now works on CUDA
41d20f6
iq4_k: AVX512 implementation
be34f76 
For LLaMA-3.1-8B we get PP-512 = 182.6 t/s, TG-128 = 13.6 t/s,
so almost the same as q4_K_S.
iq4_k: AVX2 implementation
db87f76 
For LLaMA-3.1-8B we get PP-512 = 203.1 t/s, TG-128 = 12.9 t/s
on the Ryzen-5975X.
iq4_k: NEON implementation
d89c88e 
For LLaMA-3.1-8B we get PP-512 = 60.7 t/s, TG-128 = 25.0 t/s
on the M2-Max. TG is on par with q4_K_S, PP is ~10% slower.
iq4_k: Metal implementation
8ffb645 
For LLaMA-3.1-8B we get PP-512 = 445 t/s, TG-128 = 46.3 t/s
on a 30-core M2-Max GPU. This is to be compared with (currently)
PP-512 = 460 t/s, TG-128 = 51 t/s for q4_K_S.
iq4_k: scalar dot product
b29f64e
@ikawrakow ikawrakow merged commit 291066e into main Jul 28, 2024
@nux nux mentioned this pull request May 17, 2025
Closed
@ciprianveg ciprianveg mentioned this pull request May 29, 2025
Closed
@Ph0rk0z Ph0rk0z mentioned this pull request Jul 23, 2025
Open
@ChicoPinto70 ChicoPinto70 mentioned this pull request Jul 26, 2025
Merged
4 tasks
@os360 os360 mentioned this pull request Aug 27, 2025
Closed
Nexesenex pushed a commit to Nexesenex/ik_llama.cpp.nxs that referenced this pull request Oct 26, 2025
@Thireus
Merge pull request ikawrakow#6 from Thireus/revert-4-revert-3-ik/chec…
916c9c3 
…k_up_gate_fmoe

Revert "Revert "Check if ffn_up and ffn_gate are of the same type before using fmoe""
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ikawrakow