model: add Kimi-K3 text model

[pwilkin]

Hybrid KDA (linear) + MLA (full) attention as in Kimi-Linear-48B, plus five things that architecture does not have:

1. cross-layer residual attention (attn_res_block_size)
2. latent MoE (routed experts run at n_expert_latent)
3. situ activation (replaces SwiGLU everywhere)
4. MLA output gate (sigmoid gate before o_proj)
5. full-rank KDA gate (single ssm_g instead of ssm_g_a/ssm_g_b)

Reuses DeepSeek4's HC_PRE for the cross-layer residual weighted sum. Supports repack for the MXFP4 weights in conversion.

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes, ran the conversion session with Opus.

Now need someone to actually convert and test :)

[ngxson]

may need to shorten comments too, IMO some/most comments are too verbose

[ngxson]

I'd suggest renaming:

• ckpts --> resi (short for residual stream, same naming mentioned in the paper)
• drop the _cache since technically there is no cache here, resi_stack should be enough

also, it might be cleaner if these are grouped into a new struct and explicitly pass it like this:

struct attn_resi; // private struct, defined inside cpp file
void res_push(attn_resi r, int64_t n_embd, int64_t n_tokens);
ggml_tensor * res_stack(attn_resi r, int64_t n_embd, int64_t n_tokens);

[ngxson]

these might not be necessary, I suppose for compat?

[pwilkin]

Yeah, those are artifacts from the mock model runs, for the real model I'll purge them.

[Green-Sky]

https://huggingface.co/inference-optimization/Kimi-K3-0.18B

potentially useful, depending on how faithful the model is reconstructed in 0.18B .

[pwilkin]

@Green-Sky I wouldn't put it up without checking parity with a mock model 😄

[Green-Sky]

@Green-Sky I wouldn't put it up without checking parity with a mock model 😄

Isnt that a mock model?

[pwilkin]

Yeah, that's what I'm saying, already built one of my own for parity testing purposes when doing the PR.

[ngxson]

@GrEarl please put your comment inside a collapsible block, it takes up too much space & make the discussion hard to keep track

[GrEarl]

Sorry for the long text. reposting it collapsed and trimmed.

Ran this branch against the actual Kimi-K3 checkpoint. One thing needs fixing; the rest is context.

LLAMA_MAX_EXPERTS is 512, Kimi-K3 has 896

src/llama-hparams.h:11    #define LLAMA_MAX_EXPERTS 512 // Qwen3 Next
src/llama-model.cpp:1117  GGML_ASSERT(hparams.n_expert <= LLAMA_MAX_EXPERTS);

Arch-generic path, right after LLM_KV_EXPERT_COUNT is read and before any arch hook. The converter itself prints gguf: expert count = 896, so a converted file aborts at load.

It cannot be worked around on the file side: understating expert_count contradicts ne[2] = 896 on the ffn_*_exps tensors.

LLAMA_MAX_EXPERTS sizes exactly one thing in the tree — cur_experts[LLAMA_MAX_EXPERTS] in build_moe_ffn (src/llama-graph.cpp:2128) — and only [0, n_expert_used) is ever touched, so 1024 costs 4 KB of stack.

--- a/src/llama-hparams.h
+++ b/src/llama-hparams.h
@@
 // bump if necessary
 #define LLAMA_MAX_LAYERS  512
-#define LLAMA_MAX_EXPERTS 512 // Qwen3 Next
+#define LLAMA_MAX_EXPERTS 1024 // Kimi-K3

Not verified: I have not confirmed a K3 GGUF actually loads with this — I do not have a machine that can hold 938 GiB. It is inferred from the assert and the single array it bounds. Whether to include it here or split it out is your call.

Dry run: prepare_tensors() completes, all 93 layers mapped, file type = 38

convert_hf_to_gguf.py --remote moonshotai/Kimi-K3 --dry-run --outtype auto, ~3 min on 8 vCPU / 32 GiB.

• hparams read correctly: context length 1048576 / embedding length 7168 / feed forward length 33792 / head count 96 / expert count 896 / experts used count 16 / expert score gating function sigmoid
• file type = 38 (MOSTLY_MXFP4_MOE), so the MXFP4 detection and the ftype override both take effect
• tensor map produced for all 93 layers with no unmapped names, including ssm_g / ssm_conv1d_{q,k,v} / ssm_{a,beta,dt,f_a,f_b,norm} / attn_res_score / ffn_res_score / output_res_score / ffn_routed_{down,up,norm} / exp_probs_b, and the MLA layers' attn_{q_a,q_b,kv_a_mqa,k_b,v_b,gate}
• the expert writing path returned without raising, so every (layer, w1|w2|w3) group had all 896 weight_packed/weight_scale pairs
• every *_res_norm / *_res_proj pair was fused, nothing left over

I did not write a complete GGUF or load one, so none of this says anything about inference correctness.

repack_mxfp4_blocks is bit-exact — 0 mismatches over 11,010,048 values

In --dry-run the expert tensors stay lazy, so the dry run never executes repack_mxfp4_blocks. I checked it separately.

Before seeing this PR I had derived the same repack from the compressed-tensors and ggml sources. Your output is byte-identical to mine, on layers.48.block_sparse_moe.experts.0.w1 (logical [3072, 3584]), fetching 5.6 MiB via HTTP Range:

mismatched elements : 0 / 11010048
max abs error       : 0.0
bytes in/out        : 5849088 / 5849088
bpw                 : 4.2500

Reference side: compressed-tensors nvfp4/helpers.py::pack_fp4_to_uint8 (element 2i in the low nibble, sign in bit 3, kE2M1ToFloat = [0, .5, 1, 1.5, 2, 3, 4, 6]) and mx_utils.py::decompress_mx_scale (2 ** (e - 127)). ggml side: dequantize_row_mxfp4 in ggml-quants.c, kvalues_fp4 in ggml-common.h, ggml_e8m0_to_fp32_half in ggml-impl.h (bits = (x - 1) << 23 for x >= 2).

So the docstring claim — kvalues doubled, scale halved, represented value unchanged — holds exactly. E8M0 exponents on this tensor are 120..122, so the x < 2 branch is not exercised by this data. Since the diff moves DeepSeek-V4's _pack_mxfp4_blocks into base.py and aliases it back, this covers that path too.

Reproduction, from a checkout of this branch (fetches ~5.6 MiB, stores no weights):

#!/usr/bin/env python3
import json
import urllib.request

import numpy as np
import torch

from conversion.base import repack_mxfp4_blocks

REPO = "moonshotai/Kimi-K3"
SHARD = "model-00049-of-000096.safetensors"
URL = f"https://huggingface.co/{REPO}/resolve/main/{SHARD}"
E2M1 = np.array([0.0, .5, 1., 1.5, 2., 3., 4., 6.], dtype=np.float32)
KV = np.array([0, 1, 2, 3, 4, 6, 8, 12, 0, -1, -2, -3, -4, -6, -8, -12], dtype=np.int8)


def rng(a, b):
    req = urllib.request.Request(URL, headers={"Range": f"bytes={a}-{b}"})
    return urllib.request.urlopen(req).read()


n = int.from_bytes(rng(0, 7), "little")
hdr = json.loads(rng(8, 8 + n - 1))
name = next(k for k in sorted(hdr) if k.endswith("experts.0.w1.weight_packed"))


def get(k):
    s, e = hdr[k]["data_offsets"]
    return np.frombuffer(rng(8 + n + s, 8 + n + e - 1), np.uint8).reshape(hdr[k]["shape"])


packed = get(name)
scale = get(name.replace("weight_packed", "weight_scale"))
rows, cols = packed.shape[0], packed.shape[1] * 2

# reference: compressed-tensors pack_fp4_to_uint8 / decompress_mx_scale, inverted
idx = np.empty((rows, cols), np.uint8)
idx[:, 0::2], idx[:, 1::2] = packed & 0xF, packed >> 4
mag = E2M1[(idx & 7).astype(np.int32)]
ref = np.where((idx & 8) != 0, -mag, mag).reshape(rows, cols // 32, 32) \
    * np.ldexp(np.ones(scale.shape, np.float32), scale.astype(np.int32) - 127)[:, :, None]

# ggml: dequantize_row_mxfp4 on the repacked blocks
blk = repack_mxfp4_blocks(torch.from_numpy(packed.copy()),
                          torch.from_numpy(scale.copy())).reshape(rows, -1, 17)
out = np.empty((rows, blk.shape[1], 32), np.float32)
out[:, :, :16] = KV[(blk[:, :, 1:] & 0xF)]
out[:, :, 16:] = KV[(blk[:, :, 1:] >> 4)]
got = out * np.ldexp(np.ones(blk.shape[:2], np.float32),
                     blk[:, :, 0].astype(np.int32) - 128)[:, :, None]

ref, got = ref.reshape(rows, cols), got.reshape(rows, cols)
print(f"{name}  [{rows}, {cols}]")
print(f"mismatched elements : {int((ref != got).sum())} / {ref.size}")
print(f"max abs error       : {float(np.abs(ref - got).max())}")
print(f"bytes in/out        : {packed.nbytes + scale.nbytes} / {blk.nbytes}")
print(f"bpw                 : {blk.nbytes * 8 / (rows * cols):.4f}")

--remote does not fetch the tokenizer files — not a problem with this PR

This is where my dry run stopped. Relevant only if you test with --remote.

Download patterns are ["LICENSE", "*.json", "*.md", "*.txt", "tokenizer.model"]. K3 keeps its vocabulary in tiktoken.model and its tokenizer class in tokenization_kimi.py (which relative-imports encoding_k3.py), so neither arrives, and AutoTokenizer.from_pretrained(self.dir_model, trust_remote_code=True) in kimi_linear.py fails with does not appear to have a file named tokenization_kimi.py. Kimi-K2 / K2.5 reach the same call through deepseek.py.

Measured with snapshot_download: adding *.model and *.py goes from 15 files / 57.05 MiB to 33 files / 59.89 MiB (+2.84 MiB) and brings in all three. Most of the 57 MiB is model.safetensors.index.json, already pulled by *.json. self.remote_hf_model_id is available on the model instance if letting transformers resolve the code from the Hub is preferable to widening the patterns.

With a complete local snapshot, set_vocab in this PR works as-is. What I checked:

• AutoTokenizer.from_pretrained(dir, trust_remote_code=True) returns TikTokenTokenizer
• model._mergeable_ranks has 163,584 entries and special_tokens 256; 163,584 + 256 = 163,840 = vocab_size exactly, so no UNUSED holes
• get_vocab_base_pre hashes to 81212dc7cdb7e0c1074ca62c5aeab0d43c9f52b8a737be7b12a777c953027890, which base.py maps to kimi-k2 — no new pre-tokenizer needed, as your comment says
• merge reconstruction yields 163,328 entries; tokens[163584] == "[BOS]", tokens[163839] == "[PAD]"
• tokenizer.eos_id is 163585 ([EOS]) while config.json and generation_config.json both say 163586 (<|end_of_msg|>), so the eos restore in set_vocab is doing real work

One note for anyone converting: tokenization_kimi.py imports tiktoken, and tiktoken.load.read_file requires blobfile even for a local path. Neither is in requirements/requirements-convert_hf_to_gguf.txt.

AI usage disclosure: I found this while building a quantized model with the help of AI. Because of that, some of my views here may be technically wrong, or wrong about how llama.cpp is implemented. I am also not fluent in English, so I used AI to help with the translation.

[thispwd]

Tested this branch today with a converted K3 checkpoint. Everything loaded successfully and generation worked as expected.

Conversion (conversion/kimi_k3.py)

• Dry run correctly detected the model (896 experts, MXFP4_MOE ftype).
• Full export from local safetensors completed cleanly, producing 2,573 tensors split across multiple output files.
• No manual fixes were required.

Runtime

• Running with hybrid GPU + CPU offload (-ngl 99 with -ot keeping part of the expert stacks in system RAM).
• Output is coherent and accurate on factual Q&A.
• Thinking segments are parsed correctly into reasoning_content via the OpenAI-compatible server.

Two observations that may help others:

1. -ot layer placement
   When using -ot to keep only a subset of MoE experts on the GPU, avoid contiguous layer ranges. Layers are distributed evenly across devices, so configurations like "layers 0–34 on GPU" can overload the first GPU(s). Interleaving layers (e.g. every Nth layer on the GPU) results in a much more balanced memory distribution.

2. XTML chat template
   The XTML chat template currently circulating uses namespace({...}), which Minja rejects with:

   namespace() arguments must be kwargs

   Changing it to namespace(key=value, ...) resolves the issue.

Thank you.

[usrlocalben]

@GrEarl 's late-edition Q2 works with the same template issue/workaround as @thispwd describes.
edit: I now see that the Q2 had its 1-of-N GGUF shard updated with an improved chat template.

There are stray template elements (or special tokens?) in the output, but the reasoning parser still detects begin/end and eos token ends the turn properly.

e.g.

<|open|>think<|sep|>The user is asking me to "tell the one about
the cat and the fire extinguisher again" — as if we've had a prior
conversation where I told a story or joke about a cat and a fire extinguisher.

---snip---

If you remember details from the original, tell me and I'll work
them in — or I can spin a completely different take.<|close|>response<|sep|><|close|>message<|sep|>

(newlines added to avoid side-scrolling)

It would be helpful if llama warmed up the experts. It doesn't anymore, or doesn't with this model. The warmup run does not force all experts on. I know this is also a problem for DSv4 warmup.

Many long stories about cats or doubly-linked list impls are needed to get 2^7*7 mmap experts warmed up.

The no-imatrix (I think I can safely assume) Q2 quant is coherent, and seems quite capable at a glance.

[csabakecskemeti]

FYI: Here's another chat template PR on HF: https://huggingface.co/moonshotai/Kimi-K3/discussions/66

[usrlocalben]

FYI: Here's another chat template PR on HF: https://huggingface.co/moonshotai/Kimi-K3/discussions/66

This is the template I used.

[csabakecskemeti]

I can also confirm the conversion and the chat template gives coherent answer
Full precision cmoe mmap (system: 128gb vram, 1tb ram, experts loaded from nvme)

[SolshineCode]

Tested kimi-k3-text @ cf11c4c end to end with a synthetic fixture. Greedy outputs from llama-server match the HF reference implementation exactly, token for token, on all 3 test prompts.

Method: I generated a random-init shrunk K3 (90M params, f32) that keeps the full structure: hybrid KDA/MLA with the real linear_attn_config schedule shape (full attn every 4th layer plus the last), latent MoE with 16 experts and routed_expert_hidden_size, attention-residual blocks, situ, the MLA output gate, the full-rank KDA gate, and the real tokenizer. One honest caveat about what "reference" means here: the released modeling code can't run on CPU as shipped (fla's KDA kernels are Triton-only), so I rerouted them to fla's own naive torch ops (naive_recurrent_kda and friends). The match is against fla's reference math, not the fused kernels. Comparison was done by POSTing raw input_ids to /completion with top_k=1, which validates the graph independent of tokenization, and the decoded text matched too.

Hardware is deliberately ancient: 2x Xeon E5-2609v2 (AVX only, no AVX2), 512GB DDR3, CPU-only build so far. Build is clean on this ISA, conversion works (tensor mapping, res_norm x res_proj fusion, expert merge, tokenizer; transformers git-main / 5.15.0.dev0), and llama-server generates at ~55 tok/s on the tiny model.

Two problems hit along the way:

1. llama-cli hangs silently in --no-conversation mode (repro 3/3, with and without stdin closed; conversation mode is fine). Might not be K3-specific.
2. convert_hf_to_gguf breaks on transformers >= 5.15: bytes_to_unicode moved out of models.gpt2.tokenization_gpt2 (it's in convert_slow_tokenizer now). A one-line try/except in conversion/qwen.py fixes it. Happy to PR that separately.

Fixture generator, CPU shim, and reference outputs: https://gist.github.com/SolshineCode/3115760b0c3b655563a3102ba897c426. I can upstream the fixture into the test suite if useful, or defer to @200lz if their fixtures already cover this. Once a real quant exists I can also run a full-scale streamed validation on this box (512GB RAM, mmap + -ot exps=CPU), and it has sm_52 Teslas if legacy-CUDA coverage is ever wanted.

AI usage disclosure: the test harness and this report were built with Claude running on my machine; the numbers are from real runs I can rerun on request.

[200lz]

@SolshineCode Excellent validation—thank you for sharing the fixture and the CPU reference path.

My work does not duplicate your end-to-end fixture. I analyzed the released checkpoint schema using all 96 official safetensors headers, without downloading tensor payloads. The production checkpoint confirms:

• layer 0 is dense, while layers 1–92 are routed MoE;
• every MoE layer contains the exact expert set 0..895;
• KDA layers are 0–2, 4–6, ... , 88–90;
• MLA layers are 3, 7, ... , 87, 91, 92;
• the routed-expert inventory contains exactly 92 × 896 × 3 = 247,296 MXFP4 block/scale pairs;
• Attention Residual uses four per-layer tensors, two output tensors, and 12-layer block boundaries.

Please do not defer the execution fixture to me—your fixture covers an area I have not implemented and would be valuable upstream.

I can instead review the PR against the full released tensor vocabulary and contribute a compact schema-level regression test, if useful, covering the exceptional final MLA layer, the dense-to-MoE boundary, expert-set invariants, and self_attn.g_proj mapping.

I’ll first inspect the current PR branch to avoid duplicating existing tests.

[SolshineCode]

Ran the fixture against the CUDA build as well: kimi-k3-text @ cf11c4c, CUDA 12.4 with -DCMAKE_CUDA_ARCHITECTURES=52, 2x Tesla M40 (compute 5.2), -ngl 99 --tensor-split 1,1. All 3 prompts match the reference token for token, ~109 t/s vs ~55 CPU-only. One flag: load prints resolve_fused_ops: layer 3 is assigned to device CUDA0 but Flash Attention is assigned to device CPU (usually due to missing support), so at least one attention op lacks an sm_52 kernel. Logs: https://gist.github.com/SolshineCode/3115760b0c3b655563a3102ba897c426 (sm52_results.md)

@200lz thanks, schema-level checks are exactly what my fixture doesn't do, so those complement each other well. I cross-checked your header-derived layer map against the released config's linear_attn_config and they match exactly (your 0-indexed MLA 3, 7, ..., 91, 92 is the config's 1-based full_attn_layers). The fixture already encodes both structural exceptions you mention: it ends with consecutive MLA layers like 91+92, and it has the dense layer 0 to MoE boundary. I'll upstream it as an execution test to sit alongside your schema regression test, in whatever form the maintainers prefer.

[200lz]

@SolshineCode Thanks for cross-checking the released layer map and for confirming the two structural exceptions in the fixture.

That separation sounds ideal: your fixture can cover execution and token-level parity, while I’ll focus on compact schema regression coverage derived from the official checkpoint.

I’ll review the current PR tests and prepare the smallest non-duplicative schema test proposal, especially around the final MLA layer, the dense/MoE boundary, expert-set invariants, and tensor-name mapping.

[200lz]

@pwilkin I completed a read-only test-gap review of this PR against the full released K3 checkpoint schema.

The external execution fixture provides strong CPU/CUDA parity coverage, but the PR currently has no dedicated K3 schema regression test in CI.

I would like to contribute a compact converter-level test covering:

• the final consecutive MLA layers;
• layer 0 dense vs. layers 1+ MoE;
• official expert/shared-expert invariants;
• Attention Residual block grouping;
• and context-sensitive self_attn.g_proj mapping.

The test would use synthetic config/tensor metadata only—no model weights, runtime execution, or overlap with the existing fixture.

Would you prefer this added to #26185, or submitted as a small follow-up PR after merge?

[idumlupinar]

Can we run this text model on dual RTX 3090 and 128GB DDR4 memory and 5800x3d cpu on Windows 11?

[RodriMora]

Can we run this text model on dual RTX 3090 and 128GB DDR4 memory and 5800x3d cpu on Windows 11?

No, the lowest size, if by some miracle Aes or Bart make a Q1-2 with imatrix quant that is still coherent, that would even be 700GB

[csabakecskemeti]

Quality topic
Just an opinion based on my own quant, curious what's other's experience:
The intelligence per GB feels off with this model. I'm testing my Q2 quant with zero shot Mario clone, and it struggles to create a working (I've not meant errors in the html the game is not working) game. My reference is GLM-5.2 Q5 which has done it first shot at first try. I know the Q5 is less lossy than Q2 but at GB level it perform better. (At least in HTML coding)
This may an unfair comparison, or some issue with my quant. Otherwise the output is coherent and makes sense.

I also acknowledge that this is work in progress, also this maybe my local issue. Shared just for fyi

Looking forward to hear other's experience

[pwilkin]

@kat-palm good catch, thanks, messed that up on the rebase to master

[jukofyork]

Thanks for this PR — we've been running K3 on this branch for the past week on a CPU-heavy rig (2× Xeon Gold 6148, 1.5 TB DDR4-2666, RTX 4060 Ti 16 GB with -ot exps=CPU, Q3_K_M quant) and it's been solid for long agentic jobs at ~0.7 tok/s tg / ~2.3 pp. For anyone wondering whether the recycled-hardware class works: it does.

I've got a similar machine (dual Xeon Gold 6248, 1.5 TB DDR4-2666) but with dual RTX 6000 Ada in, and now I am getting:

• ~3 tokens/s TG (I had hyper-threading turned off before and only got 2.6 tokens/s).
• ~55 tokens/s (dropping to ~45 tokens/s at 128k context) PP.

for the full MXFP4 model created using:

~/llama.cpp/build/bin/llama-quantize \
  --tensor-type "ssm_=bf16" \
  --tensor-type "_exps=mxfp4" \
  Kimi-K3-BF16.gguf Kimi-K3-MXFP4.gguf Q8_0 44

---

For reference, compiled using:

#!/bin/bash

function safe_sed() {
    local file=$1
    local pattern=$2
    local replacement=$3

    # Check if pattern exists
    if ! sed -n "s/${pattern}/${replacement}/p" "$file" | grep -q .; then
        echo "Error: Pattern not found in $file: $pattern"
        return 1
    fi

    # Create backup
    cp "$file" "$file.bak"

    # Perform the replacement
    sed -i "s/${pattern}/${replacement}/g" "$file"

    # Show diff
    echo
    echo "Changes in '$file':"
    echo "-------------------"
    diff "$file.bak" "$file"

    # Clean up
    rm "$file.bak"

    echo "-------------------"
}

function safe_sed_function() {
    local file=$1
    local function_signature=$2
    local replacement=$3

    # Create backup
    cp "$file" "$file.bak"

    # Perform the replacement using address range and c command
    sed -i "/${function_signature}/,/^}/c\\${replacement}" "$file"

    # Show diff
    echo
    echo "Changes in '$file':"
    echo "-------------------"
    diff "$file.bak" "$file"

    # Clean up
    rm "$file.bak"

    echo "-------------------"
}

cd
rm -rf llama.cpp

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

# Fetch the PR and checkout
git fetch origin pull/26185/head:pr-26185
git checkout pr-26185

# Fix warmup bug.
safe_sed "src/models/kimi-k3.cpp" "hparams.n_expert," "n_expert,"
safe_sed "src/models/kimi-k3.cpp" "hparams.n_expert_used," "n_expert_used,"
safe_sed "common/common.cpp" "llama_decode(lctx, llama_batch_get_one(tmp.data(), std::min(tmp.size(), (size_t) params.n_batch)));" \
         "llama_set_warmup(lctx, true);\n            llama_decode(lctx, llama_batch_get_one(tmp.data(), std::min(tmp.size(), (size_t) params.n_batch)));\n            llama_set_warmup(lctx, false);"

# Pin each thread to its corresponding core.
safe_sed_function "ggml/src/ggml-cpu/ggml-cpu.c" \
  "^static void set_numa_thread_affinity(int thread_n) {$" \
  "static void set_numa_thread_affinity(int thread_n) {\n\
    cpu_set_t cpus;\n\
    CPU_ZERO(&cpus);\n\
    CPU_SET(thread_n, &cpus);\n\
    pthread_setaffinity_np(pthread_self(), sizeof(cpu_set_t), &cpus);\n\
}"

# Turn off to ensure we call ggml_backend_cuda_buffer_set_tensor and the cached-tensor patch to work.
safe_sed "ggml/src/ggml-backend.cpp" "if (split->graph.n_nodes > 0" "if (false"

# Use a pinned buffer to get full PCI-E transfer speed.
safe_sed_function "ggml/src/ggml-cuda/ggml-cuda.cu" \
  "^static void ggml_backend_cuda_buffer_set_tensor(" \
  "#include \"patch_ggml_backend_cuda_buffer_set_tensor_llama_cpp.cu\"\n"
  
cmake -B build -DGGML_NATIVE=ON -DGGML_CUDA=ON -DGGML_AVX512=ON -DGGML_AVX512_VNNI=ON
cmake --build build --config Release -- -j "$(nproc)"

• See here for the patch_ggml_backend_cuda_buffer_set_tensor_llama_cpp.cu file.
• The Xeon Gold 6148 will need the -DGGML_AVX512_VNNI=ON removing.

and run using this:

#!/bin/bash

PORT_NUMBER=8082

# Turn off NUMA balancing
echo 0 | sudo tee /proc/sys/kernel/numa_balancing > /dev/null

# Ask for permission to drop caches
read -p "Do you want to drop caches? (y/n) " -n 1 -r
echo    # Move to a new line
if [[ $REPLY =~ ^[Yy]$ ]]
then
    echo "Dropping caches..."
    sudo swapoff -a
    sudo swapon -a
    echo 3 | sudo tee /proc/sys/vm/drop_caches > /dev/null
fi

MODEL_NAME="Kimi-K3"
MODEL_FILE="${HOME}/models/gguf/Kimi-K3-MXFP4.gguf"
JINJA_FILE="${HOME}/models/${MODEL_NAME}.jinja"
ALIAS="$(hostname):${MODEL_NAME}"

export GGML_OP_OFFLOAD_MIN_BATCH=2048

export GGML_CUDA_EXPS_READAHEAD_THREADS=10
#export GGML_CUDA_EXPS_READAHEAD_DEBUG=1

export CUDA_VISIBLE_DEVICES=0,1
~/llama.cpp/build/bin/llama-server \
    --host $(hostname -I) \
    --port "$PORT_NUMBER" \
    --alias "$ALIAS" \
    --model "$MODEL_FILE" \
    --jinja \
    --chat-template-file "$JINJA_FILE" \
    --n-gpu-layers 99 \
    --flash-attn on \
    --numa distribute \
    --load-mode mmap \
    --threads "$(nproc)" \
    --tensor-split 43,50 \
    --override-tensor "exps=CPU" \
    --ctx_size 131072 \
    --batch-size 8192 \
    --ubatch-size 8192 \
    --parallel 1 \
    --no-cont-batching \
    --cache-ram 0 \
    --temp 1.0 \
    --top-p 0.95 \
    --min-p 0.01 \
    --verbosity 4

(*) I'm not sure if this model works properly with --cache-ram so turned it off for now, but since found my NVME the model was stored on was going bad...

---

I don't think these see that bad stats, and seem to match what I expect (eg: for kimi-2.x I get ~9 tokens/s for TG and ~150 tokens/s PP) and this model has around 3x the active parameters... It's "usable" but debatable if it's actually worthwhile over using kimi-2.6 or glm-5.2.

[jukofyork]

0.01.766.673 I print_info: arch                  = kimi-k3
0.01.766.674 I print_info: vocab_only            = 0
0.01.766.674 I print_info: no_alloc              = 0
0.01.766.674 I print_info: n_ctx_train           = 1048576
0.01.766.675 I print_info: n_embd_inp            = 7168
0.01.766.676 I print_info: n_embd                = 7168
0.01.766.676 I print_info: n_embd_out            = 7168
0.01.766.677 I print_info: n_layer               = 93
0.01.766.677 I print_info: n_layer_all           = 93
0.01.766.704 I print_info: n_head                = 96
0.01.766.714 I print_info: n_head_kv             = [0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 1]
0.01.766.714 I print_info: n_rot                 = 64
0.01.766.715 I print_info: n_swa                 = 0
0.01.766.716 I print_info: is_swa_any            = 0
0.01.766.716 I print_info: n_embd_head_k         = 576
0.01.766.716 I print_info: n_embd_head_v         = 74
0.01.766.724 I print_info: n_gqa                 = [0, 0, 0, 96, 0, 0, 0, 96, 0, 0, 0, 96, 0, 0, 0, 96, 0, 0, 0, 96, 0, 0, 0, 96, 0, 0, 0, 96, 0, 0, 0, 96, 0, 0, 0, 96, 0, 0, 0, 96, 0, 0, 0, 96, 0, 0, 0, 96, 0, 0, 0, 96, 0, 0, 0, 96, 0, 0, 0, 96, 0, 0, 0, 96, 0, 0, 0, 96, 0, 0, 0, 96, 0, 0, 0, 96, 0, 0, 0, 96, 0, 0, 0, 96, 0, 0, 0, 96, 0, 0, 0, 96, 96]
0.01.766.735 I print_info: n_embd_k_gqa          = [0, 0, 0, 576, 0, 0, 0, 576, 0, 0, 0, 576, 0, 0, 0, 576, 0, 0, 0, 576, 0, 0, 0, 576, 0, 0, 0, 576, 0, 0, 0, 576, 0, 0, 0, 576, 0, 0, 0, 576, 0, 0, 0, 576, 0, 0, 0, 576, 0, 0, 0, 576, 0, 0, 0, 576, 0, 0, 0, 576, 0, 0, 0, 576, 0, 0, 0, 576, 0, 0, 0, 576, 0, 0, 0, 576, 0, 0, 0, 576, 0, 0, 0, 576, 0, 0, 0, 576, 0, 0, 0, 576, 576]
0.01.766.742 I print_info: n_embd_v_gqa          = [0, 0, 0, 74, 0, 0, 0, 74, 0, 0, 0, 74, 0, 0, 0, 74, 0, 0, 0, 74, 0, 0, 0, 74, 0, 0, 0, 74, 0, 0, 0, 74, 0, 0, 0, 74, 0, 0, 0, 74, 0, 0, 0, 74, 0, 0, 0, 74, 0, 0, 0, 74, 0, 0, 0, 74, 0, 0, 0, 74, 0, 0, 0, 74, 0, 0, 0, 74, 0, 0, 0, 74, 0, 0, 0, 74, 0, 0, 0, 74, 0, 0, 0, 74, 0, 0, 0, 74, 0, 0, 0, 74, 74]

Just noticed this has 74 for n_embd_head_v and the MLA layers' n_embd_v_gqa. Where does this come from? I was expecting it to be 512 like the old MLA code, but can't see how it could be 74 here?

[jukofyork]

The original weights look a bit odd too:

    "qk_nope_head_dim": 128,
    "qk_rope_head_dim": 64,

as from the paper:

So what is stored in the extra 64 values created by kv_a_proj_with_mqa and q_b_proj if there is no RoPE and the heads are all dimension 128?

[jukofyork]

OK, so this guy seems to have found those 64 values are still used:

https://github.com/FareedKhan-dev/kimi-k3-in-c

but not rotated:

but I still can't work out where that 72 value comes from in llama.cpp or if it's important/used anywhere.

[fairydreaming]

Just noticed this has 74 for n_embd_head_v and the MLA layers' n_embd_v_gqa. Where does this come from? I was expecting it to be 512 like the old MLA code, but can't see how it could be 74 here?

@jukofyork Since there is no explicit attention.value_length in the GGUF llama.cpp calculates hparams.n_embd_head_v_full = hparams.n_embd / hparams.n_head() (7168 / 96) and that's how we get 74.

So the origin of this value is a certain very dark place. But I don't think this value is used anywhere in the model code.

[jukofyork]

It seems to be working really well for me.

Huge thanks to @pwilkin and @fairydreaming for getting this working so quickly!

[fairydreaming]

It seems to be working really well for me.

Huge thanks to @pwilkin and @fairydreaming for getting this working so quickly!

@jukofyork It's all @pwilkin work, I just helped with testing.

[pwilkin]

Think we should be able to merge it now.

[kat-palm]

Thanks so much for taking the time to write all this up. The warmup fix alone is gold, and the build/run scripts are exactly the kind of thing that saves a whole weekend.

Update from the recycled-hardware end: we built the same quant with your exact recipe (Q8_0 + --tensor-type "ssm_=bf16" --tensor-type "_exps=mxfp4", keep-split) on the 2× 6148 box — 7.25h to quantize, 1.44TB at 4.36 BPW. Your warmup fix is applied and the first load is running now; watching it actually touch all 896 experts on the way in is very satisfying. Noted on dropping AVX512_VNNI for the 6148, thanks.

Next on our list is your #16000 readahead port. With a single 4060 Ti 16GB we obviously won't see your pp numbers, but at this scale every multiple counts — we'll report back with before/after so the thread has a bottom-of-the-GPU-range data point.

For our use-case — an overnight second set of eyes on code review — it's a good exercise. A very large slow-cooker...

[CISC]

may need to shorten comments too, IMO some/most comments are too verbose

@pwilkin Address this please (that includes docstrings too).

[ngxson]

may need to shorten comments too, IMO some/most comments are too verbose

@pwilkin Address this please (that includes docstrings too).

btw, the most aligned way is to simply tell your agent "adapt/remove code comments to follow agents.md expectations"

[ngxson]

I'm taking over this PR now, ran a bot review pass offline and it pointed out some issues

will fix it and push commits directly here

[ngxson]

annoyingly this is the same [nope | rope] layout that I identified in #27091 , but I'm not trying to fix it here because that will require re-converting GGUF

hopefully #27120 get merged (plus all other backends support it) so that we can clean up this rope logic

[createthis]

Congrats on the merge! 🎉