UPSTREAM PR #19167: model : support LongCat-Flash-Lite (ngram embeddings)

[loci-dev]

Note

Source pull request: ggml-org/llama.cpp#19167

Target support: https://huggingface.co/meituan-longcat/LongCat-Flash-Lite

NOTE: I'm having quite limited time recently, but quite interested by the idea of ngram embedding. Pushing this PR in a rough state mostly for discussions.

The most complex part of the model is to integrate the idea of "ngram cache" inside llama.cpp. That means new tokens can "look back" to see N tokens that were added in the past.

For example: when I given ngram_n = 3 and given 5 input tokens t0, t1, t2, t3, t4. Each token can "see" these ngrams:

• t0: (none)
• t1: t0
• t2: t1 t0
• ...

This sounds similar to SWA logic but I can't wrap my head around how to reuse the code inside set_input_kq_mask_impl. CC @ggerganov if you have any ideas.

And to store the token ID of each position, llama_kv_cell_ext is extended to store this, quite similar to x/y position for m-rope.

The current implementation does return the ngram, what's left to do is to use this info to calculate the hash (quite simple, just some multiplications and modulo):

token[0] = 128000 : ngram = 0 0 0 0
token[1] = 13347 : ngram = 0 0 0 128000
token[2] = 10 : ngram = 0 0 128000 13347
token[3] = 4925 : ngram = 0 128000 13347 10
token[4] = 674 : ngram = 128000 13347 10 4925
token[5] = 220 : ngram = 13347 10 4925 674
token[6] = 15 : ngram = 10 4925 674 220
token[7] = 220 : ngram = 4925 674 220 15
token[8] = 16 : ngram = 674 220 15 220

[loci-review]

Overview

Analysis of 114,917 functions across 14 binaries reveals minimal performance impact from commit d0e14f7 implementing n-gram embedding infrastructure. Modified: 43 functions (0.037%), new: 71, removed: 10, unchanged: 114,793 (99.89%).

Power Consumption Changes:

• build.bin.libllama.so: +0.241% (249,135.79 nJ → 249,736.64 nJ)
• build.bin.llama-tts: +0.0003%
• build.bin.libmtmd.so: -0.0001%
• All other binaries (llama-cvector-generator, llama-bench, libggml-base.so, libggml-cpu.so, libggml.so, llama-tokenize, llama-quantize, llama-qwen2vl-cli, llama-gemma3-cli, llama-gguf-split, llama-llava-cli, llama-minicpmv-cli): 0.000%

Function Analysis

Root Cause: llama_kv_cell_ext structure size increased from 8 to 12 bytes (+50%) by adding llama_token id field for n-gram embedding support.

Most Impacted Functions (libllama.so):

• std::vector<llama_kv_cell_ext>::_S_max_size: Response time +159.9% (140.63ns → 365.48ns), throughput +217.5% (103.17ns → 327.57ns). Allocator max_size calculation affected by larger structure.

• std::vector<llama_kv_cell_ext>::_M_default_append: Response time +69.3% (1,947ns → 3,296ns), throughput +56.2% (197ns → 307ns). Default-constructing 50% larger structures during capacity expansion.

• std::vector<llama_kv_cell_ext>::resize: Response time +67.1% (2,019ns → 3,373ns), throughput +9.2% (64ns → 70ns). Allocating and initializing 50% more memory per cell.

• std::__new_allocator<llama_kv_cell_ext>::allocate: Response time +62.9% (101.44ns → 165.27ns), throughput +80.7% (79.49ns → 143.59ns). Allocating larger memory blocks.

• std::vector<llama_kv_cell_ext>::operator[] (const/non-const): Both +35.5% throughput (10.60ns → 14.36ns). Offset calculation changed from shift to multiplication for 12-byte elements.

• __gnu_cxx::__ops::__pred_iter (Mirostat v2): Response time +140.4% (120.32ns → 289.29ns), throughput +213.2% (79.04ns → 247.56ns). Compiler optimization variance in STL template instantiation.

• std::_Hashtable<llama_adapter_lora*>::end: Response time +138.0% (117.78ns → 280.32ns), throughput +194.6% (83.27ns → 245.36ns). Missing inline on llama_adapter_lora::get_weight() prevents optimization.

Improved Functions:

• std::unique_ptr<llm_graph_context>::operator= (GroVeMoE): Response time -8.9% (840ns → 765ns), throughput -49.5% (152ns → 77ns). Enhanced compiler inlining.

• std::__detail::_Executor::_M_is_line_terminator: Response time -18.1% (468ns → 383ns), throughput -35.5% (240ns → 155ns). Improved standard library optimization.

Other analyzed functions showed proportional changes consistent with the 50% structure size increase, all occurring in initialization paths rather than inference hot paths.

Additional Findings

Impact Isolation: Changes are well-contained to KV cache initialization. Per-token inference overhead is <10ns (<0.01% of token generation time). GPU operations (CUDA, Metal, HIP, Vulkan) remain unchanged with 0% impact on matrix operations and attention mechanisms. The implementation maintains trivial copyability and cache-friendly layout. Code marked "wip" with TODO for optimizing O(n) linear scan in get_last_n_tokens() to O(1) index-based lookup. Memory overhead: +512KB for typical 32-layer models (negligible on modern hardware).

🔎 Full breakdown: Loci Inspector.
💬 Questions? Tag @loci-dev.