UPSTREAM PR #19182: model: support Longcat-Flash (help wanted)

[loci-dev]

Mirrored from ggml-org/llama.cpp#19182

I was working on ggml-org/llama.cpp#19167 but realized that the normal (non-ngram) model is not even supported yet.

Thinking it will be simple, I gave it a try, but ended up stuck at implementing their notion of "zero-computing experts" (ref: link to paper)

The main problem is that ggml_mul_mat_id isn't made for this purpose and I have no idea how to adapt it, or which ops may need to be added to make it work.

---

To illustrate what's the problem, I will take an example of how a normal MoE FFN work:

• Calculate expert probs using the router
• Sort & get top_k experts
• Do FFN gate/up/down with the selected top_k experts, this is done via ggml_mul_mat_id
• Weighted sum the output

This means we spend the the same amount of computation for each token, proportionally to n_expert_used

However, with longcat-flash:

• After top_k expert, ONLY experts with ID < n_zero_experts will go through FFN; for the rest, they skip the FFN altogether
• This makes the amount of computation to be varied token-by-token. For example: one token can use n FFN expert, while another can use n-1, another can use 0 FFN expert (in other words, skipping the MoE altogether)

---

Apart from the weird MoE, the model has double block architecture, meaning there are 2 attentions and 2 FFNs per layer. Upon converting to GGUF, we convert it to a model of 2 * n_layer, which make the implementation much easier.

[loci-review]

Performance Review Report: llama.cpp Version Comparison

Executive Summary

Analysis of 12 functions across 5 commits (44bc40f → 1fa084e) reveals no performance impact on inference operations. All analyzed functions are C++ Standard Template Library (STL) utilities showing changes from build configuration differences, not algorithmic modifications. Cumulative overhead: <5 milliseconds per application lifecycle (<0.05% of inference time).

Impact Classification: Minor

Root Causes:

• 70% of changes: Debug build configuration (assertions enabled, reduced optimization)
• 20% of changes: Compiler upgrades (GCC-14 improvements)
• 10% of changes: Intentional debugging enhancements

Performance-Critical Areas (Unaffected):

• Matrix operations (70-90% of inference): No changes
• Attention mechanisms (10-20% of inference): No changes
• KV cache management: No changes
• Quantization kernels: No changes
• GPU/ML operations: No changes

Key Findings

Largest Changes:

1. llama_model_loader::done_getting_tensors(): +1,766ns response time (+91%) - Intentional debugging enhancement for model loading validation, called once per model load
2. std::vector<ulong>::_S_max_size(): +212ns (+151%) - Allocator utility in cvector-generator preprocessing tool
3. std::vector::begin(): +181ns (+216%) - Debug assertions added, affects iterator loops
4. std::vector::end(): -183ns (-69%) - Compiler optimization improvement
5. std::_Rb_tree::begin(): -182ns (-69%) - GCC-14 optimization improvement

Commit Context:
Work-in-progress development on model loading and tokenization ("wip" → "tokenizer" → "load ok" → "stuck here"). 65 files affected (23 modified, 38 added, 4 deleted).

Code Changes:
Only one intentional code modification: done_getting_tensors() now logs missing tensors before throwing exceptions, providing detailed diagnostics at 1.77 microsecond cost per model load. All other changes result from build configuration (debug assertions, reduced inlining) appropriate for development phase.

Power Consumption:
Aggregate throughput increased 952ns (+57%), translating to ~1-3 nanojoules additional energy per execution cycle. Relative to inference energy (10-100 millijoules per token): <0.00003% impact. No meaningful effect on battery life or datacenter power consumption.

Most-Impacted Functions:
All are non-critical utilities: vector accessors (begin/end/back), allocator functions (_S_max_size), regex constructors, and model loading validation. None in inference hot path.

Assessment

Acceptable for development build. Target version exhibits debug build characteristics (assertions, reduced optimization) appropriate for active development indicated by commit messages. Release builds with -O3 -DNDEBUG will eliminate STL overhead. The one intentional enhancement (model loading diagnostics) provides valuable debugging capability at negligible cost.

Recommendation: No action required. Focus optimization efforts on actual inference kernels (matrix operations, attention, quantization) which dominate performance by 6+ orders of magnitude.

See the complete breakdown in Version Insights
Have questions? Tag @loci-dev to ask about this PR.