UPSTREAM PR #19164: spec : add ngram-mod

[loci-dev]

Mirrored from ggml-org/llama.cpp#19164

cont #18471

Add basic ngram hasher for speculative decoding:

• For each ngram, compute a hash using LCG
• For each computed hash, store the next token
• During speculation, iteratively compute the rolling hash of the last n tokens and pick the next token from the storage

Some characteristics:

• Lightweight (~20 MB)
• Constant memory and complexity
• Can generate variable draft lengths (i.e. m is not fixed)

Currently, a single hash pool is shared across all server slots, so different requests can benefit from each other.

Sample usage:

# notes:
# - small `n` are not recommended
# - MoEs require long drafts
# - dense models: can reduce `--draft-min` and `--draft-max`

llama-server ... --spec-type ngram-mod --spec-ngram-size-n 24 --draft-min 16 --draft-max 64

Applications:

• Iterating over a block of text
• Summarization

Example:

spec-mod-0.mov

TODO:

• Reset criteria?

[loci-review]

Performance Review Report: llama.cpp Speculative Decoding Enhancement

Executive Summary

Commit 7ef5b95 ("spec: add ngram-mod") introduces n-gram based speculative decoding with negligible performance impact on existing operations. Analysis of 11 function instances across llama-tts and llama-cvector-generator binaries reveals performance changes isolated to non-critical utility code, with core inference operations completely unchanged.

Impact Classification: Minor

Files Changed: 12 modified, 39 added, 3 deleted

Performance Changes:

• Core inference: 0 nanoseconds change (llama_decode, GEMM, attention, KV cache unchanged)
• Initialization overhead: +2,000-5,000 nanoseconds total (one-time startup cost)
• Template rendering: -183 nanoseconds per call (69.6% improvement in jinja function dispatch)
• HTTP operations: +179 nanoseconds per call, but +256% throughput improvement

Key Findings

No Critical Path Impact: All performance-critical components (matrix multiplication, attention mechanisms, quantization kernels, GPU backends) remain unmodified. The new n-gram speculative decoding is opt-in and does not execute in default inference paths.

Standard Library Regressions: Observed regressions occur exclusively in STL functions (vector iterators, allocators, tree accessors) with no source code changes. Performance variations stem from compiler optimization differences between builds, not algorithmic modifications:

• std::vector::begin(): +181 nanoseconds (initialization only)
• std::allocator::deallocate(): +33 nanoseconds (initialization only)
• std::chrono::__cast(): +190 nanoseconds (logging only)

Positive Changes: Template function lookup improved by 183 nanoseconds per call, providing 4-37 microseconds savings per template render. HTTP socket validation shows 256% throughput improvement, indicating better connection pooling.

Energy Impact: Unable to quantify due to tool error, but expected to be unmeasurable given unchanged compute operations and nanosecond-scale differences in non-critical paths.

Conclusion

The commit successfully adds valuable speculative decoding functionality without impacting inference performance. All overhead occurs in initialization code (microseconds at startup) or shows net improvements (template rendering, HTTP operations). No optimization required.

See the complete breakdown in Version Insights
Have questions? Tag @loci-dev to ask about this PR.

[loci-review]

No summary available at this time. Visit Loci Inspector to review detailed analysis.