UPSTREAM PR #17044: Add MoE dynamic routing with expert caching

[DajanaV]

Mirrored from ggml-org/llama.cpp#17044

[loci-review]

Access the complete analysis in the LOCI Dashboard

Performance Analysis Summary: MoE Dynamic Routing Implementation

Overview

Pull Request #101 introduces Mixture-of-Experts (MoE) dynamic routing with expert caching, adding 2,627 lines across 23 files. The implementation includes new MoE runtime capabilities, CUDA acceleration, and expert cache management while maintaining backward compatibility through compile-time flags.

Key Findings

Performance Improvements

• Highest Impact Function: _M_const_cast in build.bin.libllama.so shows significant optimization:
  • Response Time: 67% improvement (262 ns → 85 ns)
  • Throughput: 74% improvement (241 ns → 64 ns)
  • Bottleneck Time: 84% improvement (203 ns → 32 ns)

Core Function Impact Assessment

The performance changes do not directly affect critical inference functions (llama_decode, llama_encode, llama_tokenize). The optimized _M_const_cast function operates in the model loading phase, specifically in tensor weight iterator operations during GGUF file processing. No impact on tokens per second performance is expected since inference-critical paths remain unchanged.

Power Consumption Analysis

Minimal power overhead across binaries:

• Primary Impact: build.bin.libllama.so shows 0.021% increase (280,780 nJ → 280,839 nJ)
• Secondary Binaries: 0.000-0.003% increases in affected components
• Overall system power impact remains negligible despite performance optimizations

Flame Graph and CFG Analysis

• Execution Structure: Shallow 3-level call hierarchy with 84 ns total runtime
• Optimization Source: Block consolidation eliminated one basic block, reducing instruction count and branch overhead
• Assembly Improvements: Removed unconditional branch instruction, improving pipeline efficiency and instruction cache utilization

Code Review Insights

The MoE implementation demonstrates solid engineering practices with proper feature isolation behind LLAMA_MOE_ENABLE flags. The expert caching system introduces dynamic loading capabilities for large MoE models while maintaining performance through CUDA acceleration and LRU-based memory management.

Actionable Recommendations

1. Regression Testing: Verify MoE functionality doesn't impact standard model performance when disabled
2. Memory Validation: Test expert cache behavior under various memory pressure scenarios
3. CUDA Compatibility: Ensure MoE CUDA kernels function correctly across different GPU architectures

The implementation successfully adds MoE capabilities while delivering indirect performance improvements to core tensor operations through compiler optimizations and improved memory layout.