UPSTREAM PR #19165: cuda : fix nkvo

[loci-dev]

Mirrored from ggml-org/llama.cpp#19165

fix #19158
cont #19105

[loci-review]

Performance Review Report: llama.cpp Build Comparison

Impact Classification: Minor Impact

Analysis Scope: 5 functions analyzed, all non-critical STL template instantiations with changes between 4-189 nanoseconds per operation.

Summary

Performance changes between versions stem from a single commit ("cuda: fix nkvo") that corrects CUDA Flash Attention kernel validation, plus addition of GroVeMoE architecture support. All analyzed functions are C++ standard library components used in model loading, graph construction, and memory cleanup—not in inference hot paths.

Key Findings:

1. _M_destroy (std::future cleanup): Response time increased 189 nanoseconds (+60%), but throughput improved 179% enabling 3x concurrent operations during model loading. Trade-off favors parallel tensor validation.

2. unique_ptr operator= (graph builder): Response time improved 75 nanoseconds (-9%) for GroVeMoE architecture addition. Executes once per graph construction, negligible impact.

3. _M_insert (sampling maps): Response time improved 15 nanoseconds (-3%). Throughput decreased 16%, warranting monitoring in high-throughput continuous batching scenarios.

4. Hashtable deallocation functions: Changes of 4 nanoseconds (±2-4%) during context cleanup. No practical performance impact.

Power Consumption: Estimated <0.2% increase in CPU-side energy consumption, offset by 1-3% GPU energy savings from CUDA memory efficiency improvements.

GPU Impact: CUDA fix enables K/V tensor view optimization, providing 5-15% inference speedup and 10-20% memory reduction for attention-heavy workloads—the primary performance benefit of this version.

Justification: All changes are architecturally sound. Absolute timing differences (4-189ns) are negligible compared to inference operations (microseconds to milliseconds). The CUDA fix provides meaningful production benefits while CPU-side changes show acceptable latency-throughput trade-offs.

See the complete breakdown in Version Insights
Have questions? Tag @loci-dev to ask about this PR.