UPSTREAM PR #16940: Add e2e tests for embedding raw flag

[DajanaV]

Mirrored from ggml-org/llama.cpp#16940

🧩 Summary

This PR adds a CI workflow for end-to-end embedding CLI tests (none exist today). It establishes a small, fast, reproducible baseline for validating embedding behavior (dimensions + determinism) using tiny GGUF models.

Discussion / design context: See the companion RFC in Discussions for the longer-term plan to add a native server endpoint: ggml-org/llama.cpp#16957

⚙️ What this PR includes

• A GitHub Actions job (embeddings.yml) that runs E2E embedding CLI tests with cached tiny models (e.g., TinyLlama).
• Checks output dimensions and deterministic behavior.
• Keeps runs lightweight and fast; an optional large-model stress test can be added later.

[loci-agentic-ai]

Access the complete analysis in the LOCI Dashboard

LLaMA.cpp Performance Analysis Summary

Critical Function Performance Status

Core Inference Functions

All critical inference functions show no measurable performance changes between versions:

• llama_decode: 49ms response time (unchanged) - Primary inference function
• llama_encode: 12ms response time (unchanged) - Encoder processing
• llama_tokenize: 833μs response time (unchanged) - Text tokenization
• llama_model_load_from_file: 332ms response time (unchanged) - Model loading
• llama_batch_init: 257ns response time (unchanged) - Batch initialization
• llama_memory_clear: 49ns response time (unchanged) - Memory management

Function Modification Status

Analysis confirms no source code modifications to any critical functions between the compared versions.

Key Performance Indicators Impact Assessment

1. Tokens Per Second

Impact: None

• No changes detected in core inference functions (llama_decode, llama_encode, llama_tokenize)
• Response times remain stable across all tokenization and inference pathways
• Based on the reference that 2ms slower llama_decode reduces tokens/second by 7%, the unchanged performance maintains baseline throughput

2. Power Consumption

Impact: Negligible

• build.bin.libllama.so: 0.0001% increase (306,896 nJ vs 306,896 nJ base)
• build.bin.libggml-base.so: No change (90,434 nJ)
• build.bin.libggml-cpu.so: No change (151,692 nJ)
• build.bin.libggml.so: No change (6,339 nJ)

3. Quantization Efficiency

Impact: None

• llama_model_quantize function shows no performance changes
• Quantization-related functions maintain baseline performance
• No modifications to quantization algorithms or data paths

4. Memory Usage

Impact: None

• Memory management functions (llama_memory_clear, KV cache operations) unchanged
• No modifications to memory allocation patterns
• Batch processing memory efficiency maintained

5. Batch Processing

Impact: None

• llama_batch_init performance unchanged (257ns)
• Batch allocation and processing functions maintain baseline metrics
• No changes to parallel processing efficiency

Root Cause Analysis

The minimal performance variations observed (0.096% in llm_graph_input_out_ids::can_reuse) stem from:

• Build Environment Factors: Compiler optimization differences or system-level changes
• Measurement Precision: Sub-nanosecond variations within measurement tolerance
• Binary Metadata: Different debug information or symbol placement

Action Items

Build Optimization

• Verify consistent compiler flags between builds (-O3, -march=native)
• Ensure identical build environment configuration
• Review linker optimization settings for binary layout consistency

Code Analysis

• The observed 0.06ns increase in graph parameter validation is within measurement noise
• No code-level optimizations required for core inference functions
• Focus performance efforts on higher-impact areas if needed

Monitoring

• Current performance baseline is stable across all critical functions
• No regression in core inference pathways
• Power consumption remains within expected bounds

Conclusion

The analysis reveals no significant performance impact on LLaMA.cpp's core functionality. All critical functions maintain baseline performance, with observed variations falling within measurement precision limits. The inference pipeline efficiency, memory management, and batch processing capabilities remain unchanged, ensuring stable tokens-per-second throughput and power consumption characteristics.