QVAC-11621 fix: KV cache memory validation to prevent OOM crashes

[simon-iribarren]

Summary

• Reads model GGUF header to extract architecture params and compute exact KV cache bytes/token
• Validates estimated memory usage before loading, throws structured error with suggested ctx_size if it would OOM
• Falls back to file-size heuristic brackets when GGUF metadata unavailable
• unsafeDisableMemoryValidation config flag to bypass the check

Note: This is the SDK-side implementation. A future addon-side implementation can provide even more precise calculations using the actual KV cache quantization config (cache_type_k, cache_type_v).

How it works

Exact KV cache computation from GGUF metadata:

• Parses GGUF header for block_count, head_count_kv, embedding_length, head_count
• Formula: 2 * n_layer * n_kv_heads * head_dim * dtype_size (f16 default)
• Logs [exact] or [heuristic] in validation output

Plugin-level validation (validateBeforeLoad hook):

• New optional hook on QvacPlugin interface, called before createModel()
• Compares model_size + kv_cache + overhead against 80% of available memory
• Throws ModelMemoryExceededError (code 52211) with suggestedCtxSize

Platform memory detection:

• bare-os.totalmem() with conservative fractions (70% desktop, 65% mobile)

API Changes

// New plugin hook
interface ValidateBeforeLoadParams {
  modelConfig: Record<string, unknown>;
  modelFileSize: number;
  availableMemory: number;
  kvBytesPerToken?: number;
}

// New error
catch (error) {
  if (error instanceof ModelMemoryExceededError) {
    console.log(`Try ctx_size: ${error.suggestedCtxSize}`);
  }
}

// Config flag
{ "unsafeDisableMemoryValidation": true }

Testing

• 41 unit tests: exact KV computation (Llama-1B, Gemma-4B, Llama-8B), heuristic brackets, false-positive regression against all mobile test configs on 4/6/8/12GB devices
• 5 e2e test definitions: memory exceeded, default ctx safe, moderate ctx safe, recover with suggested, load after rejection
• Manual desktop testing with extreme ctx_size values

---

Recreated from #1464 to move branch onto the fork (post-migration). No code changes vs. the original PR — only rebased onto latest main.

[jesusmb1995]

I think we need to consider KV quant from the start as well, not just F16, being 75% off seems too much. Specially since the push/hype towards TurboQuant (6x smaller than F16 with minimal quality loss on long contex) that is going on right now and our focus on embedded devices and mobile with more memory constraints.

[github-actions]

Tier-based Approval Status

**PR Tier:** TIER1

**Current Status:** ❌ PENDING

**Requirements:**
- 1 Team Member approval ❌ (0/1)
- 1 Team Lead OR Management approval ❌ (0/1)



---
*This comment is automatically updated when reviews change.*