kv-cache : SWA checkpoints store only non-masked cells - #23981
Merged
Conversation
zbrad
pushed a commit
to zbrad/llama.cpp
that referenced
this pull request
Jul 3, 2026
adrianhoehne
pushed a commit
to adrianhoehne/llama.cpp
that referenced
this pull request
Jul 5, 2026
hardcoreerik
added a commit
to hardcoreerik/TheOrc
that referenced
this pull request
Jul 6, 2026
…a-4 in the admission gate Conclusive finding after direct A/B validation at 100+-question scale: NoKvSlot on Context Fabric's Reviewer/heavy-prompt roles is specific to Gemma-4-class models, not a bug in our own code. qwen2.5-coder-7b and Meta-Llama-3.1-8B both ran full B0-B4 pipelines with zero NoKvSlot occurrences at 100-question scale; Gemma-4-12B hit it on the very first conversation of a completely fresh, empty KV pool, with SwaFull true and false producing byte-for-byte identical failures (disproving the SWA-cache-sizing theory) and force-recycle -on-NoKvSlot making no difference either (disproving cumulative cross-conversation exhaustion). llama.cpp's own native log confirms full-size KV caches were actually allocated throughout. Root cause: Gemma-4's "shared KV cache" architecture (some layers reuse another layer's K/V tensors instead of computing their own) breaks assumptions in llama.cpp's generic cache-management code -- documented by two independent upstream issues (ggml-org/llama.cpp#21468, #23720). The fix (ggml-org/llama.cpp#23981) merged 2026-06-02, after the llama.cpp commit pinned by LLamaSharp 0.27.0 (the latest release) and even after the commit pinned by LLamaSharp's own unreleased development branch -- not fixable on our end until a future LLamaSharp release bumps past it. Changes: - docs/CONTEXT_FABRIC_TEST_HARNESS.md: full investigation writeup (§7a), including the two disproven fixes, the upstream root cause with citations, and a separate (unrelated, not-yet-fixed) boundary-stitch schema-mismatch bug found on Meta-Llama-3.1-8B. - ModelAdmissionGate.cs: downgrade Gemma-4-class models (by name pattern, not just the already-rejected e4b variant) from Admitted to Provisional for Context Fabric workloads, with a caveat citing the known issue -- so the in-app admission reasoning reflects this rather than claiming "strong family" status for a family with a live crash risk. - LLamaSharpRuntime.cs: opt-in native log sink (reuses THEORC_KVCACHE_DIAGNOSTICS) that surfaces llama.cpp's own log lines during a diagnostic run -- this is what let us confirm full-size KV caches were genuinely allocated, ruling out the sizing theory conclusively. - Tools/ContextFabricBench/Program.cs: margin-adjusted the boundary-stitch diagnostic's token budget (matches the existing expanded-reader headroom fix) -- a reasonable improvement on its own, though confirmed NOT the fix for the schema-mismatch bug found during this investigation (completionTokens was 276, nowhere near the token budget). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
hardcoreerik
added a commit
to hardcoreerik/TheOrc
that referenced
this pull request
Jul 8, 2026
… Phase 1 (#40) * Fix NoKvSlot crashes: disable wasted SWA cache, margin the token-budget gates, retry on transient KV pressure Deep investigation (three parallel research agents, cross-validated with Codex CLI and Grok CLI) root-caused the CF-7 NoKvSlot crashes to a fixed, too-small native KV-cache pool rather than a leak, fragmentation, or stuck reference count: - ModelParams.SwaFull was never set and silently inherited llama.cpp's native default of true, forcing this Gemma-3-class model's 40 sliding-window-attention layers to reserve full-context KV cache instead of window-sized cache -- ~6x more cache pressure than the architecture needs, for no accuracy benefit. - ContextFabricFeasibilityRunner's two token-budget gates (InvokeAsync, BuildEvidencePack) used ContextManager.EstimateTokens (a crude chars/4 heuristic) with zero safety margin, unlike the sibling EvidencePackBuilder class, which already applies a 1.15x margin for exactly this reason. That let real, token-dense evidence prompts silently exceed the native KV pool. - InferUntilReadyAsync treated NoKvSlot as unconditionally fatal, contrary to LLamaSharp's own documented contract that it is retryable (BatchedExecutor.Infer requeues the batch internally rather than consuming it). Fixes: - Set ModelParams.SwaFull = false in LLamaSharpRuntime.LoadModelAsync. - Apply the same 1.15x safety margin to both token-budget gates in ContextFabricFeasibilityRunner.cs. - Give InferUntilReadyAsync a bounded retry-with-backoff on NoKvSlot specifically, before giving up. Ruled out via prior empirical runs (docs/CONTEXT_FABRIC_TEST_HARNESS.md §7): conversation-count recycle threshold value, and stuck/leaked ActiveCount reference counting. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Address Grok adversarial review of the NoKvSlot fix - FabricBoundaryStitcher.cs shared the identical unmargined token-budget-gate pattern Grok flagged as only partially fixed -- apply the same 1.15x margin there too. (ContextFabricBaselineRunner.cs was also flagged but does not actually gate/throw on token budget at all -- nothing to margin there without inventing a new check.) - Fix an off-by-one in the NoKvSlot retry give-up message (reported one more retry than actually attempted). - Add an env-gated (THEORC_KVCACHE_DIAGNOSTICS=1) diagnostic line per NoKvSlot retry, so a genuine single-conversation overflow shows a visible retry trail during its ~1.8s bounded backoff instead of just a delayed throw. - Clarify in-code that SwaFull=false only affects KV-cache allocation size, not the attention window itself (which is architectural and unaffected either way), and is a no-op for architectures without SWA layers -- still pending an empirical CF-7 gate re-run to confirm no output-quality regression on the live model. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Fix Run-CF7GateExpanded.ps1 crashing when exactly one GGUF model is present Set-StrictMode -Version Latest makes .Count throw PropertyNotFoundException on a single (non-array) FileInfo result from Get-ChildItem, instead of returning 1. Hit for real validating the NoKvSlot fix on HARDCORELAPTOPMSI, which had exactly one qualifying model after copying gemma-4-12B-it-qat-q4_0.gguf over. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Force role-executor recycle on first NoKvSlot instead of waiting for the conversation-count threshold A 100-question CF-7 gate run on gemma-4-12B (the actual target model for the SwaFull fix) showed NoKvSlot recurring on ~all subsequent conversations (85/88 in B3, 8/9 in B0) once it first occurred on a role's executor -- not a slow climb toward SequenceRecycleThreshold, but near-universal failure right after the first hit. That pattern is evidence the pool stays degraded after a NoKvSlot rather than a single oversized prompt being the whole story, and that the existing threshold-based recycle (every 24 conversations) recycles far too infrequently to recover from it. AdapterManager.RoleEntry now has a ForceRecycle flag, set by the new MarkForRecycle method as soon as IRoleRuntime observes any NoKvSlot (whether the bounded retry eventually recovers or gives up) -- the next GetOrCreateConversationAsync call for that role tears down and rebuilds its executor (a genuinely empty KV pool) as soon as the role goes idle, instead of continuing to serve from a pool already proven degraded. RuntimeOrchestrator.MarkRoleDegraded forwards to AdapterManager.MarkForRecycle since IRoleRuntime only holds a RuntimeOrchestrator reference. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Revert SwaFull=false: shrinks SWA cache below what Context Fabric's prompts need A 100-question CF-7 gate run (gemma-4-12B, the Gemma-3-class SWA architecture this setting targeted) proved the SwaFull=false change from an earlier commit was itself the root cause of the NoKvSlot failures on every Reviewer/Answer-stage call -- not the cumulative cross- conversation exhaustion originally suspected. With SwaFull=false, llama.cpp sizes the SWA-layer cache as min(fullContext, n_swa + UBatchSize). ModelParams.UBatchSize defaults to 512, so that's min(8192, 1024+512) = ~1536 cells, not the full 8192. Context Fabric's Reviewer/Answer-stage prompts routinely run 6,000-6,500 tokens (they re-include the evidence pack plus the Researcher's draft for review), well over that undersized window -- so every one of those calls failed NoKvSlot outright, confirmed even on a completely fresh, empty pool immediately after a forced executor recycle (the previous commit's fix). Raising UBatchSize enough to cover these prompts would erase most of the intended memory saving anyway, since Context Fabric's evidence packs are deliberately sized to use most of the available context budget -- SwaFull=false only pays off for workloads with prompts well under the context limit, which this one is not. The force-recycle-on-NoKvSlot fix (previous commit) stays: it's correct, harmless defense-in-depth for genuine cumulative-pressure scenarios, it just wasn't the fix for this specific failure mode. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Root-cause and document the Gemma-4 NoKvSlot investigation; flag Gemma-4 in the admission gate Conclusive finding after direct A/B validation at 100+-question scale: NoKvSlot on Context Fabric's Reviewer/heavy-prompt roles is specific to Gemma-4-class models, not a bug in our own code. qwen2.5-coder-7b and Meta-Llama-3.1-8B both ran full B0-B4 pipelines with zero NoKvSlot occurrences at 100-question scale; Gemma-4-12B hit it on the very first conversation of a completely fresh, empty KV pool, with SwaFull true and false producing byte-for-byte identical failures (disproving the SWA-cache-sizing theory) and force-recycle -on-NoKvSlot making no difference either (disproving cumulative cross-conversation exhaustion). llama.cpp's own native log confirms full-size KV caches were actually allocated throughout. Root cause: Gemma-4's "shared KV cache" architecture (some layers reuse another layer's K/V tensors instead of computing their own) breaks assumptions in llama.cpp's generic cache-management code -- documented by two independent upstream issues (ggml-org/llama.cpp#21468, #23720). The fix (ggml-org/llama.cpp#23981) merged 2026-06-02, after the llama.cpp commit pinned by LLamaSharp 0.27.0 (the latest release) and even after the commit pinned by LLamaSharp's own unreleased development branch -- not fixable on our end until a future LLamaSharp release bumps past it. Changes: - docs/CONTEXT_FABRIC_TEST_HARNESS.md: full investigation writeup (§7a), including the two disproven fixes, the upstream root cause with citations, and a separate (unrelated, not-yet-fixed) boundary-stitch schema-mismatch bug found on Meta-Llama-3.1-8B. - ModelAdmissionGate.cs: downgrade Gemma-4-class models (by name pattern, not just the already-rejected e4b variant) from Admitted to Provisional for Context Fabric workloads, with a caveat citing the known issue -- so the in-app admission reasoning reflects this rather than claiming "strong family" status for a family with a live crash risk. - LLamaSharpRuntime.cs: opt-in native log sink (reuses THEORC_KVCACHE_DIAGNOSTICS) that surfaces llama.cpp's own log lines during a diagnostic run -- this is what let us confirm full-size KV caches were genuinely allocated, ruling out the sizing theory conclusively. - Tools/ContextFabricBench/Program.cs: margin-adjusted the boundary-stitch diagnostic's token budget (matches the existing expanded-reader headroom fix) -- a reasonable improvement on its own, though confirmed NOT the fix for the schema-mismatch bug found during this investigation (completionTokens was 276, nowhere near the token budget). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Tier 1 retrieval fix: anchor-phrase ranking + coverage-aware evidence-pack fill Failure analysis of the Meta-Llama-3.1-8B 100-question CF-7 run proved that 100% of the 69 failed B3 questions were retrieval misses -- in every failure at least one required segment was absent from the evidence pack the model received, and in 49/69 none were present; zero failures had the evidence present with a wrong answer. The reader stage was verified NOT at fault (correct cards exist with the answer verbatim). Root mechanism, measured: the unigram bag-of-words IDF ranker dissolves multi-word entity names and hyphenated identifiers ("Station Alpha" -> {station, alpha}: 63/105 cards tie on both tokens while only 3 contain the phrase; "BR-048" -> {br, 048}), leaving ties broken by segment-ID sort. Fixes (CF_RETRIEVAL_IMPROVEMENT_PLAN.md Tier 1 -- the full tiered plan including the deliberately-deferred embedding and agentic-retrieval tiers is in that new doc): - 1a: extract anchor phrases (hyphenated identifiers, multi-word proper-noun runs with question-opener trimming) and rank them LEXICOGRAPHICALLY above unigram score -- an anchor match cannot be outvoted by common-word noise, and anchor-less questions degrade exactly to the previous ordering. - 1b: coverage-aware greedy fill -- each pick is scored against the question's not-yet- covered anchors/terms first, so multi-entity questions cover every named entity before stacking near-duplicates; once covered, ordering falls back to base scores (preserving the old fill-the-budget behavior for GlobalSynthesis). - 1c: BuildExhaustiveAnswer matches scoped hyphenated identifiers verbatim in claim text instead of the rarest-unigram heuristic, which admitted any claim sharing the numeric fragment ("grade-20" vs "20 crates"). Baselines (B0/B1/B2) deliberately untouched to preserve comparability with all prior runs logged in CF_TEST_RESULTS.md. Six new unit tests cover the measured production failure modes, including two constructed so the OLD tie-break provably picks the wrong card. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address CodeRabbit findings on the NoKvSlot retry/degrade path - Reset noKvSlotRetries after each successful decode: the cap now bounds CONSECUTIVE NoKvSlot failures rather than the lifetime total across a long drain, so separately- recovered episodes can no longer sum to a spurious give-up. - MarkForRecycle/MarkRoleDegraded no longer accept a CancellationToken: the observation "this executor produced a NoKvSlot" remains true whether or not the observing request is being cancelled, and honoring a cancelled token let the degraded executor be silently reused by the next request. Enforced at the API level rather than trusting call sites to pass CancellationToken.None. Request cancellation still propagates promptly via the retry delay that follows the mark. - The third finding (FabricBoundaryStitcher margin) was already addressed in 4542e1d; the review raced that push. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Log Tier 1 validation results: 31->45 pass rate, B3 beats B2 for the first time 100-question Meta-Llama-3.1-8B run on NEWCOREPC at 5fd6ad3, identical conditions to the baseline. Retrieval misses cut from 69 (100% of failures) to 45; 10 genuine model failures now measurable for the first time; citation precision held at 99.1%; zero NoKvSlot. Dominant remaining pure-miss bucket identified: Paraphrased questions with inverted entity word order, which contiguous anchors cannot match -- candidates are an unordered-proximity anchor variant (Tier 1.5) or Tier 2 embeddings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Tier 1.5 retrieval fix: unordered proximity-pair anchors for inverted entity word order The Tier 1 validation run's dominant residual pure-miss bucket (13 of 21) was Paraphrased questions that invert an entity's word order -- "the Meridian relay point" vs the corpus's "Relay Meridian". Contiguous anchors fail twice there: matching can't bridge the inversion, and extraction yields no anchor at all because only "Meridian" is capitalized (no 2+ word proper-noun run exists in the question). Fix: every mid-sentence capitalized word becomes an entity head paired with its nearest non-stopword neighbor (within two positions each side); a card matches when both words occur within a 2-token window in ANY order. Pairs feed the same lexicographic anchor key as verbatim anchors at half weight (0.5/df) -- a contiguous phrase match still wins wherever both exist -- and participate in the Tier 1b coverage bookkeeping. Questions yielding no pairs behave exactly as Tier 1. Three new tests, including one where the distractor shares MORE question unigrams than the correct card so plain unigram IDF provably prefers it -- only the proximity pair rescues the target. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Log Tier 1.5 validation: 45->56 pass rate, Paraphrased pure-misses 13->3 100-question Meta-Llama run on NEWCOREPC at 6d28071, identical conditions to prior runs. Cumulative across three deterministic lexical fixes: 31 -> 45 -> 56 pass rate, pure retrieval misses 49 -> 21 -> 10, citation precision 99.1% throughout, zero NoKvSlot. The partial-miss bucket (24 multi-segment questions) is now the largest failure class. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Tier 2 retrieval fix: truncate dense cards instead of rejecting them FabricEvidenceProcessor.NormalizeAndValidate previously rejected evidence cards when the model over-generated on expanded-corpus dense segments: - summary > 2000 chars: outright rejection (11 failures observed) - claims > 64 items: outright rejection - duplicate canonical claimId: error + rejection Each rejection silently removed the segment card from the evidence pack, blocking every question that needed that segment. Now: - Long summaries are truncated to MaxSummaryChars with a trailing ellipsis - Excess claims are silently truncated to the first MaxClaims entries - Duplicate canonical claimIds are silently deduped (first occurrence kept) Three new unit tests cover the new behaviors; the existing rejection test for canonical-id collisions is updated to expect dedup-acceptance. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Address CodeRabbit nitpicks: consolidate margin constant + fix reporting drift Two outstanding CodeRabbit findings from PR #40 review rounds 1-2: 1. Consolidate duplicate 1.15 safety margin FabricContextBudget.TokenSafetyMargin is now the single source of truth, referenced from ContextFabricFeasibilityRunner, FabricBoundaryStitcher, and EvidencePackBuilder. Rationale lives in FabricContextBudget (in ContextFabricContracts.cs, which compiles into NativeRuntime unlike EvidencePackBuilder.cs, which is why the comment previously explained the duplication instead of fixing it). 2. Fix reporting drift from margin-adjusted estimate reuse InvokeAsync in ContextFabricFeasibilityRunner and FabricBoundaryStitcher previously used the margin-adjusted promptTokens as both the budget gate threshold AND the FabricCallMetrics fallback -- inflating reported prompt counts by 15% whenever the native runtime did not report actual usage. Now: rawPromptTokens for reporting, gatePromptTokens (x1.15) for the gate. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * @ Phase 1: Model Benchmark window — storage schema + read-only CF-7 results display ModelBenchmarkRecord + ModelBenchmarkStore: new storage schema for per-model Context Fabric benchmark results (model-benchmark-1.0). The store scans .orc/adversarial/ on open, joining each cf7_gate_*.json to the nearest-in-time cf0_*.json B3 sub-report to recover the model filename from environment.lanes. De-duped by (modelFilename + tier), so the latest run per (model, tier) pair is kept. ModelBenchmarkWindow.axaml/.cs: new Avalonia window accessible via Models → Model Benchmark…. On open it scans .orc/adversarial/ and renders one card per model: a tier + date header, GO/NO-GO verdict badge, and metric chips for question pass rate, citation precision, segment coverage, and boundary stitch rate. Depot GGUF files with no benchmark record render as a dimmed "not benchmarked" row. "Run New Benchmark…" button present but disabled (Phase 2). Phase 1 is read-only display of existing CF-7 run artifacts. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> @ * CF bench: print per-candidate depot verdicts + disabled GGUFs on preflight rejection Today's NEWCOREPC exit-66 (SmolLM2 selected for CF) turned out to be depot state, not a resolver bug: Meta-Llama-3.1-8B was .gguf.disabled at run time, leaving only Rejected candidates. The rejection message now lists every scanned base model with its ContextFabricReader verdict and any skipped .gguf.disabled files, so this failure mode is self-explanatory next time. Also make the Model Benchmark window depot listing recursive so GGUFs in subdirectories (e.g. content-addressed 2f/) appear as unbenchmarked rows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Log HARDCORELAPTOPMSI Tier 2 validation: qwen 26->30/100, citation 77.5->88.2% Second model family confirms the Tier 2 truncation fix direction; B2 RAG baseline (49/100) still beats B3 on qwen-7b, unlike Meta-Llama on NEWCOREPC. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
zommiommy
pushed a commit
to zommiommy/llama.cpp
that referenced
this pull request
Aug 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
fix #23720
This change reduces the size of the SWA checkpoints and should make it possible to always restore them with unified KV cache.
Requirements