NoKvSlot root-cause + Tier 1 retrieval fix: anchor-phrase evidence ranking - #40
Merged
Conversation
…et gates, retry on transient KV pressure Deep investigation (three parallel research agents, cross-validated with Codex CLI and Grok CLI) root-caused the CF-7 NoKvSlot crashes to a fixed, too-small native KV-cache pool rather than a leak, fragmentation, or stuck reference count: - ModelParams.SwaFull was never set and silently inherited llama.cpp's native default of true, forcing this Gemma-3-class model's 40 sliding-window-attention layers to reserve full-context KV cache instead of window-sized cache -- ~6x more cache pressure than the architecture needs, for no accuracy benefit. - ContextFabricFeasibilityRunner's two token-budget gates (InvokeAsync, BuildEvidencePack) used ContextManager.EstimateTokens (a crude chars/4 heuristic) with zero safety margin, unlike the sibling EvidencePackBuilder class, which already applies a 1.15x margin for exactly this reason. That let real, token-dense evidence prompts silently exceed the native KV pool. - InferUntilReadyAsync treated NoKvSlot as unconditionally fatal, contrary to LLamaSharp's own documented contract that it is retryable (BatchedExecutor.Infer requeues the batch internally rather than consuming it). Fixes: - Set ModelParams.SwaFull = false in LLamaSharpRuntime.LoadModelAsync. - Apply the same 1.15x safety margin to both token-budget gates in ContextFabricFeasibilityRunner.cs. - Give InferUntilReadyAsync a bounded retry-with-backoff on NoKvSlot specifically, before giving up. Ruled out via prior empirical runs (docs/CONTEXT_FABRIC_TEST_HARNESS.md §7): conversation-count recycle threshold value, and stuck/leaked ActiveCount reference counting. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThis PR adds role recycle signaling and retry handling for KV-slot exhaustion in native inference, documents the SWA KV-cache setting during model load, and applies token safety margins to ContextFabric budget checks and related tooling. ChangesRuntime inference and KV-cache behavior
Context Fabric token budget safety margin
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
- FabricBoundaryStitcher.cs shared the identical unmargined token-budget-gate pattern Grok flagged as only partially fixed -- apply the same 1.15x margin there too. (ContextFabricBaselineRunner.cs was also flagged but does not actually gate/throw on token budget at all -- nothing to margin there without inventing a new check.) - Fix an off-by-one in the NoKvSlot retry give-up message (reported one more retry than actually attempted). - Add an env-gated (THEORC_KVCACHE_DIAGNOSTICS=1) diagnostic line per NoKvSlot retry, so a genuine single-conversation overflow shows a visible retry trail during its ~1.8s bounded backoff instead of just a delayed throw. - Clarify in-code that SwaFull=false only affects KV-cache allocation size, not the attention window itself (which is architectural and unaffected either way), and is a no-op for architectures without SWA layers -- still pending an empirical CF-7 gate re-run to confirm no output-quality regression on the live model. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
OrchestratorIDE/Core/Runtime/LLamaSharpRuntime.cs (1)
222-228: 🚀 Performance & Scalability | 🔵 TrivialKeep
SwaFullconfigurable on the batched-executor path. LLamaSharp warnsfalsecan hurt performance whenSeqMax > 1, and this persistent per-role executor can hit that case once multiple conversations are active.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@OrchestratorIDE/Core/Runtime/LLamaSharpRuntime.cs` around lines 222 - 228, Keep SwaFull configurable in LLamaSharpRuntime’s executor settings instead of hard-coding it to false. Update the batched-executor setup in the LLamaSharpRuntime path so the value can be chosen based on SeqMax or caller/runtime configuration, preserving the current memory-saving behavior only where safe while allowing the per-role executor to opt into the native setting when multiple conversations are active.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@OrchestratorIDE/Core/Runtime/IRoleRuntime.cs`:
- Around line 318-337: Reset the transient retry counter in the drain loop so it
only counts consecutive NoKvSlot failures within the current recovery streak. In
the conversation drain logic inside the method that calls
conversation.Executor.Infer, keep noKvSlotRetries from accumulating across
successful decodes by clearing it whenever result is not DecodeResult.NoKvSlot,
while preserving the existing backoff and hard-fail behavior for repeated
failures.
In `@OrchestratorIDE/Services/ContextFabric/ContextFabricFeasibilityRunner.cs`:
- Around line 867-876: The token-budget gate in FabricBoundaryStitcher still
uses raw ContextManager.EstimateTokens for the prompt-size check, which can let
oversized prompts slip through and hit NoKvSlot. Update the boundary check in
the relevant stitching path to apply the same EvidenceTokenSafetyMargin used by
ContextFabricFeasibilityRunner, and keep the failure condition comparing the
margin-adjusted promptTokens plus ReaderMaxTokens against the context limit so
both budget gates behave consistently.
- Around line 867-876: The margin-adjusted prompt token estimate in
ContextFabricFeasibilityRunner is being reused beyond the budget gate, which
causes reporting drift. Keep the EvidenceTokenSafetyMargin-applied value only
for the context limit check in the prompt budget path, and preserve the raw
ContextManager.EstimateTokens total for fallback metrics. Update the logic
around the promptTokens calculation and the
FitsContext/MaximumPromptTokens/SourceToWorkingContextRatio reporting so they
use the unadjusted estimate when usage is not reported.
---
Nitpick comments:
In `@OrchestratorIDE/Core/Runtime/LLamaSharpRuntime.cs`:
- Around line 222-228: Keep SwaFull configurable in LLamaSharpRuntime’s executor
settings instead of hard-coding it to false. Update the batched-executor setup
in the LLamaSharpRuntime path so the value can be chosen based on SeqMax or
caller/runtime configuration, preserving the current memory-saving behavior only
where safe while allowing the per-role executor to opt into the native setting
when multiple conversations are active.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: ba8130fe-c0a9-4983-8db5-ac202bdb265d
📒 Files selected for processing (3)
OrchestratorIDE/Core/Runtime/IRoleRuntime.csOrchestratorIDE/Core/Runtime/LLamaSharpRuntime.csOrchestratorIDE/Services/ContextFabric/ContextFabricFeasibilityRunner.cs
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
OrchestratorIDE/Services/ContextFabric/FabricBoundaryStitcher.cs (1)
19-19: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider consolidating the duplicated margin constant.
The same
1.15safety margin is now independently defined asEvidenceTokenSafetyMargininEvidencePackBuilder.csandContextFabricFeasibilityRunner.cs, and here asTokenSafetyMargin. This PR itself exists because one of these gates was missed when the margin was first introduced — extracting a single shared constant (e.g., inContextManageror a small shared options/constants class) would prevent future drift across the three sites.♻️ Example consolidation
- // Same under-counting risk and fix as ContextFabricFeasibilityRunner.EvidenceTokenSafetyMargin - // (NoKvSlot investigation, docs/CONTEXT_FABRIC_TEST_HARNESS.md §7) -- this gate shares the - // identical unmargined ContextManager.EstimateTokens pattern and sits on the same B-system - // gate-run crash surface. - private const double TokenSafetyMargin = 1.15; + // Shared with EvidencePackBuilder/ContextFabricFeasibilityRunner to avoid margin drift. + private const double TokenSafetyMargin = ContextFabricTokenBudget.SafetyMargin;🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@OrchestratorIDE/Services/ContextFabric/FabricBoundaryStitcher.cs` at line 19, The safety margin value is duplicated across FabricBoundaryStitcher.TokenSafetyMargin, EvidencePackBuilder.EvidenceTokenSafetyMargin, and ContextFabricFeasibilityRunner, which risks future drift. Extract the shared 1.15 margin into one common constant or shared options class (for example in ContextManager or a small constants helper) and update all three call sites to use that single source of truth.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@OrchestratorIDE/Services/ContextFabric/FabricBoundaryStitcher.cs`:
- Line 19: The safety margin value is duplicated across
FabricBoundaryStitcher.TokenSafetyMargin,
EvidencePackBuilder.EvidenceTokenSafetyMargin, and
ContextFabricFeasibilityRunner, which risks future drift. Extract the shared
1.15 margin into one common constant or shared options class (for example in
ContextManager or a small constants helper) and update all three call sites to
use that single source of truth.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 611cf369-5d38-426f-9022-23728250c495
📒 Files selected for processing (3)
OrchestratorIDE/Core/Runtime/IRoleRuntime.csOrchestratorIDE/Core/Runtime/LLamaSharpRuntime.csOrchestratorIDE/Services/ContextFabric/FabricBoundaryStitcher.cs
🚧 Files skipped from review as they are similar to previous changes (2)
- OrchestratorIDE/Core/Runtime/LLamaSharpRuntime.cs
- OrchestratorIDE/Core/Runtime/IRoleRuntime.cs
…resent Set-StrictMode -Version Latest makes .Count throw PropertyNotFoundException on a single (non-array) FileInfo result from Get-ChildItem, instead of returning 1. Hit for real validating the NoKvSlot fix on HARDCORELAPTOPMSI, which had exactly one qualifying model after copying gemma-4-12B-it-qat-q4_0.gguf over. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…the conversation-count threshold A 100-question CF-7 gate run on gemma-4-12B (the actual target model for the SwaFull fix) showed NoKvSlot recurring on ~all subsequent conversations (85/88 in B3, 8/9 in B0) once it first occurred on a role's executor -- not a slow climb toward SequenceRecycleThreshold, but near-universal failure right after the first hit. That pattern is evidence the pool stays degraded after a NoKvSlot rather than a single oversized prompt being the whole story, and that the existing threshold-based recycle (every 24 conversations) recycles far too infrequently to recover from it. AdapterManager.RoleEntry now has a ForceRecycle flag, set by the new MarkForRecycle method as soon as IRoleRuntime observes any NoKvSlot (whether the bounded retry eventually recovers or gives up) -- the next GetOrCreateConversationAsync call for that role tears down and rebuilds its executor (a genuinely empty KV pool) as soon as the role goes idle, instead of continuing to serve from a pool already proven degraded. RuntimeOrchestrator.MarkRoleDegraded forwards to AdapterManager.MarkForRecycle since IRoleRuntime only holds a RuntimeOrchestrator reference. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@OrchestratorIDE/Core/Runtime/AdapterManager.cs`:
- Around line 281-287: The recycle/degrade mark in AdapterManager.MarkForRecycle
can be skipped if the caller’s cancellation token is already canceled while
waiting on _gate, which leaves ForceRecycle unset and allows a degraded executor
to be reused. Make MarkForRecycle immune to request cancellation once NoKvSlot
has been observed by not using the request token for the gate wait, and update
the orchestrator forwarding path that calls MarkForRecycle to stop passing
through the request CancellationToken. Use the existing MarkForRecycle and
IRoleRuntime call site as the places to fix.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 489aef80-292e-4277-829d-6c6a5258b39a
📒 Files selected for processing (3)
OrchestratorIDE/Core/Runtime/AdapterManager.csOrchestratorIDE/Core/Runtime/IRoleRuntime.csOrchestratorIDE/Core/Runtime/RuntimeOrchestrator.cs
🚧 Files skipped from review as they are similar to previous changes (1)
- OrchestratorIDE/Core/Runtime/IRoleRuntime.cs
…rompts need A 100-question CF-7 gate run (gemma-4-12B, the Gemma-3-class SWA architecture this setting targeted) proved the SwaFull=false change from an earlier commit was itself the root cause of the NoKvSlot failures on every Reviewer/Answer-stage call -- not the cumulative cross- conversation exhaustion originally suspected. With SwaFull=false, llama.cpp sizes the SWA-layer cache as min(fullContext, n_swa + UBatchSize). ModelParams.UBatchSize defaults to 512, so that's min(8192, 1024+512) = ~1536 cells, not the full 8192. Context Fabric's Reviewer/Answer-stage prompts routinely run 6,000-6,500 tokens (they re-include the evidence pack plus the Researcher's draft for review), well over that undersized window -- so every one of those calls failed NoKvSlot outright, confirmed even on a completely fresh, empty pool immediately after a forced executor recycle (the previous commit's fix). Raising UBatchSize enough to cover these prompts would erase most of the intended memory saving anyway, since Context Fabric's evidence packs are deliberately sized to use most of the available context budget -- SwaFull=false only pays off for workloads with prompts well under the context limit, which this one is not. The force-recycle-on-NoKvSlot fix (previous commit) stays: it's correct, harmless defense-in-depth for genuine cumulative-pressure scenarios, it just wasn't the fix for this specific failure mode. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…a-4 in the admission gate Conclusive finding after direct A/B validation at 100+-question scale: NoKvSlot on Context Fabric's Reviewer/heavy-prompt roles is specific to Gemma-4-class models, not a bug in our own code. qwen2.5-coder-7b and Meta-Llama-3.1-8B both ran full B0-B4 pipelines with zero NoKvSlot occurrences at 100-question scale; Gemma-4-12B hit it on the very first conversation of a completely fresh, empty KV pool, with SwaFull true and false producing byte-for-byte identical failures (disproving the SWA-cache-sizing theory) and force-recycle -on-NoKvSlot making no difference either (disproving cumulative cross-conversation exhaustion). llama.cpp's own native log confirms full-size KV caches were actually allocated throughout. Root cause: Gemma-4's "shared KV cache" architecture (some layers reuse another layer's K/V tensors instead of computing their own) breaks assumptions in llama.cpp's generic cache-management code -- documented by two independent upstream issues (ggml-org/llama.cpp#21468, #23720). The fix (ggml-org/llama.cpp#23981) merged 2026-06-02, after the llama.cpp commit pinned by LLamaSharp 0.27.0 (the latest release) and even after the commit pinned by LLamaSharp's own unreleased development branch -- not fixable on our end until a future LLamaSharp release bumps past it. Changes: - docs/CONTEXT_FABRIC_TEST_HARNESS.md: full investigation writeup (§7a), including the two disproven fixes, the upstream root cause with citations, and a separate (unrelated, not-yet-fixed) boundary-stitch schema-mismatch bug found on Meta-Llama-3.1-8B. - ModelAdmissionGate.cs: downgrade Gemma-4-class models (by name pattern, not just the already-rejected e4b variant) from Admitted to Provisional for Context Fabric workloads, with a caveat citing the known issue -- so the in-app admission reasoning reflects this rather than claiming "strong family" status for a family with a live crash risk. - LLamaSharpRuntime.cs: opt-in native log sink (reuses THEORC_KVCACHE_DIAGNOSTICS) that surfaces llama.cpp's own log lines during a diagnostic run -- this is what let us confirm full-size KV caches were genuinely allocated, ruling out the sizing theory conclusively. - Tools/ContextFabricBench/Program.cs: margin-adjusted the boundary-stitch diagnostic's token budget (matches the existing expanded-reader headroom fix) -- a reasonable improvement on its own, though confirmed NOT the fix for the schema-mismatch bug found during this investigation (completionTokens was 276, nowhere near the token budget). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…-pack fill
Failure analysis of the Meta-Llama-3.1-8B 100-question CF-7 run proved that 100% of the 69
failed B3 questions were retrieval misses -- in every failure at least one required segment
was absent from the evidence pack the model received, and in 49/69 none were present; zero
failures had the evidence present with a wrong answer. The reader stage was verified NOT
at fault (correct cards exist with the answer verbatim). Root mechanism, measured: the
unigram bag-of-words IDF ranker dissolves multi-word entity names and hyphenated
identifiers ("Station Alpha" -> {station, alpha}: 63/105 cards tie on both tokens while
only 3 contain the phrase; "BR-048" -> {br, 048}), leaving ties broken by segment-ID sort.
Fixes (CF_RETRIEVAL_IMPROVEMENT_PLAN.md Tier 1 -- the full tiered plan including the
deliberately-deferred embedding and agentic-retrieval tiers is in that new doc):
- 1a: extract anchor phrases (hyphenated identifiers, multi-word proper-noun runs with
question-opener trimming) and rank them LEXICOGRAPHICALLY above unigram score -- an
anchor match cannot be outvoted by common-word noise, and anchor-less questions degrade
exactly to the previous ordering.
- 1b: coverage-aware greedy fill -- each pick is scored against the question's not-yet-
covered anchors/terms first, so multi-entity questions cover every named entity before
stacking near-duplicates; once covered, ordering falls back to base scores (preserving
the old fill-the-budget behavior for GlobalSynthesis).
- 1c: BuildExhaustiveAnswer matches scoped hyphenated identifiers verbatim in claim text
instead of the rarest-unigram heuristic, which admitted any claim sharing the numeric
fragment ("grade-20" vs "20 crates").
Baselines (B0/B1/B2) deliberately untouched to preserve comparability with all prior runs
logged in CF_TEST_RESULTS.md.
Six new unit tests cover the measured production failure modes, including two constructed
so the OLD tie-break provably picks the wrong card.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
hardcoreerik
changed the title
- Reset noKvSlotRetries after each successful decode: the cap now bounds CONSECUTIVE NoKvSlot failures rather than the lifetime total across a long drain, so separately- recovered episodes can no longer sum to a spurious give-up. - MarkForRecycle/MarkRoleDegraded no longer accept a CancellationToken: the observation "this executor produced a NoKvSlot" remains true whether or not the observing request is being cancelled, and honoring a cancelled token let the degraded executor be silently reused by the next request. Enforced at the API level rather than trusting call sites to pass CancellationToken.None. Request cancellation still propagates promptly via the retry delay that follows the mark. - The third finding (FabricBoundaryStitcher margin) was already addressed in 4542e1d; the review raced that push. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…first time 100-question Meta-Llama-3.1-8B run on NEWCOREPC at 5fd6ad3, identical conditions to the baseline. Retrieval misses cut from 69 (100% of failures) to 45; 10 genuine model failures now measurable for the first time; citation precision held at 99.1%; zero NoKvSlot. Dominant remaining pure-miss bucket identified: Paraphrased questions with inverted entity word order, which contiguous anchors cannot match -- candidates are an unordered-proximity anchor variant (Tier 1.5) or Tier 2 embeddings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… entity word order The Tier 1 validation run's dominant residual pure-miss bucket (13 of 21) was Paraphrased questions that invert an entity's word order -- "the Meridian relay point" vs the corpus's "Relay Meridian". Contiguous anchors fail twice there: matching can't bridge the inversion, and extraction yields no anchor at all because only "Meridian" is capitalized (no 2+ word proper-noun run exists in the question). Fix: every mid-sentence capitalized word becomes an entity head paired with its nearest non-stopword neighbor (within two positions each side); a card matches when both words occur within a 2-token window in ANY order. Pairs feed the same lexicographic anchor key as verbatim anchors at half weight (0.5/df) -- a contiguous phrase match still wins wherever both exist -- and participate in the Tier 1b coverage bookkeeping. Questions yielding no pairs behave exactly as Tier 1. Three new tests, including one where the distractor shares MORE question unigrams than the correct card so plain unigram IDF provably prefers it -- only the proximity pair rescues the target. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
100-question Meta-Llama run on NEWCOREPC at 6d28071, identical conditions to prior runs. Cumulative across three deterministic lexical fixes: 31 -> 45 -> 56 pass rate, pure retrieval misses 49 -> 21 -> 10, citation precision 99.1% throughout, zero NoKvSlot. The partial-miss bucket (24 multi-segment questions) is now the largest failure class. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FabricEvidenceProcessor.NormalizeAndValidate previously rejected evidence cards when the model over-generated on expanded-corpus dense segments: - summary > 2000 chars: outright rejection (11 failures observed) - claims > 64 items: outright rejection - duplicate canonical claimId: error + rejection Each rejection silently removed the segment card from the evidence pack, blocking every question that needed that segment. Now: - Long summaries are truncated to MaxSummaryChars with a trailing ellipsis - Excess claims are silently truncated to the first MaxClaims entries - Duplicate canonical claimIds are silently deduped (first occurrence kept) Three new unit tests cover the new behaviors; the existing rejection test for canonical-id collisions is updated to expect dedup-acceptance. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ing drift Two outstanding CodeRabbit findings from PR #40 review rounds 1-2: 1. Consolidate duplicate 1.15 safety margin FabricContextBudget.TokenSafetyMargin is now the single source of truth, referenced from ContextFabricFeasibilityRunner, FabricBoundaryStitcher, and EvidencePackBuilder. Rationale lives in FabricContextBudget (in ContextFabricContracts.cs, which compiles into NativeRuntime unlike EvidencePackBuilder.cs, which is why the comment previously explained the duplication instead of fixing it). 2. Fix reporting drift from margin-adjusted estimate reuse InvokeAsync in ContextFabricFeasibilityRunner and FabricBoundaryStitcher previously used the margin-adjusted promptTokens as both the budget gate threshold AND the FabricCallMetrics fallback -- inflating reported prompt counts by 15% whenever the native runtime did not report actual usage. Now: rawPromptTokens for reporting, gatePromptTokens (x1.15) for the gate. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Phase 1: Model Benchmark window — storage schema + read-only CF-7 results display ModelBenchmarkRecord + ModelBenchmarkStore: new storage schema for per-model Context Fabric benchmark results (model-benchmark-1.0). The store scans .orc/adversarial/ on open, joining each cf7_gate_*.json to the nearest-in-time cf0_*.json B3 sub-report to recover the model filename from environment.lanes. De-duped by (modelFilename + tier), so the latest run per (model, tier) pair is kept. ModelBenchmarkWindow.axaml/.cs: new Avalonia window accessible via Models → Model Benchmark…. On open it scans .orc/adversarial/ and renders one card per model: a tier + date header, GO/NO-GO verdict badge, and metric chips for question pass rate, citation precision, segment coverage, and boundary stitch rate. Depot GGUF files with no benchmark record render as a dimmed "not benchmarked" row. "Run New Benchmark…" button present but disabled (Phase 2). Phase 1 is read-only display of existing CF-7 run artifacts. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> @
…light rejection Today's NEWCOREPC exit-66 (SmolLM2 selected for CF) turned out to be depot state, not a resolver bug: Meta-Llama-3.1-8B was .gguf.disabled at run time, leaving only Rejected candidates. The rejection message now lists every scanned base model with its ContextFabricReader verdict and any skipped .gguf.disabled files, so this failure mode is self-explanatory next time. Also make the Model Benchmark window depot listing recursive so GGUFs in subdirectories (e.g. content-addressed 2f/) appear as unbenchmarked rows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
….5->88.2% Second model family confirms the Tier 2 truncation fix direction; B2 RAG baseline (49/100) still beats B3 on qwen-7b, unlike Meta-Llama on NEWCOREPC. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
6 tasks
This was referenced Jul 15, 2026
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR tells a complete investigation story across four layers, each validated against real 100-question CF-7 gate runs on real hardware (all runs logged in `docs/CF_TEST_RESULTS.md`):
1. NoKvSlot crash fixes and root-cause (commits 1–6)
2. Retrieval failure analysis + Tier 1 fix (commit 7)
Failure categorization of the Meta-Llama-3.1-8B 100-question run proved 100% of B3 failures were retrieval misses (69/69 — the required segment was absent from the evidence pack; zero model failures). Root mechanism, measured: unigram bag-of-words IDF dissolves multi-word entities ("Station Alpha" → 63/105 cards tie; 3 contain the phrase) and hyphenated identifiers ("BR-048" → {br, 048}).
Fix: anchor-phrase extraction ranked lexicographically above unigram score, coverage-aware greedy fill for multi-entity questions, verbatim identifier matching on the Exhaustive path. Validated: 31/100 → 45/100, B3 beats B2 RAG baseline for the first time.
3. Tier 1.5 — Proximity pairs for inverted entity word order (commit 8)
Paraphrased questions invert entity word order ("the Meridian relay point" vs corpus "Relay Meridian"). Contiguous phrase anchors fail silently there. Added unordered proximity pairs: mid-sentence capitalized word + nearest non-stopword neighbor within ±2 positions, matched within 2-token window in either order, at half verbatim anchor weight. Validated: 45/100 → 56/100, Paraphrased pure-misses 13 → 3.
4. Tier 2 — Reader-rejection fix (commit 9)
`FabricEvidenceProcessor.NormalizeAndValidate` rejected cards outright when the model over-generated on expanded-corpus dense segments: summary > 2 000 chars, claims > 64, or duplicate canonical claimId. Each rejection silently removed the segment card from the evidence pack. Now:
Validation run in progress on all three fleet machines with Meta-Llama.
5. CodeRabbit cleanup (commit 10)
Documentation
Test plan
🤖 Generated with Claude Code