CF benchmark remediation: honest evidence-selection fix + Foundry F-1 start

[hardcoreerik]

Summary

This branch does two things:

1. Closes out the CF-7 benchmark remediation that started after an adversarial pass found the prior "GO" verdict was overclaimed (the old corpus had answer-hint markup CF could pattern-match). Builds an un-marked 128-segment corpus, 150 real held-out questions (85 host-templated + 65 externally-authored, verified against the rendered corpus before freezing), a permanent JSON-recovery fix for token-boundary artifacts (falseC/trueX), and a live re-run script.
2. Starts Foundry F-1 (theorc-toolcaller v0): a frozen, code-verified tool inventory, a new dataset capture schema, a working mechanical validator, and a live capture hook so TheOrc generates its own F-1 training data from real swarm tool-call decisions instead of synthetic-only authoring.

The honest result

The full 120-question gate run against the real held-out set closed NO-GO:

System	Result
B0 — closed-book	16/120 (13%)
B1 — truncated prompt	31/120 (26%)
B2 — top-k RAG (pre-fix)	25/120 (21%)
B3 — single-node Context Fabric	56/120 (47%)

CF clearly beats every baseline, and every honesty metric passed cleanly (citation_precision 100%, segment_terminal_coverage 100%, boundary_stitch_pass_rate 100%) — CF never fabricates evidence. The only failing gate is question_pass_rate (target 100%).

Root cause, found and fixed

Comparing expectedSegmentIds against includedSegmentIds on failing questions showed CF frequently never gathered the segments containing the answer — an evidence-selection bug, not a reasoning failure. Two real production-code fixes landed:

• BuildEvidencePack (ContextFabricFeasibilityRunner.cs — the actual production Context Fabric answering path used by FabricNativeReaderService and HiveNativeRoleExecutorAdapter, not benchmark-only code) had a hardcoded maxCards cap (1/2/4) with no documented cost/latency justification, structurally incapable of answering GlobalSynthesis questions needing up to 8 segments. Replaced with IDF-weighted, stopword-aware scoring and budget-fill selection instead of a fixed count.
• The same fix was applied to the B2 benchmark baseline (ContextFabricBaselineRunner.BuildTopKText) first — it had the identical bug, and its old naive implementation actually scored worse (21%) than the dumber B1 baseline (26%), which is itself good evidence the bug was real.

Known, explicitly out-of-scope remainder: FabricQuestionKind.Exhaustive (0/12) doesn't go through BuildEvidencePack at all — it hits a separate method, BuildExhaustiveAnswer, with its own literal-keyword-overlap filter. Different root cause, not fixed here, tracked as a follow-up.

Neither fix has been validated against a fresh full 120-question run yet — that's the natural next step once this PR is reviewed.

Test plan

• Full unit suite: 467 pass, 0 fail, 4 skipped
• Solution builds clean (0 warnings on touched projects)
• Both selection-logic fixes have direct unit tests exercising real behavior (not just compilation): rare-term preference over common-word noise, no more fixed segment/card caps, correct empty/budget-exhausted fallbacks
• End-to-end proof: a real toolcaller capture produced by the new live capture hook was run through the real Tools/ToolcallerBench validator and passed with 0 findings
• Full 120-question NEWCOREPC re-run with both fixes in place (~5 hours, not yet started — pending review)

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added open-extraction reading mode, expanded deterministic corpus generation, host-templated question generation, and deterministic dev/held-out splitting.
  • Added tool-capture staging with a settings toggle and status-bar indicator, plus a new toolcaller-bench validator that outputs JSON/Markdown reports.
  • Extended Context Fabric benchmarks with ledger export/merge workflows and the “cf7-gate-expanded” suite.

• Bug Fixes

  • Improved JSON recovery (lenient parsing and keyword-literal suffix sanitization) and refined model admission thresholding.

• Tests

  • Added/expanded unit tests covering top-k/evidence selection, exhaustive answering, parsing/sanitization, dataset capture staging, and gating logic.

• Documentation

  • Added toolcaller-v0 frozen inventory and capture schema documentation.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: adf0994c-4a27-48c2-a170-246720a71480

📥 Commits

Reviewing files that changed from the base of the PR and between 40d79e1 and 2e18805.

📒 Files selected for processing (7)

• OrchestratorIDE.Avalonia/MainWindow.axaml
• OrchestratorIDE.Avalonia/MainWindow.axaml.cs
• OrchestratorIDE.Avalonia/UI/Panels/SettingsPanel.axaml
• OrchestratorIDE.Avalonia/UI/Panels/SettingsPanel.axaml.cs
• OrchestratorIDE.UnitTests/ToolcallerDatasetCaptureTests.cs
• OrchestratorIDE/Core/AppSettings.cs
• OrchestratorIDE/Services/Swarm/ToolcallerDatasetCapture.cs

✅ Files skipped from review due to trivial changes (1)

• OrchestratorIDE.Avalonia/MainWindow.axaml

🚧 Files skipped from review as they are similar to previous changes (2)

• OrchestratorIDE/Services/Swarm/ToolcallerDatasetCapture.cs
• OrchestratorIDE.UnitTests/ToolcallerDatasetCaptureTests.cs

---

📝 Walkthrough

Walkthrough

This PR expands Context Fabric with an expanded deterministic corpus, authored-question tooling, IDF-based scoring, open-extraction reading, and JSON recovery. It also adds live toolcaller capture staging, a frozen-tool validation benchmark, CLI/runner wiring, an admission-gate adjustment, and documentation updates including the Warpath whitepaper.

Changes

Context Fabric Expansion and Scoring

Layer / File(s)	Summary
Contracts and corpus generation OrchestratorIDE/Services/ContextFabric/ContextFabricContracts.cs, OrchestratorIDE/Services/ContextFabric/DeterministicExpandedFabricCorpus.cs, OrchestratorIDE.NativeRuntime/OrchestratorIDE.NativeRuntime.csproj, OrchestratorIDE.UnitTests/ContextFabricExpandedCorpusTests.cs, OrchestratorIDE.UnitTests/ContextFabricExhaustiveAnswerTests.cs	Adds the expanded corpus manifest and deterministic generator, plus shared contract changes and corpus validation tests.
Question generation, merging, and splitting OrchestratorIDE/Services/ContextFabric/ExpandedFabricQuestion*.cs, ExpandedFabricAuthoredQuestionMerger.cs, ExpandedFabricLedgerExport.cs, OrchestratorIDE.UnitTests/ExpandedFabric*Tests.cs	Adds host-templated question generation, authored-draft parsing and merging, ledger export, deterministic splitting, and tests.
Scoring, extraction, and JSON recovery OrchestratorIDE/Services/ContextFabric/ContextFabricBaselineRunner.cs, ContextFabricFeasibilityRunner.cs, ContextFabricValidation.cs, OrchestratorIDE.UnitTests/ContextFabricB2TopKRagTests.cs, ContextFabricEvidencePackTests.cs, ContextFabricCf0Tests.cs, ContextFabricOpenExtractionTests.cs, ModelDepotTests.cs	Reworks top-k RAG and evidence packing around IDF scoring, adds open-extraction reading and JSON recovery, and updates supporting tests.
Benchmark CLI and gate wiring Tools/ContextFabricBench/Program.cs, Run-CF7GateExpanded.ps1, OrchestratorIDE/Core/Runtime/ModelAdmissionGate.cs, OrchestratorIDE.UnitTests/ModelDepotTests.cs, docs/CONTEXT_FABRIC_BENCHMARK_MANIFEST.md	Adds expanded benchmark suites and runner support, the CF-7 expanded script, the model admission tier change, and related docs.

Estimated code review effort: 4 (Complex) | ~75 minutes

Toolcaller Dataset Capture and Validation

Layer / File(s)	Summary
Capture staging and UI wiring OrchestratorIDE/Services/Swarm/ToolcallerDatasetCapture.cs, OrchestratorIDE/Agents/SwarmSession.cs, OrchestratorIDE.Avalonia/..., OrchestratorIDE.UnitTests/ToolcallerDatasetCaptureTests.cs, OrchestratorIDE.Avalonia/OrchestratorIDE.Avalonia.csproj	Stages tool-call and no-tool examples from the swarm worker loop and adds capture tests plus UI/settings and project wiring.
Frozen-tool benchmark and validator Tools/ToolcallerBench/*, OrchestratorIDE.slnx, training_pit/schemas/toolcaller_v0_frozen_tools.json, docs/README.md, docs/TOOLCALLER_V0_FROZEN_INVENTORY.md, training_pit/TOOLCALLER_CAPTURE_SCHEMA.md	Adds the frozen-tool validator CLI, data contracts, report writer, schema fixture, solution entry, and related docs.

Estimated code review effort: 3 (Moderate) | ~30 minutes

Repository Configuration and Documentation

Layer / File(s)	Summary
Publish ignore pattern .gitignore	Broadens the ignore rule to match publish-prefixed directories.
Warpath whitepaper docs/THEORC_WARPATH_GAMIFICATION_WHITEPAPER.md	Adds the Warpath gamification specification document.

Estimated code review effort: 2 (Simple) | ~15 minutes

Sequence Diagram(s)

sequenceDiagram
  participant SwarmSession
  participant ToolcallerDatasetCapture
  participant StagingDir
  SwarmSession->>ToolcallerDatasetCapture: StageCallAsync(...)
  ToolcallerDatasetCapture->>StagingDir: write capture JSON
  SwarmSession->>ToolcallerDatasetCapture: StageNoToolAsync(...)
  ToolcallerDatasetCapture->>StagingDir: write capture JSON

Loading

sequenceDiagram
  participant Program
  participant ToolcallerCaptureValidator
  participant ToolcallerReportWriter
  Program->>ToolcallerCaptureValidator: Validate(captures, frozenTools, hash)
  ToolcallerCaptureValidator-->>Program: ToolcallerValidationReport
  Program->>ToolcallerReportWriter: WriteAsync(report, outputDir)
  ToolcallerReportWriter-->>Program: JSON + Markdown paths

Loading

Possibly related PRs

• hardcoreerik/TheOrc#15: Shares the Context Fabric execution path touched by the open-extraction and feasibility changes here.
• hardcoreerik/TheOrc#29: Connects through citation/source-label plumbing used by Context Fabric verification and answer rendering.
• hardcoreerik/TheOrc#32: Modifies ContextFabricBaselineRunner.cs, which is also rewritten in this PR.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately captures the two main themes: CF benchmark remediation and the new Foundry F-1 toolcaller capture work.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/cf-benchmark-remediation

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 13

🧹 Nitpick comments (14)

OrchestratorIDE/Agents/SwarmSession.cs (1)

2066-2070: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Un-timeboxed, awaited capture write sits in the hot tool-dispatch path.

StageCallAsync is awaited synchronously before every real tool call is dispatched (once per tool call, not once per run like the existing DatasetCapture.StageAsync precedent). Its internal try/catch swallows exceptions, but a slow/unresponsive staging directory (e.g. network share, low disk) has no timeout and will stall the worker loop rather than just failing the capture. Consider fire-and-forget (_ = StageCallAsync(...)) or wrapping the write in a short timeout so capture latency/hangs never propagate to real tool execution.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@OrchestratorIDE/Agents/SwarmSession.cs` around lines 2066 - 2070,
`SwarmSession` is blocking the hot tool-dispatch path by awaiting
`ToolcallerDatasetCapture.StageCallAsync` before each real tool call, so capture
latency can stall execution. Update the call site to avoid propagating staging
delays by making the capture fire-and-forget or by wrapping `StageCallAsync` in
a short timeout, while keeping the existing best-effort behavior and not letting
failures or hangs affect tool dispatch.

OrchestratorIDE/Services/Swarm/ToolcallerDatasetCapture.cs (1)

98-107: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Hardcoded policy_gap_tool list duplicates ToolPolicyEngine internals.

policy_gap_tool = call.Name is "grep_code" or "ask_user" hardcodes knowledge that ToolPolicyEngine.Evaluate() has no dedicated case for these two tools (per docs/TOOLCALLER_V0_FROZEN_INVENTORY.md's verification notes). If ToolPolicyEngine ever adds a dedicated case for either tool, this literal silently goes stale and captures will keep reporting a gap that no longer exists.

Consider adding a unit test in the ToolPolicyEngine test suite that fails if a tool this file lists is given a dedicated Evaluate() case, so the two stay in sync.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@OrchestratorIDE/Services/Swarm/ToolcallerDatasetCapture.cs` around lines 98 -
107, The Toolcaller dataset capture is hardcoding `policy_gap_tool` in
`ToolcallerDatasetCapture` using `call.Name is "grep_code" or "ask_user"`, which
duplicates `ToolPolicyEngine` knowledge and can drift. Move this knowledge into
a sync check by adding a unit test in the `ToolPolicyEngine` test suite that
asserts any tool listed here still lacks a dedicated `Evaluate()` case, and
update the capture logic only through that shared contract if the engine
changes.

training_pit/TOOLCALLER_CAPTURE_SCHEMA.md (1)

187-190: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Fenced code block missing language identifier.

Flagged by markdownlint (MD040).

📝 Fix

-```
+```text
 training_pit/datasets/toolcaller/
   toolcaller_capture_{split}_{example_id}.json

</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @training_pit/TOOLCALLER_CAPTURE_SCHEMA.md around lines 187 - 190, The fenced
code block in TOOLCALLER_CAPTURE_SCHEMA.md is missing a language identifier and
should be updated to satisfy markdownlint MD040. Locate the Markdown snippet
containing the dataset path example and change the opening fence to use a text
language tag so the rendered block remains unchanged while complying with the
linter.

</details>

<!-- cr-comment:v1:495ec2b30960f5ea899030ac -->

_Source: Linters/SAST tools_

</blockquote></details>
<details>
<summary>OrchestratorIDE.UnitTests/ToolcallerDatasetCaptureTests.cs (1)</summary><blockquote>

`79-95`: _📐 Maintainability & Code Quality_ | _🔵 Trivial_ | _⚡ Quick win_

**Test name promises "ask_user" coverage but only exercises `grep_code`.**

`StageCallAsync_FlagsPolicyGapTool_ForGrepCodeAndAskUser` never constructs an `ask_user` call, so the `ask_user` branch of `policy_gap_tool` is unverified by this test despite the name.

<details>
<summary>💡 Suggested addition</summary>

```diff
     [Test]
     public async Task StageCallAsync_FlagsPolicyGapTool_ForGrepCodeAndAskUser()
     {
         var stagingDir = NewTempDir();
         var workspaceRoot = NewTempDir();
         var task = new SwarmTask { Title = "Find usages", Description = "Find usages of Foo.", Role = SwarmWorkerRole.Researcher };
         var call = new ToolCall { Name = "grep_code", Arguments = new() { ["pattern"] = "Foo" } };
         var availableTools = new List<ToolDefinition> { new() { Name = "grep_code", Description = "Search.", Parameters = new() } };

         await ToolcallerDatasetCapture.StageCallAsync(
             "20260703_130000", task, "qwen2.5-coder:14b", call, availableTools, workspaceRoot, stagingDir);

         var file = Directory.GetFiles(stagingDir, "toolcaller_capture_*.json").Single();
         using var doc = JsonDocument.Parse(await File.ReadAllTextAsync(file));

         Assert.That(doc.RootElement.GetProperty("policy_outcome").GetProperty("policy_gap_tool").GetBoolean(), Is.True);
+
+        // also verify ask_user
+        var askCall = new ToolCall { Name = "ask_user", Arguments = new() { ["question"] = "Which config?" } };
+        await ToolcallerDatasetCapture.StageCallAsync(
+            "20260703_130001", task, "qwen2.5-coder:14b", askCall, availableTools, workspaceRoot, stagingDir);
+        var askFile = Directory.GetFiles(stagingDir, "toolcaller_capture_*20260703_130001*.json").Single();
+        using var askDoc = JsonDocument.Parse(await File.ReadAllTextAsync(askFile));
+        Assert.That(askDoc.RootElement.GetProperty("policy_outcome").GetProperty("policy_gap_tool").GetBoolean(), Is.True);
     }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@OrchestratorIDE.UnitTests/ToolcallerDatasetCaptureTests.cs` around lines 79 -
95, The test name implies coverage for both grep_code and ask_user, but
StageCallAsync_FlagsPolicyGapTool_ForGrepCodeAndAskUser only exercises
grep_code. Update ToolcallerDatasetCaptureTests by adding an ask_user ToolCall
path in the same test or splitting into a separate test, and assert the
policy_outcome.policy_gap_tool result for both branches so the behavior in
StageCallAsync is actually covered.

Tools/ToolcallerBench/ToolcallerCaptureValidator.cs (1)

146-158: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Lineage-group-split gate is currently unreachable for organic captures.

Per the ToolcallerDatasetCapture.cs context snippet, lineage_group_id is set equal to the unique example_id for organic swarm captures ("organic capture, no paraphrase/repair siblings yet"), so every group here will have exactly one member and this gate can never fire today. Not a bug, just worth noting it's dormant until paraphrase/repair lineage groups are introduced.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@Tools/ToolcallerBench/ToolcallerCaptureValidator.cs` around lines 146 - 158,
The lineage_group_split_conflict check in ToolcallerCaptureValidator is
currently dormant for organic captures because lineage_group_id maps 1:1 to
example_id, so each GroupBy(c => c.LineageGroupId) bucket has only one capture.
Update this validation to only run when a lineage group can actually contain
multiple members (for example, after paraphrase/repair siblings exist), or add
an explicit guard around the group.Count() case so the Fail path in the Split
consistency check only applies to real multi-capture lineage groups.

OrchestratorIDE/Services/ContextFabric/ContextFabricFeasibilityRunner.cs (5)

642-652: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Duplicate <summary> XML doc blocks on the same method.

Two separate <summary> elements are stacked back-to-back on BuildEvidencePack (lines 642-651 and 652). This is invalid/confusing XML doc structure; most tooling will only surface the first block, making the second effectively dead documentation.

✏️ Proposed fix: merge into a single summary

     /// <summary>
     /// Builds the evidence pack sent to the answerer for a question. Ranks every evidence card by
     /// IDF-weighted term overlap (rare, distinctive terms count more than common words) and greedily
     /// fills the actual context budget, rather than a fixed per-question-kind card count. A fixed
     /// count is structurally unable to answer questions whose evidence spans more cards than that
     /// count regardless of ranking quality -- GlobalSynthesis questions can need up to 8 segments'
     /// worth of evidence, and the old hardcoded caps (1/2/4) had no documented latency or cost
     /// justification. This mirrors the same fix already applied to the B2 benchmark baseline
     /// (ContextFabricBaselineRunner.BuildTopKText).
+    /// Internal (not private) so unit tests can exercise evidence selection directly.
     /// </summary>
-    /// <summary>Internal (not private) so unit tests can exercise evidence selection directly.</summary>
     internal EvidencePack BuildEvidencePack(

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@OrchestratorIDE/Services/ContextFabric/ContextFabricFeasibilityRunner.cs`
around lines 642 - 652, Merge the two stacked XML documentation summaries on
BuildEvidencePack into one valid <summary> block, and keep all of the existing
description in a single doc comment so tooling only sees one authoritative
summary for ContextFabricFeasibilityRunner.BuildEvidencePack.

---

264-290: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Duplicated reader prompt boilerplate between the open/marked branches.

Both the [FABRIC_READER_OPEN] and [FABRIC_READER] system prompts repeat the same output-shape JSON contract, citation rules, and IDs-exactly language almost verbatim, differing mainly in extraction strategy. Consider factoring the shared suffix (schema/output-shape/citation-offset instructions) into a constant reused by both branches to avoid drift between the two prompts over time.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@OrchestratorIDE/Services/ContextFabric/ContextFabricFeasibilityRunner.cs`
around lines 264 - 290, The two branches in ContextFabricFeasibilityRunner’s
SystemMessage are duplicating the same schema/output-contract and citation
guidance, which risks prompt drift. Factor the shared boilerplate
(schemaVersion, promptVersion, citation rules, IDs-exactly wording, and output
shape JSON) into a reusable constant/helper, then compose it into both the
openExtraction and non-openExtraction prompts while keeping only the
branch-specific extraction instructions different.

---

926-939: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Naming convention: _evidencePackStopwords deviates from the local s_ static-field convention.

FabricJson in the same PR uses s_lenientOptions/s_jsonKeywords for private static readonly fields, while this new field uses a bare underscore prefix, which typically denotes an instance field in this codebase's convention.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@OrchestratorIDE/Services/ContextFabric/ContextFabricFeasibilityRunner.cs`
around lines 926 - 939, The new private static readonly stopword set in
ContextFabricFeasibilityRunner should follow the established static-field naming
convention used by symbols like s_lenientOptions and s_jsonKeywords. Rename
_evidencePackStopwords to a s_-prefixed name and update any references in
BuildEvidencePack or related scoring helpers so the field name clearly indicates
it is static.

---

653-704: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Quadratic re-serialization cost in the evidence-budget loop.

For each candidate card, BuildEvidencePack appends it to evidence, re-serializes the entire growing evidence array via FabricJson.Serialize(input), and re-estimates tokens -- an O(n²) cost as the number of ranked cards grows. The docstring on the sibling test file confirms this is the real production answering path (used by FabricNativeReaderService/HiveNativeRoleExecutorAdapter), and the corpus this now needs to scale to is 128 segments (up from the old 16-segment fixture), so this cost is no longer negligible.

Consider tracking a running token estimate incrementally (base overhead once, plus each candidate's own estimated contribution) instead of re-serializing the full accumulated evidence set on every iteration.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@OrchestratorIDE/Services/ContextFabric/ContextFabricFeasibilityRunner.cs`
around lines 653 - 704, BuildEvidencePack is re-serializing the full growing
evidence list on every candidate, causing quadratic token-estimation cost.
Update the loop to avoid calling FabricJson.Serialize(input) and EstimateTokens
over the entire projected AnswerInput each time; instead, in BuildEvidencePack
track a running budget incrementally using a fixed base cost plus each
candidate’s estimated contribution. Keep the existing ranking/filtering flow
around ordered, evidence, and included, but compute the budget check without
reconstructing projected arrays on every iteration.

---

920-925: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Remove the unused Score helper
OrchestratorIDE/Services/ContextFabric/ContextFabricFeasibilityRunner.cs:920 no longer calls this method; BuildExhaustiveAnswer inlines the same token-count logic and BuildEvidencePack uses ScoreIdf. Keeping it adds dead-code drift.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@OrchestratorIDE/Services/ContextFabric/ContextFabricFeasibilityRunner.cs`
around lines 920 - 925, The unused Score helper in
ContextFabricFeasibilityRunner should be removed because its token-count logic
is no longer referenced by BuildExhaustiveAnswer or BuildEvidencePack. Delete
the Score method and keep CardHaystack only if it is still used elsewhere in the
class, ensuring there are no remaining references to Score or dead-code drift in
the feasibility runner.

OrchestratorIDE.UnitTests/ContextFabricCf0Tests.cs (1)

206-216: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Test doesn't actually exercise the sanitizer it's named for.

The payload {"...,"answer":"falsehood is trueColor nullValue","abstained":false,"claims":[]} is already valid JSON, so ParseModelObject's strict fast-path deserializes it successfully before TrySanitizeLiteralSuffixes is ever invoked (see ContextFabricValidation.cs ParseModelObject). The test would pass identically even if the sanitizer had a bug that corrupted string contents, since that code path is never reached.

✅ Proposed fix: force an outside-string defect so sanitization actually runs

     public void JsonParser_SanitizesKeywordSuffix_DoesNotCorruptStringContents()
     {
         // "falsehood" and "trueColor" inside JSON string values must be preserved verbatim —
         // the sanitizer is only allowed to modify tokens that appear outside string boundaries.
+        // "abstained":falseX forces the sanitizer path to actually run.
         var parsed = FabricJson.ParseModelObject<FabricAnswerDraft>(
-            "{\"schemaVersion\":\"cf0-answer-1.0\",\"answer\":\"falsehood is trueColor nullValue\",\"abstained\":false,\"claims\":[]}");
+            "{\"schemaVersion\":\"cf0-answer-1.0\",\"answer\":\"falsehood is trueColor nullValue\",\"abstained\":falseX,\"claims\":[]}");

         Assert.That(parsed.Answer, Is.EqualTo("falsehood is trueColor nullValue"));
         Assert.That(parsed.Abstained, Is.False);
     }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@OrchestratorIDE.UnitTests/ContextFabricCf0Tests.cs` around lines 206 - 216,
The JsonParser_SanitizesKeywordSuffix_DoesNotCorruptStringContents test is
currently hitting the strict ParseModelObject fast-path, so
TrySanitizeLiteralSuffixes is never exercised. Change the payload in
ContextFabricCf0Tests so it includes a non-string JSON defect that forces
sanitization to run in ParseModelObject/FabricJson, while still keeping the
answer text with falsehood/trueColor/nullValue inside string boundaries. Keep
the assertions on FabricAnswerDraft.Answer and Abstained to verify the sanitizer
preserves string contents and still parses the object correctly.

Tools/ContextFabricBench/Run-CF7GateExpanded.ps1 (1)

223-238: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Rename $Args — it shadows PowerShell's automatic variable.

Static analysis flags $Args as an automatic variable; assigning to it can have undesired side effects (e.g. if this logic is ever moved into a function, $Args/$args would collide with the function's own unbound-argument collection).

♻️ Proposed rename

-$Args = @(
+$benchArgs = @(
     "--suite",            "cf7-gate-expanded",
     ...
 )

 if ($MaxQuestions -gt 0) {
-    $Args += @("--max-questions", $MaxQuestions)
+    $benchArgs += @("--max-questions", $MaxQuestions)
 }

 if ($GpuLayers -ne -1) {
-    $Args += @("--gpu-layers", $GpuLayers)
+    $benchArgs += @("--gpu-layers", $GpuLayers)
 }
...
-& $BenchExe `@Args` 2>&1 | Tee-Object -FilePath $LogFile
+& $BenchExe `@benchArgs` 2>&1 | Tee-Object -FilePath $LogFile

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@Tools/ContextFabricBench/Run-CF7GateExpanded.ps1` around lines 223 - 238,
Rename the $Args array used in the cf7-gate-expanded argument assembly so it
does not shadow PowerShell’s automatic $args variable. Update the variable and
all append/use sites in the script’s argument-building block, including the
conditional additions for max-questions and gpu-layers, to a distinct name that
clearly represents the command arguments.

Source: Linters/SAST tools

OrchestratorIDE/Services/ContextFabric/ExpandedFabricAuthoredQuestionMerger.cs (1)

20-31: 🚀 Performance & Scalability | 🔵 Trivial

Bracket-slicing JSON extraction has no recovery path for truncated/malformed model output.

ParseDrafts takes everything between the first [ and last ]. If the authored draft text contains any [/] characters before/after the actual array (e.g. markdown, footnotes, or a truncated completion missing the closing bracket), this either mis-slices the JSON or throws unhandled JsonException, aborting the entire merge-authored run rather than isolating the bad draft. The PR description mentions a JSON-recovery fix was added elsewhere for token-boundary artifacts; consider reusing that same recovery logic here for consistency, since authored drafts plausibly come from LLM completions subject to the same truncation risk.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@OrchestratorIDE/Services/ContextFabric/ExpandedFabricAuthoredQuestionMerger.cs`
around lines 20 - 31, ParseDrafts currently slices from the first `[` to the
last `]`, which can mis-extract or fail hard on markdown noise or truncated LLM
output. Update `ExpandedFabricAuthoredQuestionMerger.ParseDrafts` to reuse the
same JSON recovery/parsing approach used elsewhere in the merge flow instead of
relying on bracket slicing, so malformed authored draft text is isolated and
does not abort the entire `merge-authored` run. Keep the fix localized to the
JSON extraction/deserialization path and ensure `JsonException` from one bad
draft is handled consistently with the existing recovery logic.

Tools/ContextFabricBench/Program.cs (1)

216-357: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy lift

Large duplication between Cf7GateExpanded and Cf7Gate gate-running logic.

The new Cf7GateExpanded block (Lines 216-303) and the existing Cf7Gate block (Lines 305-357) both run quote-anchor diagnostics, boundary-stitch diagnostics, the B0/B1/B2 baseline loop, load the B4 artifact, evaluate the gate, and print the same shaped verdict/report output — differing mainly in which fixture/run-options feed the calls. Consider extracting a shared RunGateAsync(fixture, runOptions, diagnosticsFixture, label, ...) helper to avoid the two paths drifting independently as gate logic evolves.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@Tools/ContextFabricBench/Program.cs` around lines 216 - 357, The
Cf7GateExpanded and Cf7Gate paths duplicate nearly the same gate-evaluation
flow, so extract the shared quote/stitch diagnostics, B0/B1/B2 baseline loop, B4
artifact loading, gate evaluation, and verdict/report printing into a common
helper in Program.cs. Keep the helper parameterized by the fixture and
runOptions differences so both the Cf7GateExpanded and Cf7Gate branches call the
same logic and won’t drift as the gate pipeline evolves.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/THEORC_WARPATH_GAMIFICATION_WHITEPAPER.md`:
- Around line 774-789: The trophy schema for The Black Anvil depends on
foundry.candidate.deployed_artifact_verified, but that event is not defined
anywhere in the canonical event inventory. Add the missing event definition to
the Foundry event inventory/source-of-truth and ensure the trophy definition in
the trophy schema references that exact event name consistently, so the
validator and badge rules use a single canonical symbol.
- Around line 324-349: The Warpath event identifier format in the event record
needs to avoid collisions across concurrent emitters. Update the event
schema/example around the immutable event object to use a collision-resistant ID
in event_id (for example UUID or ULID) while keeping occurred_at as a separate
timestamp field, and adjust any text or examples that currently imply sequential
wp_evt_..._0001-style IDs. Refer to the event payload shown in the Warpath event
section when making the change.
- Around line 888-896: The GitHub badge export section currently relies on
remote shields.io image URLs, which breaks the local-only export goal and can
leak metadata. Update the “GitHub Badge Export” content to use embedded/local
assets or route this example through a plain-text export path instead of
external badge links, keeping the change localized to the badge export snippet
in the whitepaper.
- Around line 699-710: The starter badge list in the HIVE and Warband Badges
section includes a cloud-specific reward that conflicts with the local-first
Warpath spec. Remove the Cloud Raider badge from the MVP badge table, or
rename/reframe it so it only rewards local or distributed execution with no
cloud dependency. Keep the remaining badges in the list unchanged and ensure the
badge triggers described in the whitepaper stay aligned with the product
promise.

In `@OrchestratorIDE.UnitTests/ContextFabricB2TopKRagTests.cs`:
- Around line 111-133: The test in
BuildTopKText_SkipsOverBudgetSegment_ButStillFitsShorterLowerRankedOne does not
match the intended ranking because Tokenize dedupes terms per segment, so
repeated “checksum” in the big segment does not raise its score. Update the test
data so the segment referenced by big genuinely outranks small under the
BuildTopKText scoring logic in ContextFabricBaselineRunner, while still being
too large for the token budget; keep small as the shorter fallback that fits and
is selected when big is skipped.

In `@OrchestratorIDE/Services/ContextFabric/ContextFabricContracts.cs`:
- Around line 93-101: `FabricQuestionKind` must remain ordinal-stable because
`FabricJson.Options` persists the enum as numeric values. Update the
`FabricQuestionKind` declaration in `ContextFabricContracts.cs` so existing
members keep their current numeric meanings by assigning explicit values or only
appending new members at the end; if `Paraphrased` must stay in the middle, add
a migration for any stored payloads that depend on the old ordinals. Make the
change in the `FabricQuestionKind` enum without reordering existing values.

In `@OrchestratorIDE/Services/ContextFabric/DeterministicExpandedFabricCorpus.cs`:
- Around line 261-282: ValidateManifestAgainstRenderedText currently skips
ExhaustiveCategories, leaving the deterministic case-ledger rows unverified.
Extend the existing self-check in DeterministicExpandedFabricCorpus so it also
validates each FabricExhaustiveCategory by reconstructing the expected row
statement from CategoryId and OccurrenceIds, then comparing it against the
rendered slot text. Use the exhaustiveCategories/exhaustiveRowSlots data already
built in this method to assert every exhaustive row is present and exact, and
surface mismatches the same way the other manifest checks do.

In
`@OrchestratorIDE/Services/ContextFabric/ExpandedFabricAuthoredQuestionMerger.cs`:
- Around line 33-81: The three merge helpers in
ExpandedFabricAuthoredQuestionMerger currently assume draft.TargetId is always
present, but a null value will make the dictionary lookup throw. Update
MergeParaphraseQuestions, MergeMultiHopQuestions, and
MergeGlobalSynthesisQuestions to filter out drafts with missing TargetId before
calling ContainsKey or indexing by ID, keeping the existing byId lookup logic
unchanged.

In `@Tools/ContextFabricBench/Program.cs`:
- Around line 467-476: The `PrintUsage` help text in `Program.cs` is missing the
`export-ledger` and `merge-authored` suite options even though `ParseSuite` and
the unknown-suite message already support them. Update the `Console.WriteLine`
inside `PrintUsage` so the `--suite` list includes `export-ledger` and
`merge-authored` alongside the existing `BenchmarkSuite` values, keeping the
help output aligned with `ParseSuite`.

In `@Tools/ContextFabricBench/Run-CF7GateExpanded.ps1`:
- Around line 152-181: The pre-flight checks in Run-CF7GateExpanded.ps1 are
using Write-Error with string concatenation directly, which PowerShell parses as
separate arguments and causes a positional-parameter error. Update each of the
four Write-Error call sites in the held-out, B4 artifact, ModelRoot, and
GgufCount checks so the full message is parenthesized or built first in a
variable before passing it to Write-Error. Use the existing guard blocks and
identifiers like $HeldOutPath, $B4Artifact, $ModelRoot, and $GgufCount to locate
them.

In `@Tools/ToolcallerBench/ToolcallerCaptureValidator.cs`:
- Around line 64-131: Add explicit validation in ToolcallerCaptureValidator so
unexpected capture.Expected.Decision values fail fast instead of falling through
the non-call branch. Introduce a known allowed set in the validator around the
existing decision gates (the clarify/unsupported reason_code check and the
call/non-call split), and if Decision is anything other than the recognized
values, call Fail with a clear invalid_decision error before any other checks.

In `@Tools/ToolcallerBench/ToolcallerContracts.cs`:
- Around line 26-43: The ToolcallerCapture contract currently relies on
non-nullable types alone, which still allows omitted JSON fields to deserialize
as null; update the ToolcallerCapture record so the critical members are
explicitly required (or otherwise enforced via stricter deserialization
settings) to reject malformed capture payloads at admission. Focus on the
ToolcallerCapture record and its JsonPropertyName-backed members, especially the
non-optional fields like SchemaVersion, ToolSchemaHash, ExampleId,
LineageGroupId, Provenance, Role, Request, AvailableTools, ApprovalState,
Expected, ReviewStatus, and Split.

In `@training_pit/TOOLCALLER_CAPTURE_SCHEMA.md`:
- Around line 4-9: The documented toolcaller staging location is incorrect and
should match the implementation in ToolcallerDatasetCapture and
RunWorkerLoopAsync. Update the description to say captures are written under
.orc/swarm/dataset-staging/ and identified by the toolcaller_capture_ filename
prefix, rather than implying a toolcaller/ subdirectory. Keep the wording
aligned with the actual capture flow and naming used by the dataset staging
code.

---

Nitpick comments:
In `@OrchestratorIDE.UnitTests/ContextFabricCf0Tests.cs`:
- Around line 206-216: The
JsonParser_SanitizesKeywordSuffix_DoesNotCorruptStringContents test is currently
hitting the strict ParseModelObject fast-path, so TrySanitizeLiteralSuffixes is
never exercised. Change the payload in ContextFabricCf0Tests so it includes a
non-string JSON defect that forces sanitization to run in
ParseModelObject/FabricJson, while still keeping the answer text with
falsehood/trueColor/nullValue inside string boundaries. Keep the assertions on
FabricAnswerDraft.Answer and Abstained to verify the sanitizer preserves string
contents and still parses the object correctly.

In `@OrchestratorIDE.UnitTests/ToolcallerDatasetCaptureTests.cs`:
- Around line 79-95: The test name implies coverage for both grep_code and
ask_user, but StageCallAsync_FlagsPolicyGapTool_ForGrepCodeAndAskUser only
exercises grep_code. Update ToolcallerDatasetCaptureTests by adding an ask_user
ToolCall path in the same test or splitting into a separate test, and assert the
policy_outcome.policy_gap_tool result for both branches so the behavior in
StageCallAsync is actually covered.

In `@OrchestratorIDE/Agents/SwarmSession.cs`:
- Around line 2066-2070: `SwarmSession` is blocking the hot tool-dispatch path
by awaiting `ToolcallerDatasetCapture.StageCallAsync` before each real tool
call, so capture latency can stall execution. Update the call site to avoid
propagating staging delays by making the capture fire-and-forget or by wrapping
`StageCallAsync` in a short timeout, while keeping the existing best-effort
behavior and not letting failures or hangs affect tool dispatch.

In `@OrchestratorIDE/Services/ContextFabric/ContextFabricFeasibilityRunner.cs`:
- Around line 642-652: Merge the two stacked XML documentation summaries on
BuildEvidencePack into one valid <summary> block, and keep all of the existing
description in a single doc comment so tooling only sees one authoritative
summary for ContextFabricFeasibilityRunner.BuildEvidencePack.
- Around line 264-290: The two branches in ContextFabricFeasibilityRunner’s
SystemMessage are duplicating the same schema/output-contract and citation
guidance, which risks prompt drift. Factor the shared boilerplate
(schemaVersion, promptVersion, citation rules, IDs-exactly wording, and output
shape JSON) into a reusable constant/helper, then compose it into both the
openExtraction and non-openExtraction prompts while keeping only the
branch-specific extraction instructions different.
- Around line 926-939: The new private static readonly stopword set in
ContextFabricFeasibilityRunner should follow the established static-field naming
convention used by symbols like s_lenientOptions and s_jsonKeywords. Rename
_evidencePackStopwords to a s_-prefixed name and update any references in
BuildEvidencePack or related scoring helpers so the field name clearly indicates
it is static.
- Around line 653-704: BuildEvidencePack is re-serializing the full growing
evidence list on every candidate, causing quadratic token-estimation cost.
Update the loop to avoid calling FabricJson.Serialize(input) and EstimateTokens
over the entire projected AnswerInput each time; instead, in BuildEvidencePack
track a running budget incrementally using a fixed base cost plus each
candidate’s estimated contribution. Keep the existing ranking/filtering flow
around ordered, evidence, and included, but compute the budget check without
reconstructing projected arrays on every iteration.
- Around line 920-925: The unused Score helper in ContextFabricFeasibilityRunner
should be removed because its token-count logic is no longer referenced by
BuildExhaustiveAnswer or BuildEvidencePack. Delete the Score method and keep
CardHaystack only if it is still used elsewhere in the class, ensuring there are
no remaining references to Score or dead-code drift in the feasibility runner.

In
`@OrchestratorIDE/Services/ContextFabric/ExpandedFabricAuthoredQuestionMerger.cs`:
- Around line 20-31: ParseDrafts currently slices from the first `[` to the last
`]`, which can mis-extract or fail hard on markdown noise or truncated LLM
output. Update `ExpandedFabricAuthoredQuestionMerger.ParseDrafts` to reuse the
same JSON recovery/parsing approach used elsewhere in the merge flow instead of
relying on bracket slicing, so malformed authored draft text is isolated and
does not abort the entire `merge-authored` run. Keep the fix localized to the
JSON extraction/deserialization path and ensure `JsonException` from one bad
draft is handled consistently with the existing recovery logic.

In `@OrchestratorIDE/Services/Swarm/ToolcallerDatasetCapture.cs`:
- Around line 98-107: The Toolcaller dataset capture is hardcoding
`policy_gap_tool` in `ToolcallerDatasetCapture` using `call.Name is "grep_code"
or "ask_user"`, which duplicates `ToolPolicyEngine` knowledge and can drift.
Move this knowledge into a sync check by adding a unit test in the
`ToolPolicyEngine` test suite that asserts any tool listed here still lacks a
dedicated `Evaluate()` case, and update the capture logic only through that
shared contract if the engine changes.

In `@Tools/ContextFabricBench/Program.cs`:
- Around line 216-357: The Cf7GateExpanded and Cf7Gate paths duplicate nearly
the same gate-evaluation flow, so extract the shared quote/stitch diagnostics,
B0/B1/B2 baseline loop, B4 artifact loading, gate evaluation, and verdict/report
printing into a common helper in Program.cs. Keep the helper parameterized by
the fixture and runOptions differences so both the Cf7GateExpanded and Cf7Gate
branches call the same logic and won’t drift as the gate pipeline evolves.

In `@Tools/ContextFabricBench/Run-CF7GateExpanded.ps1`:
- Around line 223-238: Rename the $Args array used in the cf7-gate-expanded
argument assembly so it does not shadow PowerShell’s automatic $args variable.
Update the variable and all append/use sites in the script’s argument-building
block, including the conditional additions for max-questions and gpu-layers, to
a distinct name that clearly represents the command arguments.

In `@Tools/ToolcallerBench/ToolcallerCaptureValidator.cs`:
- Around line 146-158: The lineage_group_split_conflict check in
ToolcallerCaptureValidator is currently dormant for organic captures because
lineage_group_id maps 1:1 to example_id, so each GroupBy(c => c.LineageGroupId)
bucket has only one capture. Update this validation to only run when a lineage
group can actually contain multiple members (for example, after
paraphrase/repair siblings exist), or add an explicit guard around the
group.Count() case so the Fail path in the Split consistency check only applies
to real multi-capture lineage groups.

In `@training_pit/TOOLCALLER_CAPTURE_SCHEMA.md`:
- Around line 187-190: The fenced code block in TOOLCALLER_CAPTURE_SCHEMA.md is
missing a language identifier and should be updated to satisfy markdownlint
MD040. Locate the Markdown snippet containing the dataset path example and
change the opening fence to use a text language tag so the rendered block
remains unchanged while complying with the linter.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 14eef075-a8a0-42c0-9671-3a4bec991b9c

📥 Commits

Reviewing files that changed from the base of the PR and between 2db7b40 and c68e01c.

📒 Files selected for processing (40)

• .gitignore
• OrchestratorIDE.Avalonia/OrchestratorIDE.Avalonia.csproj
• OrchestratorIDE.NativeRuntime/OrchestratorIDE.NativeRuntime.csproj
• OrchestratorIDE.UnitTests/ContextFabricB2TopKRagTests.cs
• OrchestratorIDE.UnitTests/ContextFabricCf0Tests.cs
• OrchestratorIDE.UnitTests/ContextFabricEvidencePackTests.cs
• OrchestratorIDE.UnitTests/ContextFabricExpandedCorpusTests.cs
• OrchestratorIDE.UnitTests/ContextFabricOpenExtractionTests.cs
• OrchestratorIDE.UnitTests/ExpandedFabricAuthoredQuestionMergerTests.cs
• OrchestratorIDE.UnitTests/ExpandedFabricQuestionGeneratorTests.cs
• OrchestratorIDE.UnitTests/ExpandedFabricQuestionSplitterTests.cs
• OrchestratorIDE.UnitTests/ModelDepotTests.cs
• OrchestratorIDE.UnitTests/ToolcallerDatasetCaptureTests.cs
• OrchestratorIDE.slnx
• OrchestratorIDE/Agents/SwarmSession.cs
• OrchestratorIDE/Core/Runtime/ModelAdmissionGate.cs
• OrchestratorIDE/Services/ContextFabric/ContextFabricBaselineRunner.cs
• OrchestratorIDE/Services/ContextFabric/ContextFabricContracts.cs
• OrchestratorIDE/Services/ContextFabric/ContextFabricFeasibilityRunner.cs
• OrchestratorIDE/Services/ContextFabric/ContextFabricValidation.cs
• OrchestratorIDE/Services/ContextFabric/DeterministicExpandedFabricCorpus.cs
• OrchestratorIDE/Services/ContextFabric/ExpandedFabricAuthoredQuestionMerger.cs
• OrchestratorIDE/Services/ContextFabric/ExpandedFabricLedgerExport.cs
• OrchestratorIDE/Services/ContextFabric/ExpandedFabricQuestionGenerator.cs
• OrchestratorIDE/Services/ContextFabric/ExpandedFabricQuestionSplitter.cs
• OrchestratorIDE/Services/Swarm/ToolcallerDatasetCapture.cs
• Tools/ContextFabricBench/Program.cs
• Tools/ContextFabricBench/Run-CF7GateExpanded.ps1
• Tools/ToolcallerBench/Program.cs
• Tools/ToolcallerBench/ToolcallerBench.csproj
• Tools/ToolcallerBench/ToolcallerCaptureValidator.cs
• Tools/ToolcallerBench/ToolcallerContracts.cs
• Tools/ToolcallerBench/ToolcallerReportWriter.cs
• docs/CONTEXT_FABRIC_BENCHMARK_MANIFEST.md
• docs/README.md
• docs/THEORC_WARPATH_GAMIFICATION_WHITEPAPER.md
• docs/TOOLCALLER_V0_FROZEN_INVENTORY.md
• training_pit/PLAN_CAPTURE_SCHEMA.md
• training_pit/TOOLCALLER_CAPTURE_SCHEMA.md
• training_pit/schemas/toolcaller_v0_frozen_tools.json

[coderabbitai]

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Use collision-resistant event IDs.

wp_evt_..._0001-style IDs will collide once multiple emitters/workers produce events, which breaks dedupe and badge traceability. Use UUID/ULID (and keep the timestamp as a separate field) instead.

Proposed fix

-  "event_id": "wp_evt_20260703_183012_0001",
+  "event_id": "018f3e8e-7c2b-7f8d-9c6a-1d2f6e7b9a01",
+  "sequence": 1,

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	A Warpath event is an immutable record of something that happened.
	```json
	{
	"event_id": "wp_evt_20260703_183012_0001",
	"schema_version": "warpath-event-v1",
	"occurred_at": "2026-07-03T18:30:12-07:00",
	"workspace_id": "sha256-of-normalized-workspace-root-or-null",
	"run_id": "optional-swarm-or-chat-run-id",
	"event_type": "swarm.run.completed",
	"source_system": "SwarmSession",
	"actor": "system",
	"role": "CODER",
	"model": "qwen2.5-coder:14b",
	"node_id": "optional-hive-node-id",
	"payload": {
	"success": true,
	"files_changed": 3,
	"tests_passed": true,
	"review_verdict": "CLEAN"
	},
	"privacy": {
	"contains_user_content": false,
	"safe_for_share_card": true
	}
	}
	A Warpath event is an immutable record of something that happened.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/THEORC_WARPATH_GAMIFICATION_WHITEPAPER.md` around lines 324 - 349, The
Warpath event identifier format in the event record needs to avoid collisions
across concurrent emitters. Update the event schema/example around the immutable
event object to use a collision-resistant ID in event_id (for example UUID or
ULID) while keeping occurred_at as a separate timestamp field, and adjust any
text or examples that currently imply sequential wp_evt_..._0001-style IDs.
Refer to the event payload shown in the Warpath event section when making the
change.

[coderabbitai]

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Remove cloud-work rewards from the starter badge list.

Cloud Raider rewards cloud-hosted execution even though the spec says Warpath is local-first and has no cloud dependency. That incentive conflicts with the product promise; keep it out of the MVP list or reframe it around local/distributed execution only.

Proposed fix

-| Cloud Raider | Blood | First cloud Warband completes a task |

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	#### HIVE and Warband Badges
	| Badge | Tier | Trigger |
	|---|---|---|
	| Campfire Lit | Bone | HIVE enabled |
	| First Ally | Bone | First node paired |
	| Crowned | Iron | Local machine elected Warchief |
	| Crown Transfer | Blood | Warchief election succeeds after node loss |
	| Three Fires Burning | Blood | 3 nodes online |
	| Warband Deployed | Blood | First headless Warband connected |
	| Cloud Raider | Blood | First cloud Warband completes a task |
	| Fleet Commander | Gold Crown | 5+ nodes paired |
	#### HIVE and Warband Badges
	| Badge | Tier | Trigger |
	|---|---|---|
	| Campfire Lit | Bone | HIVE enabled |
	| First Ally | Bone | First node paired |
	| Crowned | Iron | Local machine elected Warchief |
	| Crown Transfer | Blood | Warchief election succeeds after node loss |
	| Three Fires Burning | Blood | 3 nodes online |
	| Warband Deployed | Blood | First headless Warband connected |
	| Fleet Commander | Gold Crown | 5+ nodes paired |

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/THEORC_WARPATH_GAMIFICATION_WHITEPAPER.md` around lines 699 - 710, The
starter badge list in the HIVE and Warband Badges section includes a
cloud-specific reward that conflicts with the local-first Warpath spec. Remove
the Cloud Raider badge from the MVP badge table, or rename/reframe it so it only
rewards local or distributed execution with no cloud dependency. Keep the
remaining badges in the list unchanged and ensure the badge triggers described
in the whitepaper stay aligned with the product promise.

[coderabbitai]

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Define the canonical Foundry event before depending on it.

foundry.candidate.deployed_artifact_verified is required by the trophy schema, but it never appears in the event inventory. That leaves the validator and badge rules without a single source of truth for the event name.

Proposed fix

+foundry.candidate.deployed_artifact_verified

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/THEORC_WARPATH_GAMIFICATION_WHITEPAPER.md` around lines 774 - 789, The
trophy schema for The Black Anvil depends on
foundry.candidate.deployed_artifact_verified, but that event is not defined
anywhere in the canonical event inventory. Add the missing event definition to
the Foundry event inventory/source-of-truth and ensure the trophy definition in
the trophy schema references that exact event name consistently, so the
validator and badge rules use a single canonical symbol.

[coderabbitai]

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Avoid remote badge URLs in share-card exports.

These badge images make the “local markdown only” export fetch third-party assets and leak run metadata when rendered. Prefer embedded/local assets or a plain-text export path.

Proposed fix

-![TheOrc Rank](https://img.shields.io/badge/TheOrc-Iron%20Warchief-39FF6A)
-![Clean Runs](https://img.shields.io/badge/Clean%20Runs-62-blue)
-![Local AI](https://img.shields.io/badge/Local%20AI-100%25-brightgreen)
+![TheOrc Rank](./warpath-rank-badge.svg)
+![Clean Runs](./warpath-clean-runs-badge.svg)
+![Local AI](./warpath-local-ai-badge.svg)

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	### 13.4 GitHub Badge Export
	Optional future export:
	```md
	
	
	
	```
	### 13.4 GitHub Badge Export
	Optional future export:

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/THEORC_WARPATH_GAMIFICATION_WHITEPAPER.md` around lines 888 - 896, The
GitHub badge export section currently relies on remote shields.io image URLs,
which breaks the local-only export goal and can leak metadata. Update the
“GitHub Badge Export” content to use embedded/local assets or route this example
through a plain-text export path instead of external badge links, keeping the
change localized to the badge export snippet in the whitepaper.

[coderabbitai]

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Test doesn't actually exercise its stated scenario — Tokenize dedupes per segment, so "big" doesn't outscore "small".

Tokenize returns a HashSet<string> per segment (ContextFabricBaselineRunner.cs line 478-483), so repeated occurrences of "checksum" in big don't increase its score — only presence matters. Computing IDF scores here: checksum has document-frequency 2 (df across both segments), noted has df 1 (only in small). That gives small a higher score (0.5 + 1.0 = 1.5) than big (0.5), not lower as the in-line comment claims ("even though 'big' scores higher"). The test still passes because the budget happens to only fit small regardless of ranking, but it never actually exercises the intended case — a higher-ranked-but-oversized segment being skipped in favor of a genuinely lower-ranked-but-smaller one that still fits.

🧪 Suggested fix: make "big" genuinely outrank "small"

-        var big = new FabricSegment("seg-big", 1, "Big",
-            "checksum CK-777 checksum CK-777 checksum CK-777 padding padding padding padding padding padding padding",
-            FabricHashing.Sha256("big"), 500);
-        var small = new FabricSegment("seg-small", 2, "Small", "checksum CK-777 noted briefly.",
-            FabricHashing.Sha256("small"), 15);
+        // "big" must match strictly more distinct question terms than "small" to actually outrank
+        // it under presence/IDF scoring (Tokenize dedupes per segment, so repeated mentions of the
+        // same term do not increase score).
+        var big = new FabricSegment("seg-big", 1, "Big",
+            "checksum CK-777 was noted and verified with padding padding padding padding padding padding padding",
+            FabricHashing.Sha256("big"), 500);
+        var small = new FabricSegment("seg-small", 2, "Small", "checksum CK-777 noted briefly.",
+            FabricHashing.Sha256("small"), 15);

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	[Test]
	public void BuildTopKText_SkipsOverBudgetSegment_ButStillFitsShorterLowerRankedOne()
	{
	// The highest-scoring segment is too large to fit; a shorter, lower-scoring but still
	// relevant segment should still be included rather than leaving the budget unused.
	var big = new FabricSegment("seg-big", 1, "Big",
	"checksum CK-777 checksum CK-777 checksum CK-777 padding padding padding padding padding padding padding",
	FabricHashing.Sha256("big"), 500);
	var small = new FabricSegment("seg-small", 2, "Small", "checksum CK-777 noted briefly.",
	FabricHashing.Sha256("small"), 15);
	var corpus = new FabricCorpus("corpus-5", "doc-5", "gen-5", "digest-5", "1.0", [big, small], 515);
	var question = new FabricBenchmarkQuestion(
	"q-5", FabricQuestionKind.LocalFact, "What checksum was noted?", ["CK-777"], ["seg-small"]);
	var fixture = new FabricBenchmarkFixture(corpus, [question]);
	// AnswerMaxTokens tuned so the effective budget (~50 tokens) fits "small" (15 tokens) but
	// not "big" (500 tokens), even though "big" scores higher (three checksum mentions).
	var runner = new ContextFabricBaselineRunner(new ScriptedFabricRuntime(), Options(answerMaxTokens: 1632));
	var text = runner.BuildTopKText(fixture, question);
	Assert.That(text, Does.Contain("noted briefly"));
	Assert.That(text, Does.Not.Contain("padding"));
	}
	[Test]
	public void BuildTopKText_SkipsOverBudgetSegment_ButStillFitsShorterLowerRankedOne()
	{
	// The highest-scoring segment is too large to fit; a shorter, lower-scoring but still
	// relevant segment should still be included rather than leaving the budget unused.
	// "big" must match strictly more distinct question terms than "small" to actually outrank
	// it under presence/IDF scoring (Tokenize dedupes per segment, so repeated mentions of the
	// same term do not increase score).
	var big = new FabricSegment("seg-big", 1, "Big",
	"checksum CK-777 was noted and verified with padding padding padding padding padding padding padding",
	FabricHashing.Sha256("big"), 500);
	var small = new FabricSegment("seg-small", 2, "Small", "checksum CK-777 noted briefly.",
	FabricHashing.Sha256("small"), 15);
	var corpus = new FabricCorpus("corpus-5", "doc-5", "gen-5", "digest-5", "1.0", [big, small], 515);
	var question = new FabricBenchmarkQuestion(
	"q-5", FabricQuestionKind.LocalFact, "What checksum was noted?", ["CK-777"], ["seg-small"]);
	var fixture = new FabricBenchmarkFixture(corpus, [question]);
	// AnswerMaxTokens tuned so the effective budget (~50 tokens) fits "small" (15 tokens) but
	// not "big" (500 tokens), even though "big" scores higher (three checksum mentions).
	var runner = new ContextFabricBaselineRunner(new ScriptedFabricRuntime(), Options(answerMaxTokens: 1632));
	var text = runner.BuildTopKText(fixture, question);
	Assert.That(text, Does.Contain("noted briefly"));
	Assert.That(text, Does.Not.Contain("padding"));
	}

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@OrchestratorIDE.UnitTests/ContextFabricB2TopKRagTests.cs` around lines 111 -
133, The test in
BuildTopKText_SkipsOverBudgetSegment_ButStillFitsShorterLowerRankedOne does not
match the intended ranking because Tokenize dedupes terms per segment, so
repeated “checksum” in the big segment does not raise its score. Update the test
data so the segment referenced by big genuinely outranks small under the
BuildTopKText scoring logic in ContextFabricBaselineRunner, while still being
too large for the token budget; keep small as the shorter fallback that fits and
is selected when big is skipped.

[coderabbitai]

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

PrintUsage suite list omits export-ledger and merge-authored.

ParseSuite accepts export-ledger, merge-authored, and cf7-gate-expanded (Line 467-470), and the unknown-suite error message lists all three, but the --suite help line at Line 476 only mentions cf0 | quote-anchor | stitch | cf7-gate | cf7-gate-expanded | scale. Users running --help won't discover the two new suites.

📝 Proposed fix

-        Console.WriteLine("  --suite <name>            cf0 | quote-anchor | stitch | cf7-gate | cf7-gate-expanded | scale (default cf0)");
+        Console.WriteLine("  --suite <name>            cf0 | quote-anchor | stitch | cf7-gate | cf7-gate-expanded | scale | export-ledger | merge-authored (default cf0)");

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	"export-ledger" => BenchmarkSuite.ExportLedger,
	"merge-authored" => BenchmarkSuite.MergeAuthored,
	"cf7-gate-expanded" => BenchmarkSuite.Cf7GateExpanded,
	_ => throw new ArgumentException("Unknown suite. Use cf0, quote-anchor, stitch, cf7-gate, scale, export-ledger, merge-authored, or cf7-gate-expanded."),
	};
	private static void PrintUsage()
	{
	Console.WriteLine("Usage: context-fabric-bench --model-root <folder> [options]");
	Console.WriteLine(" --suite <name> cf0 | quote-anchor | stitch | cf7-gate | scale (default cf0)");
	Console.WriteLine(" --suite <name> cf0 | quote-anchor | stitch | cf7-gate | cf7-gate-expanded | scale (default cf0)");
	"export-ledger" => BenchmarkSuite.ExportLedger,
	"merge-authored" => BenchmarkSuite.MergeAuthored,
	"cf7-gate-expanded" => BenchmarkSuite.Cf7GateExpanded,
	_ => throw new ArgumentException("Unknown suite. Use cf0, quote-anchor, stitch, cf7-gate, scale, export-ledger, merge-authored, or cf7-gate-expanded."),
	};
	private static void PrintUsage()
	{
	Console.WriteLine("Usage: context-fabric-bench --model-root <folder> [options]");
	Console.WriteLine(" --suite <name> cf0 | quote-anchor | stitch | cf7-gate | cf7-gate-expanded | scale | export-ledger | merge-authored (default cf0)");

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@Tools/ContextFabricBench/Program.cs` around lines 467 - 476, The `PrintUsage`
help text in `Program.cs` is missing the `export-ledger` and `merge-authored`
suite options even though `ParseSuite` and the unknown-suite message already
support them. Update the `Console.WriteLine` inside `PrintUsage` so the
`--suite` list includes `export-ledger` and `merge-authored` alongside the
existing `BenchmarkSuite` values, keeping the help output aligned with
`ParseSuite`.

[coderabbitai]

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the target region.
nl -ba Tools/ContextFabricBench/Run-CF7GateExpanded.ps1 | sed -n '145,190p'

# Probe PowerShell parsing/runtime behavior for the exact pattern if pwsh is available.
if command -v pwsh >/dev/null 2>&1; then
  pwsh -NoLogo -NoProfile -Command '
    $ErrorActionPreference = "Stop"
    try {
      Write-Error "A" + "B"
      "NO_ERROR"
    } catch {
      "CAUGHT: $($_.FullyQualifiedErrorId)"
      "MESSAGE: $($_.Exception.Message)"
    }
  '
else
  echo "pwsh not available"
fi

Repository: hardcoreerik/TheOrc

Length of output: 197

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

python3 - <<'PY'
from pathlib import Path
path = Path("Tools/ContextFabricBench/Run-CF7GateExpanded.ps1")
start, end = 145, 190
for i, line in enumerate(path.read_text().splitlines(), 1):
    if start <= i <= end:
        print(f"{i:4}: {line}")
PY

if command -v pwsh >/dev/null 2>&1; then
  pwsh -NoLogo -NoProfile -Command '
    $ErrorActionPreference = "Stop"
    foreach ($expr in @(
      { Write-Error "A" + "B" },
      { $x = "A" + "B"; Write-Error $x },
      { Write-Error ("A" + "B") }
    )) {
      try {
        & $expr
        "NO_ERROR"
      } catch {
        "CAUGHT: $($_.FullyQualifiedErrorId)"
        "MESSAGE: $($_.Exception.Message)"
      }
      "---"
    }
  '
else
  echo "pwsh not available"
fi

Repository: hardcoreerik/TheOrc

Length of output: 2576

---

Parenthesize the Write-Error messages Write-Error "..." + "..." is parsed as separate arguments in PowerShell, so these pre-flight checks fail with a positional-parameter error instead of showing the intended message. Wrap each concatenation in parentheses (or assign the string first) at all four call sites.

🧰 Tools

🪛 PSScriptAnalyzer (1.25.0)

[warning] Missing BOM encoding for non-ASCII encoded file 'Run-CF7GateExpanded.ps1'

(PSUseBOMForUnicodeEncodedFile)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@Tools/ContextFabricBench/Run-CF7GateExpanded.ps1` around lines 152 - 181, The
pre-flight checks in Run-CF7GateExpanded.ps1 are using Write-Error with string
concatenation directly, which PowerShell parses as separate arguments and causes
a positional-parameter error. Update each of the four Write-Error call sites in
the held-out, B4 artifact, ModelRoot, and GgufCount checks so the full message
is parenthesized or built first in a variable before passing it to Write-Error.
Use the existing guard blocks and identifiers like $HeldOutPath, $B4Artifact,
$ModelRoot, and $GgufCount to locate them.

[coderabbitai]

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Unrecognized expected.decision values pass validation silently.

The gate logic only branches on Decision == "call" (Line 72) vs an implicit "everything else" (Line 121), and only checks reason_code for "clarify"/"unsupported" (Line 65). A capture with a typo'd or invalid decision string (e.g. "cal", "unknown") falls through both branches without any failure — it's silently treated like a valid no_tool-style decision and can be marked Passed. Since this tool's entire purpose is mechanical admission gating, an unrecognized decision value should be a hard failure.

Proposed fix: validate decision against a known set

+        var knownDecisions = new HashSet<string>(StringComparer.Ordinal) { "call", "no_tool", "clarify", "unsupported" };
+
         foreach (var capture in captures)
         {
+            if (!knownDecisions.Contains(capture.Expected.Decision))
+            {
+                Fail(capture, "unknown_decision", $"expected.decision '{capture.Expected.Decision}' is not a recognized value.");
+            }
+
             // Gate: stale schema hash — ...

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	// Gate: reason_code required for clarify/unsupported.
	var needsReasonCode = capture.Expected.Decision is "clarify" or "unsupported";
	if (needsReasonCode && string.IsNullOrWhiteSpace(capture.Expected.ReasonCode))
	{
	Fail(capture, "missing_reason_code",
	$"Decision '{capture.Expected.Decision}' requires a non-null reason_code.");
	}
	if (capture.Expected.Decision == "call")
	{
	// Gate: call examples must name a tool.
	if (string.IsNullOrWhiteSpace(capture.Expected.Tool))
	{
	Fail(capture, "call_missing_tool", "Decision 'call' requires expected.tool.");
	}
	else
	{
	// Gate: target tool must exist in the frozen universe.
	if (!toolsByName.TryGetValue(capture.Expected.Tool, out var tool))
	{
	Fail(capture, "tool_outside_frozen_universe",
	$"expected.tool '{capture.Expected.Tool}' is not in the frozen v0 tool set.");
	}
	else
	{
	// Gate: target tool must be in this example's own available_tools.
	if (!capture.AvailableTools.Contains(capture.Expected.Tool, StringComparer.Ordinal))
	{
	Fail(capture, "tool_outside_available_tools",
	$"expected.tool '{capture.Expected.Tool}' is not in this example's available_tools.");
	}
	// Gate: no invented arguments, no missing required arguments.
	var arguments = capture.Expected.Arguments ?? new Dictionary<string, System.Text.Json.JsonElement>();
	var invented = arguments.Keys.Where(k => !tool.Parameters.ContainsKey(k)).ToArray();
	if (invented.Length > 0)
	{
	Fail(capture, "invented_argument",
	$"Argument(s) not in {tool.Name}'s frozen schema: {string.Join(", ", invented)}.");
	}
	var missingRequired = tool.Required.Where(r => !arguments.ContainsKey(r)).ToArray();
	if (missingRequired.Length > 0)
	{
	Fail(capture, "missing_required_argument",
	$"{tool.Name} requires argument(s) not present: {string.Join(", ", missingRequired)}.");
	}
	}
	}
	// Gate: a proposed call must have policy_outcome evaluated.
	if (capture.PolicyOutcome is null || !capture.PolicyOutcome.Evaluated)
	{
	Fail(capture, "call_missing_policy_outcome",
	"Decision 'call' requires policy_outcome.evaluated == true.");
	}
	}
	else
	{
	// Non-call decisions should not carry an evaluated policy outcome —
	// there is no proposed call to evaluate against ToolPolicyEngine.
	if (capture.PolicyOutcome is { Evaluated: true })
	{
	Info(capture, "policy_outcome_evaluated_without_call",
	$"Decision '{capture.Expected.Decision}' has policy_outcome.evaluated == true; " +
	"expected only for 'call' decisions.");
	}
	}
	// Gate: reason_code required for clarify/unsupported.
	var needsReasonCode = capture.Expected.Decision is "clarify" or "unsupported";
	if (needsReasonCode && string.IsNullOrWhiteSpace(capture.Expected.ReasonCode))
	{
	Fail(capture, "missing_reason_code",
	$"Decision '{capture.Expected.Decision}' requires a non-null reason_code.");
	}
	if (capture.Expected.Decision == "call")
	{
	// Gate: call examples must name a tool.
	if (string.IsNullOrWhiteSpace(capture.Expected.Tool))
	{
	Fail(capture, "call_missing_tool", "Decision 'call' requires expected.tool.");
	}
	else
	{
	// Gate: target tool must exist in the frozen universe.
	if (!toolsByName.TryGetValue(capture.Expected.Tool, out var tool))
	{
	Fail(capture, "tool_outside_frozen_universe",
	$"expected.tool '{capture.Expected.Tool}' is not in the frozen v0 tool set.");
	}
	else
	{
	// Gate: target tool must be in this example's own available_tools.
	if (!capture.AvailableTools.Contains(capture.Expected.Tool, StringComparer.Ordinal))
	{
	Fail(capture, "tool_outside_available_tools",
	$"expected.tool '{capture.Expected.Tool}' is not in this example's available_tools.");
	}
	// Gate: no invented arguments, no missing required arguments.
	var arguments = capture.Expected.Arguments ?? new Dictionary<string, System.Text.Json.JsonElement>();
	var invented = arguments.Keys.Where(k => !tool.Parameters.ContainsKey(k)).ToArray();
	if (invented.Length > 0)
	{
	Fail(capture, "invented_argument",
	$"Argument(s) not in {tool.Name}'s frozen schema: {string.Join(", ", invented)}.");
	}
	var missingRequired = tool.Required.Where(r => !arguments.ContainsKey(r)).ToArray();
	if (missingRequired.Length > 0)
	{
	Fail(capture, "missing_required_argument",
	$"{tool.Name} requires argument(s) not present: {string.Join(", ", missingRequired)}.");
	}
	}
	}
	// Gate: a proposed call must have policy_outcome evaluated.
	if (capture.PolicyOutcome is null || !capture.PolicyOutcome.Evaluated)
	{
	Fail(capture, "call_missing_policy_outcome",
	"Decision 'call' requires policy_outcome.evaluated == true.");
	}
	}
	else
	{
	// Non-call decisions should not carry an evaluated policy outcome —
	// there is no proposed call to evaluate against ToolPolicyEngine.
	if (capture.PolicyOutcome is { Evaluated: true })
	{
	Info(capture, "policy_outcome_evaluated_without_call",
	$"Decision '{capture.Expected.Decision}' has policy_outcome.evaluated == true; " +
	"expected only for 'call' decisions.");
	}
	}

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@Tools/ToolcallerBench/ToolcallerCaptureValidator.cs` around lines 64 - 131,
Add explicit validation in ToolcallerCaptureValidator so unexpected
capture.Expected.Decision values fail fast instead of falling through the
non-call branch. Introduce a known allowed set in the validator around the
existing decision gates (the clarify/unsupported reason_code check and the
call/non-call split), and if Decision is anything other than the recognized
values, call Fail with a clear invalid_decision error before any other checks.

[coderabbitai]

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🌐 Web query:

Does System.Text.Json enforce non-nullable reference type properties on deserialization by default in .NET 10?

💡 Result:

No, System.Text.Json does not enforce non-nullable reference type properties by default in.NET 10 [1][2]. While support for respecting nullability annotations was introduced in.NET 9, it is an opt-in feature to maintain backward compatibility with existing applications [3][4]. To enable this behavior, you must explicitly set the RespectNullableAnnotations property to true in your JsonSerializerOptions [3][1]: JsonSerializerOptions options = new { RespectNullableAnnotations = true }; JsonSerializer.Deserialize(json, options); You can also enable this globally for an application using the System.Text.Json.Serialization.RespectNullableAnnotationsDefault feature switch in your project file [1][2]. Important limitations to consider: - Enforcement is restricted to non-generic properties, fields, and constructor parameters due to how nullable reference types are represented at runtime [3][1]. - It does not apply to top-level types, collection elements, or generic parameters [5][1]. - Nullability validation is independent of "required-ness" validation; to ensure properties are actually present in the JSON payload, you should use the required keyword or enable the RespectRequiredConstructorParameters option [1][6]. Microsoft recommends that new applications enable both RespectNullableAnnotations and RespectRequiredConstructorParameters to ensure stricter data validation [1][6].

Citations:

• 1: https://learn.microsoft.com/en-us/dotnet/api/system.text.json.jsonserializeroptions.respectnullableannotations?view=net-9.0
• 2: https://learn.microsoft.com/uk-ua/dotnet/api/system.text.json.jsonserializeroptions.respectnullableannotations?view=net-10.0&viewFallbackFrom=net-9.0-pp
• 3: https://devblogs.microsoft.com/dotnet/system-text-json-in-dotnet-9/
• 4: Add a STJ feature flag recognizing nullable reference type annotations on properties dotnet/runtime#100144
• 5: https://learn.microsoft.com/en-us/dotnet/standard/serialization/system-text-json/nullable-annotations
• 6: https://learn.microsoft.com/en-us/dotnet/standard/serialization/system-text-json/required-properties

---

Mark critical capture fields as required Non-nullable ToolcallerCapture members can still deserialize as null when omitted unless required or stricter System.Text.Json options are enabled. That lets malformed capture JSON slip past the admission gate and fail later in validation or downstream access.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@Tools/ToolcallerBench/ToolcallerContracts.cs` around lines 26 - 43, The
ToolcallerCapture contract currently relies on non-nullable types alone, which
still allows omitted JSON fields to deserialize as null; update the
ToolcallerCapture record so the critical members are explicitly required (or
otherwise enforced via stricter deserialization settings) to reject malformed
capture payloads at admission. Focus on the ToolcallerCapture record and its
JsonPropertyName-backed members, especially the non-optional fields like
SchemaVersion, ToolSchemaHash, ExampleId, LineageGroupId, Provenance, Role,
Request, AvailableTools, ApprovalState, Expected, ReviewStatus, and Split.

[coderabbitai]

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Confirm no other code path appends a "toolcaller" subfolder before writing captures
rg -n 'DatasetStagingDir|dataset-staging' OrchestratorIDE/Agents/SwarmSession.cs OrchestratorIDE/Services/Swarm/ToolcallerDatasetCapture.cs OrchestratorIDE/Services/Swarm/DatasetCapture.cs

Repository: hardcoreerik/TheOrc

Length of output: 856

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n== SwarmSession relevant lines ==\n'
sed -n '115,140p;2040,2080p' OrchestratorIDE/Agents/SwarmSession.cs

printf '\n== DatasetCapture header ==\n'
sed -n '1,40p' OrchestratorIDE/Services/Swarm/DatasetCapture.cs

printf '\n== Toolcaller capture references ==\n'
rg -n 'StageCallAsync|StageNoToolAsync|toolcaller_capture_|toolcaller' OrchestratorIDE/Services/Swarm/ToolcallerDatasetCapture.cs OrchestratorIDE/Agents/SwarmSession.cs

Repository: hardcoreerik/TheOrc

Length of output: 6673

---

Align the documented staging path with the code — toolcaller captures go to .orc/swarm/dataset-staging/ and are distinguished by the toolcaller_capture_ filename prefix, not a toolcaller/ subdirectory.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@training_pit/TOOLCALLER_CAPTURE_SCHEMA.md` around lines 4 - 9, The documented
toolcaller staging location is incorrect and should match the implementation in
ToolcallerDatasetCapture and RunWorkerLoopAsync. Update the description to say
captures are written under .orc/swarm/dataset-staging/ and identified by the
toolcaller_capture_ filename prefix, rather than implying a toolcaller/
subdirectory. Keep the wording aligned with the actual capture flow and naming
used by the dataset staging code.

[hardcoreerik]

Reviewer pass

Strong PR overall. I like the honesty here: the previous CF-7 “GO” was overclaimed, the new expanded corpus/held-out suite exposes that, and the PR reports the real result as NO-GO instead of trying to dress it up. That is exactly the kind of truth discipline this project needs.

What looks good

• The evidence-selection diagnosis is believable: failing questions often never included the answer-bearing segments, which points at retrieval/evidence-pack selection rather than answer synthesis.
• Replacing fixed 1/2/4-card caps with IDF-weighted, stopword-aware, budget-fill selection is the right class of fix.
• The unit tests for rare-term preference, selecting more than four relevant cards, empty/budget-exhausted fallbacks, and shorter lower-ranked fallback are meaningful, not just compile theater.
• The B2 baseline fix is also good. A baseline that performs worse than truncated prompt because of naive top-k selection is not a fair baseline.
• The PR is admirably honest that Exhaustive questions still have a separate root cause and are out of scope.
• The Toolcaller F-1 direction is valuable: frozen tool inventory, hash, mechanical validator, live capture hook, and pending/reviewed split discipline are the right foundation.

Main concern before merge

I would not treat this as fully merge-ready until the capture behavior is clarified:

ToolcallerDatasetCapture.IsEnabled defaults to true, and the hook silently stages live swarm requests/tool arguments. Even if this is local-only and pending review, those captures can contain real repo content, paths, shell commands, file contents, or accidental secrets. For a local-first/privacy-first project, organic training capture should probably be one of:

1. opt-in by default, or
2. controlled by an explicit user/dev setting, or
3. very clearly documented with the staging path, gitignore/sanitizer flow, and “do not share raw captures” warning.

The docs already say captures need sanitizer + human review, which is good. I’d still prefer the runtime hook default to off unless the operator intentionally enables Foundry capture.

Merge caveats

• CodeRabbit is still processing/pending, so wait for that result.
• The PR itself says the full 120-question rerun with both fixes has not happened yet. That is fine for this PR if the goal is “land remediation code + harness,” but do not update public claims as if the fix improved the score until that rerun exists.
• Keep old/pre-fix vs new/post-fix benchmark numbers clearly labeled. This PR changes B2 and B3 selection behavior, so comparisons need timestamps/commit SHAs.
• B4 is explicitly frozen prior HIVE evidence, not revalidated against the expanded corpus. That disclosure is correct and should remain visible in reports.

Verdict

Good direction, good honesty, good tests. I would call this approve-after-capture-default/visibility is resolved and CodeRabbit completes, with the full 120-question rerun as the next required proof artifact, not a hidden assumption.

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@OrchestratorIDE.UnitTests/ContextFabricExhaustiveAnswerTests.cs`:
- Around line 79-88: The exhaustive-answer test setup is inconsistent with the
“all entries” expectation, since FabricBenchmarkQuestion is only initialized
with CASE-01-0 and segmentIds[0] while the assertion expects all five segments.
Update the question fixture in ContextFabricExhaustiveAnswerTests to use
expectations that match the full ledger-case scenario, and keep the
BuildExhaustiveAnswer assertion aligned so the test verifies the intended
all-entries behavior rather than a partial case.

In `@OrchestratorIDE/Services/ContextFabric/ContextFabricFeasibilityRunner.cs`:
- Around line 996-1005: Add “under” to the _scoringStopwords set in
ContextFabricFeasibilityRunner so it is ignored by mostDistinctiveTerms. Update
the stopword list where the existing terms are defined to prevent “under” from
being selected as a distinctive term and causing ledger cards to be missed.
- Around line 771-785: The selection logic in ContextFabricFeasibilityRunner
currently drops all but one matching claim per card because the projection uses
FirstOrDefault() after filtering claims. Update the selected pipeline so it
returns every claim that matches the distinctive-term filter, preserving the
existing card ordering by corpus.Segments ordinal and the ScoreTextIdf ranking
only where needed, and use the unique symbols cards, Claim, TokenizeForScoring,
ScoreTextIdf, and selected to locate the query.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 836ccc3f-aa02-4b4c-8a8c-eb23ecc2b6da

📥 Commits

Reviewing files that changed from the base of the PR and between c68e01c and 3ef5fb0.

📒 Files selected for processing (2)

• OrchestratorIDE.UnitTests/ContextFabricExhaustiveAnswerTests.cs
• OrchestratorIDE/Services/ContextFabric/ContextFabricFeasibilityRunner.cs

[coderabbitai]

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Align the benchmark question expectations with the “all entries” scenario.

This test says all five ledger entries should be included, but the question only declares CASE-01-0 and segmentIds[0] as expected. That can mask verifier regressions and makes the fixture contradict the behavior under test.

Proposed test tightening

+        var expectedAnswers = Enumerable.Range(0, 5)
+            .Select(i => $"CASE-01-{i}")
+            .ToArray();
         var question = new FabricBenchmarkQuestion(
             "q-2", FabricQuestionKind.Exhaustive,
             "List every case-file ID recorded under ledger case-ledger-01, in any order.",
-            ["CASE-01-0"], [segmentIds[0]]);
+            expectedAnswers, segmentIds);
 
         var runner = new ContextFabricFeasibilityRunner(new ScriptedFabricRuntime(), FabricRunOptions.Default);
         var result = runner.BuildExhaustiveAnswer(corpus, question, cards);
 
-        Assert.That(result.IncludedSegmentIds, Has.Count.EqualTo(5));
+        Assert.Multiple(() =>
+        {
+            Assert.That(result.IncludedSegmentIds, Is.EquivalentTo(segmentIds));
+            foreach (var expected in expectedAnswers)
+                Assert.That(result.Answer?.Answer, Does.Contain(expected));
+        });

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	var question = new FabricBenchmarkQuestion(
	"q-2", FabricQuestionKind.Exhaustive,
	"List every case-file ID recorded under ledger case-ledger-01, in any order.",
	["CASE-01-0"], [segmentIds[0]]);
	var runner = new ContextFabricFeasibilityRunner(new ScriptedFabricRuntime(), FabricRunOptions.Default);
	var result = runner.BuildExhaustiveAnswer(corpus, question, cards);
	Assert.That(result.IncludedSegmentIds, Has.Count.EqualTo(5));
	}
	var expectedAnswers = Enumerable.Range(0, 5)
	.Select(i => $"CASE-01-{i}")
	.ToArray();
	var question = new FabricBenchmarkQuestion(
	"q-2", FabricQuestionKind.Exhaustive,
	"List every case-file ID recorded under ledger case-ledger-01, in any order.",
	expectedAnswers, segmentIds);
	var runner = new ContextFabricFeasibilityRunner(new ScriptedFabricRuntime(), FabricRunOptions.Default);
	var result = runner.BuildExhaustiveAnswer(corpus, question, cards);
	Assert.Multiple(() =>
	{
	Assert.That(result.IncludedSegmentIds, Is.EquivalentTo(segmentIds));
	foreach (var expected in expectedAnswers)
	Assert.That(result.Answer?.Answer, Does.Contain(expected));
	});

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@OrchestratorIDE.UnitTests/ContextFabricExhaustiveAnswerTests.cs` around lines
79 - 88, The exhaustive-answer test setup is inconsistent with the “all entries”
expectation, since FabricBenchmarkQuestion is only initialized with CASE-01-0
and segmentIds[0] while the assertion expects all five segments. Update the
question fixture in ContextFabricExhaustiveAnswerTests to use expectations that
match the full ledger-case scenario, and keep the BuildExhaustiveAnswer
assertion aligned so the test verifies the intended all-entries behavior rather
than a partial case.

[coderabbitai]

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Add under to scoring stopwords to avoid rare-term misselection.

The exhaustive questions use “under ledger ...”. If any unrelated card contains under once, it becomes rarer than 01, so mostDistinctiveTerms can become {"under"} and exclude the actual ledger cards.

Proposed fix

-        "every", "list", "order",
+        "every", "list", "order", "under",

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	private static readonly HashSet<string> _scoringStopwords = new(StringComparer.Ordinal)
	{
	var haystack = string.Join(' ', card.Claims.Select(claim => claim.Text).Prepend(card.Summary));
	return Tokenize(haystack).Count(terms.Contains);
	"the", "and", "for", "are", "was", "were", "this", "that", "these", "those", "with",
	"from", "into", "onto", "than", "then", "there", "here", "when", "where", "what",
	"which", "who", "whom", "whose", "why", "how", "not", "nor", "but", "does", "did",
	"has", "have", "had", "will", "would", "should", "can", "could", "may", "might",
	"shall", "must", "its", "his", "her", "their", "our", "your", "you", "she", "him",
	"they", "them", "been", "being", "any", "all", "each", "some", "such", "own", "same",
	"is", "at", "to", "of", "in", "on", "by", "no", "an", "we", "it", "as", "or", "be", "do",
	"every", "list", "order",
	private static readonly HashSet<string> _scoringStopwords = new(StringComparer.Ordinal)
	{
	"the", "and", "for", "are", "was", "were", "this", "that", "these", "those", "with",
	"from", "into", "onto", "than", "then", "there", "here", "when", "where", "what",
	"which", "who", "whom", "whose", "why", "how", "not", "nor", "but", "does", "did",
	"has", "have", "had", "will", "would", "should", "can", "could", "may", "might",
	"shall", "must", "its", "his", "her", "their", "our", "your", "you", "she", "him",
	"they", "them", "been", "being", "any", "all", "each", "some", "such", "own", "same",
	"is", "at", "to", "of", "in", "on", "by", "no", "an", "we", "it", "as", "or", "be", "do",
	"every", "list", "order", "under",

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@OrchestratorIDE/Services/ContextFabric/ContextFabricFeasibilityRunner.cs`
around lines 996 - 1005, Add “under” to the _scoringStopwords set in
ContextFabricFeasibilityRunner so it is ignored by mostDistinctiveTerms. Update
the stopword list where the existing terms are defined to prevent “under” from
being selected as a distinctive term and causing ledger cards to be missed.

[hardcoreerik]

Addressed in 2e18805: ToolcallerDatasetCapture.IsEnabled now defaults to off and is driven by a new opt-in toggle (Settings → "Foundry F-1 dataset capture"). While on, a "Dataset Gathering Active" pill shows in the status bar (click opens Settings) — capture is never silent.

Confirmed .orc/swarm/dataset-staging/ is already gitignored, but that alone didn't address the review's point (visibility/consent, not just avoiding a git commit), so this went with option 2 from the comment (explicit user setting) plus the visible indicator from option 3, rather than relying on gitignore. The doc comments on the setting and on ToolcallerDatasetCapture itself carry the "not sanitized — don't share raw captures" warning.

Full test suite still green (470/0/4). Still waiting on the full 120-question re-run before claiming any benchmark improvement — not done yet.