Toolcaller v1: capture organic OrcChat decisions - #45
Merged
Conversation
…sion)
Widens toolcaller data collection beyond Swarm to OrcChat (single-agent
chat) -- a materially higher-volume everyday usage surface that was
capturing nothing. Deliberate v1 revision, not an edit to v0: v0's own doc
anticipated this ("treat that as a new frozen-inventory revision with its
own hash, not a silent addition to v0") and nothing about v0's schema,
hash, datasets, or shipped model (theorc-toolcaller:qwen25-1.5b, r3) is
touched.
- training_pit/schemas/toolcaller_v1_frozen_tools.json: v0's 6 tools + 10
OrcChat tools (web_search, fetch_page, fetch_url, get_outline, 4 library
tools, run_tests, save_markdown_document), each pulled from its live
ToolDefinition registration, not hand-typed. 16 tools, hash 58a0e50d...
- docs/TOOLCALLER_V1_FROZEN_INVENTORY.md: the v1 decision record, mirroring
v0's doc shape (motivation, decision, per-tool verification, new "chat"
role token, capture-only posture)
- ChatEngine: new OnToolcallerDecision event, fired once per turn at the
model's FIRST decision (before any tool-loop iteration) -- the "single
bounded request -> decision" shape the schema wants. Engine itself stays
workspace/staging-agnostic by design; no default subscriber, so this
cannot change existing behavior (27 pre-existing ChatEngineTests still
pass unmodified)
- ToolcallerDatasetCapture.StageChatDecisionAsync: v1 sibling to the v0
StageCallAsync/StageNoToolAsync pair, own schema_version/hash/role
("chat")/staging directory (.orc/chat/dataset-staging, "toolcaller_v1_
chat_" filename prefix) so v0 and v1 can never be mixed by an exporter
that doesn't check schema_version
- ChatPanel subscribes and owns the opt-in side effect (same
AppSettings.ToolcallerDatasetCaptureEnabled flag as swarm capture -- one
toggle, two sources), matching SwarmSession's ownership of the v0 capture
10 new unit tests (6 capture-shape, 3 ChatEngine event-firing, 1
v0/v1-never-mixed). Smoke-tested live: sent a real OrcChat message,
verified an actual toolcaller-v1 capture landed on disk with correct role,
available_tools (the 14 tools OrcChat actually offers -- no run_shell/
ask_user, matching how v0 role subsets already work), and decision.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
📝 WalkthroughWalkthroughAdds a frozen 16-tool ChangesToolcaller v1 chat capture
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant ChatPanel
participant ChatEngine
participant ModelRuntime
participant ToolcallerDatasetCapture
participant Workspace
ChatPanel->>ChatEngine: subscribe OnToolcallerDecision
ChatEngine->>ModelRuntime: request first completion
ModelRuntime-->>ChatEngine: native/ReAct/no-tool decision
ChatEngine->>ChatPanel: ChatToolDecision
ChatPanel->>ToolcallerDatasetCapture: StageChatDecisionAsync
ToolcallerDatasetCapture->>Workspace: write toolcaller_v1_chat JSON
ChatPanel->>ChatEngine: unsubscribe after send
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/TOOLCALLER_V1_FROZEN_INVENTORY.md`:
- Around line 74-77: Update the OrcChat `chat` role description in the frozen
inventory to distinguish the 16-tool frozen v1 universe from the chat-eligible
subset: explicitly exclude `run_shell` and `ask_user`, and state that chat
captures expose only the remaining eligible tools. Preserve the role token and
clarify that `available_tools` reflects actual chat availability rather than the
full universe.
In `@OrchestratorIDE/Services/Swarm/ToolcallerDatasetCapture.cs`:
- Around line 239-248: Update the call-processing loop around BuildChatCapture
to skip any call whose name is absent from availableTools before allocating an
example ID, evaluating policy, or staging the capture. Preserve staging for
known tools, and update the multiple-calls test fixture to provide the tools
referenced by its calls instead of an empty available-tool list.
In `@training_pit/TOOLCALLER_CAPTURE_SCHEMA.md`:
- Around line 18-19: Update the documented differences in
TOOLCALLER_CAPTURE_SCHEMA.md to include provenance.source_type, specifying
chat_capture for chat and swarm_capture for swarm, while preserving the existing
four listed differences.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: d6ab0c06-740f-429d-9045-4a62bb7ec72e
📒 Files selected for processing (8)
OrchestratorIDE.Avalonia/UI/Panels/ChatPanel.axaml.csOrchestratorIDE.UnitTests/ChatEngineTests.csOrchestratorIDE.UnitTests/ToolcallerDatasetCaptureTests.csOrchestratorIDE/Research/ChatEngine.csOrchestratorIDE/Services/Swarm/ToolcallerDatasetCapture.csdocs/TOOLCALLER_V1_FROZEN_INVENTORY.mdtraining_pit/TOOLCALLER_CAPTURE_SCHEMA.mdtraining_pit/schemas/toolcaller_v1_frozen_tools.json
Comment on lines
+74
to
+77
| OrcChat has no `SwarmWorkerRole` equivalent — it's single-agent, not a swarm worker with | ||
| a role-scoped tool subset. v1 introduces exactly one new role token: **`chat`**, whose | ||
| `available_tools` is the full 16-tool set (OrcChat offers all of them to every | ||
| conversation; there is no per-role tool restriction in chat mode). |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Correct the chat available-tools contract.
This says every chat capture offers all 16 tools, but Lines 35-36 explicitly exclude run_shell and ask_user from OrcChat. Describe 16 as the frozen v1 universe and chat captures as the actual chat-eligible subset; otherwise downstream consumers may validate or train against tools OrcChat never offered.
Proposed fix
-OrcChat has no `SwarmWorkerRole` equivalent — it's single-agent, not a swarm worker with
-a role-scoped tool subset. v1 introduces exactly one new role token: **`chat`**, whose
-`available_tools` is the full 16-tool set (OrcChat offers all of them to every
-conversation; there is no per-role tool restriction in chat mode).
+OrcChat has no `SwarmWorkerRole` equivalent — it's single-agent, not a swarm worker with
+a role-scoped tool subset. v1 introduces exactly one new role token: **`chat`**. The
+frozen v1 universe contains all 16 tools, while OrcChat captures list the chat-eligible
+tools actually offered for that decision (excluding `run_shell` and `ask_user`).📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| OrcChat has no `SwarmWorkerRole` equivalent — it's single-agent, not a swarm worker with | |
| a role-scoped tool subset. v1 introduces exactly one new role token: **`chat`**, whose | |
| `available_tools` is the full 16-tool set (OrcChat offers all of them to every | |
| conversation; there is no per-role tool restriction in chat mode). | |
| OrcChat has no `SwarmWorkerRole` equivalent — it's single-agent, not a swarm worker with | |
| a role-scoped tool subset. v1 introduces exactly one new role token: **`chat`**. The | |
| frozen v1 universe contains all 16 tools, while OrcChat captures list the chat-eligible | |
| tools actually offered for that decision (excluding `run_shell` and `ask_user`). |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/TOOLCALLER_V1_FROZEN_INVENTORY.md` around lines 74 - 77, Update the
OrcChat `chat` role description in the frozen inventory to distinguish the
16-tool frozen v1 universe from the chat-eligible subset: explicitly exclude
`run_shell` and `ask_user`, and state that chat captures expose only the
remaining eligible tools. Preserve the role token and clarify that
`available_tools` reflects actual chat availability rather than the full
universe.
Comment on lines
+239
to
+248
| foreach (var call in calls) | ||
| { | ||
| var exampleId = NextChatExampleId(runId); | ||
| var policy = workspaceRoot is not null | ||
| ? ToolPolicyEngine.Evaluate(call.Name, call.Arguments, workspaceRoot) | ||
| : null; | ||
| await WriteChatAsync(BuildChatCapture( | ||
| exampleId, model, request, availableTools, | ||
| decision: "call", tool: call.Name, arguments: call.Arguments, policy: policy), | ||
| exampleId, stagingDir); |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Do not stage calls absent from availableTools.
A model can propose an unknown tool; ChatEngine.ExecuteTool handles that as Unknown tool, but this loop records it as a valid "call" label. That produces examples whose expected tool is not in available_tools or the frozen inventory. Skip such calls (and update the multiple-calls test, which currently supplies an empty available-tool list).
Proposed fix
foreach (var call in calls)
{
+ if (!availableTools.Any(t =>
+ t.Name.Equals(call.Name, StringComparison.OrdinalIgnoreCase)))
+ continue;
+
var exampleId = NextChatExampleId(runId);📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| foreach (var call in calls) | |
| { | |
| var exampleId = NextChatExampleId(runId); | |
| var policy = workspaceRoot is not null | |
| ? ToolPolicyEngine.Evaluate(call.Name, call.Arguments, workspaceRoot) | |
| : null; | |
| await WriteChatAsync(BuildChatCapture( | |
| exampleId, model, request, availableTools, | |
| decision: "call", tool: call.Name, arguments: call.Arguments, policy: policy), | |
| exampleId, stagingDir); | |
| foreach (var call in calls) | |
| { | |
| if (!availableTools.Any(t => | |
| t.Name.Equals(call.Name, StringComparison.OrdinalIgnoreCase))) | |
| continue; | |
| var exampleId = NextChatExampleId(runId); | |
| var policy = workspaceRoot is not null | |
| ? ToolPolicyEngine.Evaluate(call.Name, call.Arguments, workspaceRoot) | |
| : null; | |
| await WriteChatAsync(BuildChatCapture( | |
| exampleId, model, request, availableTools, | |
| decision: "call", tool: call.Name, arguments: call.Arguments, policy: policy), | |
| exampleId, stagingDir); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@OrchestratorIDE/Services/Swarm/ToolcallerDatasetCapture.cs` around lines 239
- 248, Update the call-processing loop around BuildChatCapture to skip any call
whose name is absent from availableTools before allocating an example ID,
evaluating policy, or staging the capture. Preserve staging for known tools, and
update the multiple-calls test fixture to provide the tools referenced by its
calls instead of an empty available-tool list.
Comment on lines
+18
to
+19
| > shape below is identical for both; only `schema_version`, `tool_schema_hash`, `role`, | ||
| > and `approval_state` (`"n/a"` for chat — no swarm-style approval gate) differ. |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Include provenance.source_type in the documented differences.
The streams also differ at provenance.source_type (chat_capture vs. swarm_capture), so “only” the listed four fields differ is inaccurate for consumers validating literal capture values.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@training_pit/TOOLCALLER_CAPTURE_SCHEMA.md` around lines 18 - 19, Update the
documented differences in TOOLCALLER_CAPTURE_SCHEMA.md to include
provenance.source_type, specifying chat_capture for chat and swarm_capture for
swarm, while preserving the existing four listed differences.
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Widens toolcaller data collection to OrcChat (single-agent chat), which was capturing nothing despite likely being the higher-volume everyday usage surface compared to Swarm.
Why v1, not an edit to v0
v0's own frozen-inventory doc anticipated this exact expansion: "If a later Foundry phase wants toolcaller coverage for [tools outside the six], treat that as a new frozen-inventory revision with its own hash, not a silent addition to v0." This PR is that revision — additive only.
theorc-toolcaller:qwen25-1.5b(r3), the sealed Arena/Gauntlet eval sets, and v0's frozen tool schema/hash are all untouched.What's new
training_pit/schemas/toolcaller_v1_frozen_tools.json— v0's 6 tools + 10 OrcChat tools (web_search, fetch_page, fetch_url, get_outline, 4 library tools, run_tests, save_markdown_document). Every definition pulled from its liveToolDefinitionregistration, not hand-typed. 16 tools total, hash58a0e50d....docs/TOOLCALLER_V1_FROZEN_INVENTORY.md— the decision record, same shape as v0's doc: motivation, per-tool verification table, the new"chat"role token (OrcChat has noSwarmWorkerRoleequivalent), capture-only posture (no training authorized by this doc).ChatEngine.OnToolcallerDecision— new event, fires once per turn at the model's first decision (before any tool-loop iteration) — the "single bounded request → decision" shape the schema wants.ChatEnginestays workspace/staging-agnostic by design (its own doc comment says it should be reusable across chat surfaces); no default subscriber, so nothing about existing behavior changes (all 27 pre-existingChatEngineTestspass unmodified).ToolcallerDatasetCapture.StageChatDecisionAsync— v1 sibling to the existing v0StageCallAsync/StageNoToolAsync. Ownschema_version, hash, role, and staging directory (.orc/chat/dataset-staging/,toolcaller_v1_chat_filename prefix) — v0 and v1 can never be mixed by an exporter that doesn't checkschema_version.ChatPanelsubscribes and owns turning the event into a capture — sameAppSettings.ToolcallerDatasetCaptureEnabledopt-in flag as the swarm stream (one toggle, two sources), matchingSwarmSession's ownership of v0 capture.Testing
StageChatDecisionAsync's capture shape (including one asserting v0/v1 filenames can never collide), 3 onChatEngine.OnToolcallerDecisionfiring correctly (no-tools-offered → no fire; tools offered + no call →no_tool; native tool call → fires before dispatch).toolcaller-v1capture landed on disk with correctrole: "chat",available_tools(the 14 tools OrcChat actually offers — norun_shell/ask_user, mirroring how v0's per-role subsets already work), and decision.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation
Tests