Skip to content

Repair tool rollout contracts - #2046

Merged
Aaronontheweb merged 1 commit into
refactor/remove-bulk-workspace-toolsfrom
fix/repair-tool-rollout-contracts
Aug 26, 2026
Merged

Repair tool rollout contracts#2046
Aaronontheweb merged 1 commit into
refactor/remove-bulk-workspace-toolsfrom
fix/repair-tool-rollout-contracts

Conversation

@Aaronontheweb

@Aaronontheweb Aaronontheweb commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • keep spill files internal and expose only an opaque call id through tool_output_read
  • describe load_tool as schema exposure while retaining normal dispatch authorization
  • direct agents to load a known exact tool name without a redundant search
  • replay the catalog fixture through a real child actor and inspect its model-visible tools
  • preserve the reduced 12-tool core footprint

Validation

  • Release build: 0 warnings, 0 errors
  • focused dispatcher, replay, schema, prompt, and skill tests: 153 passed
  • strict OpenSpec: passed
  • deterministic eval harness checks: passed
  • headers: passed
  • Slopwatch: 0 findings
  • diff check: passed

Evaluation scope

This PR changes model-visible guidance and spill behavior. The final-head hosted behavioral rerun is still required; only PII-free aggregate evidence will be posted here.

Stack

This is PR 3 of 3. Its base is PR #2045, now refreshed at c19edbde. No merge or auto-merge is enabled.

@Aaronontheweb
Aaronontheweb force-pushed the fix/repair-tool-rollout-contracts branch 2 times, most recently from 78cefb3 to bebf821 Compare August 20, 2026 15:36
@Aaronontheweb

Copy link
Copy Markdown
Collaborator Author

Final-head behavioral eval summary (five hosted trials per case; aggregate only):

  • Composed bounded reads: 5/5. Both requested values were recovered through two file_read calls, with no shell fallback.
  • Deferred specialty use: 5/5 task completion. Four trials used load_tool → list_reminders directly; one used search_tools → load_tool → list_reminders.
  • Opaque spill continuation: 3/5 contract compliance. All five trials recovered the correct value. Three used tool_output_read without rerunning shell; one recomputed with a second shell call, and one paged before rerunning shell.

The initial spill run exposed an eval defect: the model could add a filter that prevented spilling, while the headless harness denied the modified command. The final harness now requires the exact producer command, verifies its structured arguments, and grants that synthetic producer only inside the isolated eval configuration.

Deterministic replay/evidence checks: 190 passed, 2 platform-specific skips, 0 failed. Full solution validation on the final product code: 7,749 passed, 17 expected skips, 0 failed. Strict OpenSpec, headers, diff check, and Slopwatch passed.

No raw transcripts, prompts, endpoints, credentials, host paths, session identifiers, or model/provider details are included here.

@Aaronontheweb
Aaronontheweb force-pushed the fix/repair-tool-rollout-contracts branch from bebf821 to bd9d029 Compare August 20, 2026 15:37
@Aaronontheweb Aaronontheweb added documentation Improvements or additions to documentation context-pipeline LLM context assembly: prompt layers, dynamic injection, memory recall, temporal grounding subagents spawn_agent, SubAgentActor, definition loader, discovery context layer, and related features skills Skill loading, handling, authoring, indexing, and evals. tests All issues related to testing, quality assurance, and smoke testing. tools Issues related to agent tools: file_read, web_search, shell_execute, image processing, etc. evals LLM efficacy evals for Netclaw labels Aug 20, 2026
@Aaronontheweb
Aaronontheweb force-pushed the fix/repair-tool-rollout-contracts branch from bd9d029 to 01176b9 Compare August 20, 2026 15:50

@Aaronontheweb Aaronontheweb left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Focused review of the rollout and eval contracts.

"personal": {
"shell_execute": [
{
"verb": "awk",

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This grant applies to every eval case, and awk can execute arbitrary programs. That can mask approval behavior outside the spill scenario. Please prefer the existing deterministic large fixture with a reviewed-safe producer such as cat, or scope this grant to the one case.

/// tool in its private exposure set.
/// Exposes one deferred first-party or MCP tool schema by exact name.
/// The owning actor intercepts successful results and adds the requested tool
/// to its private exposure set. Dispatch still runs normal authorization.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important semantic boundary: loading controls schema exposure only. A recalled deferred name may still dispatch, and normal authorization remains authoritative. Please keep the direct-dispatch authorization regression and avoid describing this as an execution grant or lease.

var inline = BoundedOutputReader.Window(modelFacingResult, budget);
var spillPath = await TryWriteSpillAsync(spillContent, toolCallId, context, ct);
return Compose(inline, spillPath, modelFacingResult.Length, budget);
var spillCallId = await TryWriteSpillAsync(spillContent, toolCallId, context, ct);

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This production hunk renames the local value; it does not remove the inline head/tail window. Please keep the PR claim scoped to opaque call-id/path contract alignment rather than implying that no inline spill content is returned.

@Aaronontheweb
Aaronontheweb force-pushed the fix/repair-tool-rollout-contracts branch from 01176b9 to a7f28d6 Compare August 21, 2026 13:37
@Aaronontheweb
Aaronontheweb force-pushed the fix/repair-tool-rollout-contracts branch from a7f28d6 to 20f35a5 Compare August 24, 2026 18:05
@Aaronontheweb
Aaronontheweb force-pushed the fix/repair-tool-rollout-contracts branch from 20f35a5 to cd64cbd Compare August 24, 2026 18:55
@Aaronontheweb
Aaronontheweb force-pushed the fix/repair-tool-rollout-contracts branch from cd64cbd to 1046bbc Compare August 24, 2026 19:09
@Aaronontheweb
Aaronontheweb force-pushed the fix/repair-tool-rollout-contracts branch from 1046bbc to 7fcce09 Compare August 25, 2026 20:34
@Aaronontheweb
Aaronontheweb force-pushed the fix/repair-tool-rollout-contracts branch from 7fcce09 to cc22d6f Compare August 25, 2026 20:51
@Aaronontheweb
Aaronontheweb force-pushed the fix/repair-tool-rollout-contracts branch from cc22d6f to ea975a5 Compare August 25, 2026 20:57

@Aaronontheweb Aaronontheweb left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - mostly tool hints and guidance

@Aaronontheweb
Aaronontheweb merged commit e05f294 into dev Aug 26, 2026
23 checks passed
@Aaronontheweb
Aaronontheweb deleted the fix/repair-tool-rollout-contracts branch August 26, 2026 03:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

context-pipeline LLM context assembly: prompt layers, dynamic injection, memory recall, temporal grounding documentation Improvements or additions to documentation evals LLM efficacy evals for Netclaw skills Skill loading, handling, authoring, indexing, and evals. subagents spawn_agent, SubAgentActor, definition loader, discovery context layer, and related features tests All issues related to testing, quality assurance, and smoke testing. tools Issues related to agent tools: file_read, web_search, shell_execute, image processing, etc.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant