Skip to content

Complete structured tool rollout proof - #2039

Merged
Aaronontheweb merged 3 commits into
goal/pit-of-success-output-contractsfrom
goal/pit-of-success-rollout-proof
Aug 20, 2026
Merged

Complete structured tool rollout proof#2039
Aaronontheweb merged 3 commits into
goal/pit-of-success-output-contractsfrom
goal/pit-of-success-rollout-proof

Conversation

@Aaronontheweb

@Aaronontheweb Aaronontheweb commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Replay the seven sanitized tool-friction fixtures through real registration, policy, dispatch, typed outcome, and working-context paths.
  • Prefer structured workspace tools and progressive discovery in parent and child guidance.
  • Add count-only tool exposure diagnostics and outcome-category diagnostics without payloads.
  • Extend the behavioral suite for recursive search, bounded batch reads, JSON selection, and image metadata.

This PR is the seventh and final implementation slice in stack #2023. It depends on #2038.

Hosted behavior comparison

The selected comparison ran five trials per scenario on the production patch now at commit 55f3ff9. Only aggregate results are reported here. Raw transcripts and artifacts are not published.

Scenario Result Observed tool choice
Recursive repository search 5/5 file_search in every trial; no shell fallback
Known batch read 5/5 file_read_many in every trial; no shell fallback
Known JSON selection 5/5 json_read in every trial; no shell fallback
Known image metadata 5/5 file_read in every trial; no shell fallback
Deferred tool discovery 5/5 search_tools in every trial; no shell fallback
Spilled output continuation 0/5 shell used in every trial; catalog searched once; tool_output_read was never loaded or called

The five structured-selection scenarios passed 25/25. The spill-continuation failure remains recorded as a rollout finding; its assertion was not weakened.

Validation

  • Release build: 0 warnings, 0 errors
  • Full solution tests: 7,751 passed; 16 expected platform or external-service skips; 0 failed
  • Sanitized friction replay: 8/8 passed
  • Strict OpenSpec validation passed
  • Copyright-header verification passed
  • Changed-file formatting validation passed
  • Slopwatch: 0 issues
  • Diff check passed
  • Added-line privacy scan found no credentials, endpoints, email addresses, URLs, or operator home paths
  • No public API or durable wire-format change

Stack created with GitHub Stacks CLI

@Aaronontheweb Aaronontheweb changed the title goal/pit of success rollout proof Complete structured tool rollout proof Aug 20, 2026
Comment on lines +380 to +384
foreach (var result in results)
{
if (result.Receipt is { } receipt)
_logger.Info("Tool outcome category={OutcomeCategory}", receipt.Category);
}
@Aaronontheweb
Aaronontheweb force-pushed the goal/pit-of-success-rollout-proof branch from 58cc36b to 3f1e262 Compare August 20, 2026 07:14
@Aaronontheweb
Aaronontheweb merged commit f08a2fb into dev Aug 20, 2026
23 checks passed
@Aaronontheweb
Aaronontheweb deleted the goal/pit-of-success-rollout-proof branch August 20, 2026 08:29
@Aaronontheweb

Copy link
Copy Markdown
Collaborator Author

Post-merge binary-swap evidence for f08a2fb8 (sanitized aggregate only):

  • 6 fresh headless sessions; 6/6 bounded objectives completed
  • 0 approval events and 0 denial events
  • 39 tool calls: 28 structured calls and 11 shell_execute calls across 2 sessions
  • Session-scratch write/read, image metadata, source search, and deferred reminder discovery completed through structured tooling
  • Deferred reminder discovery used search_tools then load_tool successfully
  • The batch-read case did not select file_read_many (0/1)
  • The JSON-projection case did not select json_read (0/1)
  • Shell fallback therefore remains material even though this sample completed without approval spam

Raw captures stayed local and were removed after aggregation. This comment contains no prompts, responses, paths, session/call identifiers, provider details, endpoints, credentials, or host details.

@Aaronontheweb Aaronontheweb added tools Issues related to agent tools: file_read, web_search, shell_execute, image processing, etc. tests All issues related to testing, quality assurance, and smoke testing. evals LLM efficacy evals for Netclaw documentation Improvements or additions to documentation context-pipeline LLM context assembly: prompt layers, dynamic injection, memory recall, temporal grounding subagents spawn_agent, SubAgentActor, definition loader, discovery context layer, and related features labels Aug 20, 2026
@Aaronontheweb Aaronontheweb mentioned this pull request Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

context-pipeline LLM context assembly: prompt layers, dynamic injection, memory recall, temporal grounding documentation Improvements or additions to documentation evals LLM efficacy evals for Netclaw subagents spawn_agent, SubAgentActor, definition loader, discovery context layer, and related features tests All issues related to testing, quality assurance, and smoke testing. tools Issues related to agent tools: file_read, web_search, shell_execute, image processing, etc.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant