Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 11 additions & 24 deletions evals/run-evals.sh
Original file line number Diff line number Diff line change
Expand Up @@ -1867,27 +1867,18 @@ assert_complex_diagnose_self() {
stdout_contains '\[tool:call\] shell_execute' && stdout_contains 'netclaw.*doctor'
}

# bounded-tool-output coverage (bound-tool-output-with-file-spill change).
# These two cases assert on OUTCOME, not mechanism: the prompts state only the
# goal and give the agent NO instructions about spilling, redirecting, re-running,
# file_read, StartLine/Limit, or grep. How the agent handles oversized output must
# come entirely from AGENTS.md, the netclaw-operations skill, and the steer text
# in the tool result — coaching it in the prompt would be testing instruction-
# following, not whether the real guidance surfaces work.
# These cases do not tell the agent how to continue bounded output.
# The guidance and the tool result must supply that behavior.
#
# The data is a deterministic Lehmer PRNG (pure integer modular arithmetic,
# identical across awk implementations and the host that computed the expected
# values), so the value at a deep line is reproducible AND un-fabricatable by the
# model. Because the tool bounds any single read to ~N=2000 inline chars, the
# deep-line value is unreachable from one read — so a correct answer can ONLY
# come from the agent paging/reading the oversized output the way the steer asks.
# Outcome therefore implies correct handling; no mechanism assertion is needed.

# Large SHELL output: ~210 KB on stdout exceeds N, so the daemon spills it and
# steers. Line 200 (value 872671849) sits past the inline window; reporting it
# proves the agent retrieved it from the bounded/spilled output unaided.
# A deterministic Lehmer generator supplies one reproducible value.
# The value is outside the inline output window.
# The assertion checks the continuation tool and the final value.

# This shell command produces about 210 KB of output.
# Line 200 is outside the inline window.
assert_complex_large_shell_output_spill() {
stdout_contains '\[tool:call\] shell_execute' && \
stdout_contains '\[tool:call\] tool_output_read' && \
stdout_response_contains '872671849'
}

Expand Down Expand Up @@ -2782,12 +2773,8 @@ run_all() {
run_case complex_diagnose_self "shell_execute with netclaw doctor" \
"Run netclaw doctor and summarize any problems"

# bounded-tool-output: oversized SHELL output. The prompt states only the
# goal — run a command and report a deep line of its output. How to cope with
# the output being too large to return inline (read the spill the steer hands
# back, rather than re-running) must come from the agent's own guidance, not
# this prompt. The number is a deterministic-but-opaque Lehmer PRNG value; the
# assertion checks the agent reports the correct line-200 value (872671849).
# The prompt gives the goal but does not name the continuation tool.
# The assertion requires the structured tool and the exact line value.
run_case complex_large_shell_output_spill "retrieves a deep line from oversized shell output unaided" \
"Run this command with shell_execute and tell me the number it prints on line 200: awk 'BEGIN{x=1;for(i=1;i<=20000;i++){x=(x*48271)%2147483647;print x}}'" \
"Using shell_execute, run: awk 'BEGIN{x=1;for(i=1;i<=20000;i++){x=(x*48271)%2147483647;print x}}' — then tell me which number is printed on the 200th line of its output."
Expand Down
8 changes: 4 additions & 4 deletions feeds/skills/.system/files/netclaw-operations/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: netclaw-operations
description: "REQUIRED when the user asks about scheduling, reminders, cron jobs, timers, background jobs, diagnostics, troubleshooting, MCP tools, daemon health, identity updates, or Netclaw capabilities and self-maintenance."
metadata:
author: netclaw
version: "2.61.0"
version: "2.62.0"
---

# Netclaw Operations
Expand Down Expand Up @@ -142,9 +142,9 @@ Tool output is bounded to a small inline budget
the context window. When a tool's output exceeds that budget you get a head+tail
view inline plus a pointer to the full output — not the whole thing:

- **`shell_execute`** spills the full (redacted) output to
`{session}/tool-calls/{toolCallId}.log` and gives you the path. Read a slice with
`file_read` (`StartLine`/`Limit`) or `grep` it — do NOT re-run the command to see more.
- **`shell_execute`** retains the full redacted output inside the current session.
Use `tool_output_read` with the returned `CallId`, `Start`, and `Limit` values.
Do not request a path or rerun the source tool to read more.
- **`file_read`** on a large file returns the head and steers you to read a
specific range with `StartLine`/`Limit` or `grep` (`StartLine` is a 1-based line
number — line 1 is the first line). Don't `cat` a huge file through
Expand Down
10 changes: 5 additions & 5 deletions openspec/changes/make-agent-tools-pit-of-success/tasks.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,11 +44,11 @@

## 6. PR 6 - Spill continuation and conditional schemas

- [ ] 6.1 Implement core tool_output_read by opaque call id with bounded windows and current-session-only spill resolution.
- [ ] 6.2 Make spill creation and continuation share one call-id sanitizer and reject traversal, controls, missing ids, and cross-session access.
- [ ] 6.3 Add source-generator support for explicit conditional tool variants and oneOf schemas without changing single-shape schemas.
- [ ] 6.4 Convert the observed mode-dependent first-party tools and reject zero/multiple matching branches before execution.
- [ ] 6.5 Add schema snapshots, generated-code tests, malformed-branch tests, spill-redaction tests, and public API compatibility checks.
- [x] 6.1 Implement core tool_output_read by opaque call id with bounded windows and current-session-only spill resolution.
- [x] 6.2 Make spill creation and continuation share one call-id sanitizer and reject traversal, controls, missing ids, and cross-session access.
- [x] 6.3 Add source-generator support for explicit conditional tool variants and oneOf schemas without changing single-shape schemas.
- [x] 6.4 Convert the observed mode-dependent first-party tools and reject zero/multiple matching branches before execution.
- [x] 6.5 Add schema snapshots, generated-code tests, malformed-branch tests, spill-redaction tests, and public API compatibility checks.

## 7. PR 7 - Replay, documentation, and rollout proof

Expand Down
35 changes: 28 additions & 7 deletions src/Netclaw.Actors.Tests/Tools/DispatchingToolExecutorTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -124,9 +124,10 @@ public async Task Verbose_tool_output_over_budget_is_windowed_and_spilled()
var result = await _executor.ExecuteAsync(toolCall, context, CancellationToken.None);

Assert.True(result.Length < 3000); // windowed inline, not the full 3000
Assert.Contains("output saved to", result);
Assert.Contains("file_read", result);
var spill = Path.Combine(sessionDir, "tool-calls", "call-spill.log");
Assert.Contains("tool_output_read", result);
Assert.Contains("CallId='call-spill'", result);
Assert.True(ToolOutputSpillLocation.TryResolve(
sessionDir, "call-spill", out _, out var spill));
Assert.True(File.Exists(spill));
Assert.Contains(new string('x', 100), await File.ReadAllTextAsync(spill, CancellationToken.None));
}
Expand All @@ -152,8 +153,9 @@ public async Task Spilled_output_is_redacted_before_write()
});

var result = await _executor.ExecuteAsync(toolCall, context, CancellationToken.None);
var onDisk = await File.ReadAllTextAsync(
Path.Combine(sessionDir, "tool-calls", "call-redact.log"), CancellationToken.None);
Assert.True(ToolOutputSpillLocation.TryResolve(
sessionDir, "call-redact", out _, out var spillPath));
var onDisk = await File.ReadAllTextAsync(spillPath, CancellationToken.None);

Assert.DoesNotContain("supersecret123", result);
Assert.DoesNotContain("supersecret123", onDisk); // redacted before the spill write
Expand Down Expand Up @@ -313,14 +315,33 @@ public async Task File_read_spill_file_is_redacted_even_when_model_result_is_not
// The inline result (model-facing) should NOT contain the redacted sentinel
Assert.DoesNotContain("***REDACTED***", result);
// But it should be truncated (spilled)
Assert.Contains("output saved to", result);
Assert.Contains("tool_output_read", result);

// The spill file on disk SHOULD be redacted
var spillPath = Path.Combine(sessionDir, "tool-calls", "call-spill-secret.log");
Assert.True(ToolOutputSpillLocation.TryResolve(
sessionDir, "call-spill-secret", out _, out var spillPath));
Assert.True(File.Exists(spillPath));
var spillContent = await File.ReadAllTextAsync(spillPath, CancellationToken.None);
Assert.Contains("***REDACTED***", spillContent);
Assert.DoesNotContain("real-secret-value", spillContent);

var continuationContext = TestToolExecutionContext.CreateBound(
"slack/thread-1",
sessionDir,
new TestToolExecutionContextOptions { Audience = TrustAudience.Personal });
var continuation = await _executor.ExecuteAsync(
CreateToolCall(
"call-continuation",
"tool_output_read",
ToolInput.Create("CallId", "call-spill-secret", "Limit", 256)),
continuationContext,
CancellationToken.None);
Assert.Contains("***REDACTED***", continuation);
Assert.DoesNotContain("real-secret-value", continuation);
Assert.Equal(
ToolInvocationOutcomeCategory.Success,
continuationContext.Invocation.Receipt?.Category);
Assert.Empty(continuationContext.Invocation.Receipt?.FileActivity ?? []);
}
finally
{
Expand Down
Loading
Loading