Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -380,7 +380,7 @@ Phase 6 — Hardening:

These are validated patterns from research (see `docs/research/pattern-analysis.md`). Implement them when building the relevant module.

**Tool nudging:** When an LLM responds with text instead of tool calls in the first 2 iterations, inject "Please proceed and use the available tools." Implement in `SpacebotHook.on_completion_response()`. Workers benefit most.
**Tool nudging / outcome gate:** Workers cannot exit with a text-only response until they signal a terminal outcome via `set_status(kind: "outcome")`. If a worker returns text without an outcome signal, the hook fires `Terminate` and retries with a nudge prompt (up to 2 retries). After retries are exhausted the worker fails with `PromptCancelled`. See `docs/design-docs/tool-nudging.md`.

**Fire-and-forget DB writes:** `tokio::spawn` for conversation history saves, memory writes, worker log persistence. User gets their response immediately.

Expand Down
80 changes: 44 additions & 36 deletions docs/design-docs/tool-nudging.md
Original file line number Diff line number Diff line change
@@ -1,40 +1,50 @@
# Tool Nudging

Automatic retry mechanism that encourages workers to use tools when they respond with text-only instead of calling tools.
Automatic retry mechanism that prevents workers from exiting with text-only responses before signaling a terminal outcome.

## Problem

Workers sometimes respond with text like "I'll help you with that" without actually calling any tools. This is particularly common:
Workers sometimes respond with text like "I'll help you with that" or "Let me create the email now..." without actually calling any tools. This is common:
- At the start of a worker loop when the LLM is "thinking out loud"
- When the task description is vague and the LLM wants clarification
- With certain models that have a conversational tendency
- **Mid-task**, after making a few tool calls (e.g. `read_skill`, `set_status`), the model returns narration instead of continuing with tools

Without intervention, the worker wastes tokens on non-actionable responses and may never complete the task.
Without intervention, the worker silently reaches `Done` state with no useful output. In Rig's agent loop, any text-only response (no tool calls) terminates the loop — the worker exits as if it completed successfully.

## Solution

Tool nudging detects text-only responses early in the worker loop and automatically retries with a nudge prompt: "Please proceed and use the available tools."
Workers must explicitly signal a terminal outcome via `set_status(kind: "outcome")` before they can exit with a text-only response. Until that signal is received, any text-only response triggers a nudge that sends the worker back to work.

### How It Works

```
Worker loop starts
→ First completion call
→ If text-only response (no tool calls)
→ Terminate with special "tool_nudge" reason
→ Retry with nudge prompt
→ LLM completion
→ If response includes tool calls → continue normally
→ If text-only response:
→ Has outcome been signaled via set_status(kind: "outcome")? → allow exit
→ No outcome signal? → Terminate with "tool_nudge" reason → retry with nudge prompt
→ Max 2 retries per prompt request
→ If tool call present
→ Continue normally
→ If retries exhausted → worker fails (PromptCancelled)
```

### Outcome Signaling

The `set_status` tool has a `kind` field:

- `kind: "progress"` (default) — intermediate status update, does not unlock exit
- `kind: "outcome"` — terminal result signal, allows text-only exit

Workers are instructed to call `set_status(kind: "outcome")` with a result summary before finishing. The hook detects this in `on_tool_call` by parsing the args.

### Policy Scoping

Tool nudging is scoped by process type:

| Process Type | Default Policy | Reason |
|--------------|----------------|--------|
| Worker | Enabled | Workers must use tools to complete tasks |
| Worker | Enabled | Workers must complete tasks before exiting |
| Branch | Disabled | Branches are for thinking, not doing |
| Channel | Disabled | Channels should be conversational |

Expand All @@ -47,23 +57,25 @@ let hook = SpacebotHook::new(...)

### Implementation Details

**Detection** (`src/hooks/spacebot.rs:should_nudge_tool_usage`):
- Only active on first 2 completion calls (`TOOL_NUDGE_MAX_RETRIES = 2`)
- Checks if response contains any `AssistantContent::ToolCall`
- Ignores empty text responses
- Stops nudging after any tool call is seen
**Outcome detection** (`src/hooks/spacebot.rs:on_tool_call`):
- When a `set_status` call has `kind: "outcome"` in its args, `outcome_signaled` is set to `true`
- The flag persists for the rest of the prompt request

**Retry Flow** (`prompt_with_tool_nudge_retry`):
1. Reset nudge state at start of prompt
2. Track completion call count via atomic counter
3. On text-only response: terminate with `TOOL_NUDGE_REASON`
4. Catch termination in retry loop, prune history, retry with nudge prompt
5. On success: prune the nudge prompt from history to keep context clean
**Nudge decision** (`src/hooks/spacebot.rs:should_nudge_tool_usage`):
- Returns `true` when: policy enabled, nudge active, no outcome signaled, response is text-only
- Returns `false` when: outcome signaled, response has tool calls, policy disabled

**History Hygiene**:
**Retry flow** (`prompt_with_tool_nudge_retry`):
1. Reset nudge state at start of prompt (clears `outcome_signaled`)
2. On text-only response without outcome: terminate with `TOOL_NUDGE_REASON`
3. Catch termination in retry loop, prune history, retry with nudge prompt
4. On success: prune the nudge prompt from history to keep context clean
5. After `TOOL_NUDGE_MAX_RETRIES` (2) exhausted: `PromptCancelled` propagates to worker → `WorkerState::Failed`

**History hygiene**:
- Synthetic nudge prompts are removed from history on both success and retry
- Failed assistant turns are pruned but user prompts are preserved
- Prevents accumulation of "Please proceed..." noise in context
- Prevents accumulation of nudge noise in context

### Configuration

Expand All @@ -86,23 +98,19 @@ let follow_up_hook = hook
The nudging behavior has comprehensive test coverage:

- **Unit tests** (`src/hooks/spacebot.rs`):
- `nudges_only_on_first_two_text_only_completion_calls`
- `nudges_on_every_text_only_response_without_outcome` — nudge fires on every text-only response
- `nudges_after_tool_calls_without_outcome` — the exact bug case (read_skill + progress status + text exit)
- `outcome_signal_allows_text_only_completion` — outcome signal unlocks exit
- `progress_status_does_not_signal_outcome` — explicit progress kind doesn't unlock
- `default_status_kind_does_not_signal_outcome` — omitted kind doesn't unlock
- `does_not_nudge_when_completion_contains_tool_call`
- `does_not_nudge_after_any_tool_call_has_started`
- `process_scoped_policy_*` variants for Branch/Channel/Worker
- `tool_nudge_retry_history_hygiene_*` for history pruning

- **Integration tests** (`tests/tool_nudge.rs`):
- End-to-end nudge flow with mock model
- Verification that branches/channels don't nudge
- History accumulation prevention

### Metrics

When the `metrics` feature is enabled:
- `tool_calls_total` - Count of tool calls (existing)
- `tool_call_duration_seconds` - Duration of tool calls (existing)
- Nudge retry count is logged but not yet a dedicated metric
- Public API surface tests (constants, policy enum, hook creation)
- Event emission verification
- Process-type scoping

### Future Considerations

Expand Down
2 changes: 1 addition & 1 deletion prompts/en/tools/set_status_description.md.j2
Original file line number Diff line number Diff line change
@@ -1 +1 @@
Report the current status of your work. Use this to update the channel on your progress. The status will appear in the channel's status block. Keep statuses concise (1-2 sentences) and informative.
Report the current status of your work. The status appears in the channel's status block. Keep statuses concise (1-2 sentences) and informative. Use kind "progress" (default) for intermediate updates. Use kind "outcome" when the task has reached a terminal result — you must signal an outcome before finishing.
20 changes: 15 additions & 5 deletions prompts/en/worker.md.j2
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,9 @@ Sandbox mode is **disabled**.

## Your Role

Execute the task you were given. Use your tools. Report your status as you make progress. Return the result when you're done.
Execute the task you were given. Use your tools. Report your status as you make progress.

**When the task is complete** (or you've reached a definitive result), call `set_status` with `kind: "outcome"` and a summary of what happened. Only then provide your final text response. If you try to finish without signaling an outcome, the system will send you back to keep working.

## Task

Expand All @@ -51,20 +53,28 @@ Your task is provided in the first message. It contains everything you need to k

### set_status

Update your visible status. The channel sees this in its status block. Use it to report meaningful progress, not every micro-step.
Update your visible status. The channel sees this in its status block. Has two modes:

**Progress updates** (`kind: "progress"`, the default): Report meaningful intermediate progress, not every micro-step.

Good status updates:
Good progress updates:

- "running tests, 3/7 passing"
- "refactored auth module, updating imports"
- "found 3 matching files, analyzing"

Bad status updates:
Bad progress updates:

- "thinking..."
- "starting"
- "reading file"

**Outcome** (`kind: "outcome"`): Signal that you have reached a terminal result. You **must** call `set_status` with `kind: "outcome"` before finishing your task. Include a concise summary of the result. Examples:

- `set_status(status: "Email sent to jamie@spacedrive.com", kind: "outcome")`
- `set_status(status: "Build failed: 3 type errors in auth module", kind: "outcome")`
- `set_status(status: "Deployed v2.1.0 to staging", kind: "outcome")`

### shell

Execute shell commands. Use this for running builds, tests, git operations, package management, and any system commands.
Expand Down Expand Up @@ -121,7 +131,7 @@ Do not log or echo the secret value after storing it.
1. Do the work. Don't describe what you would do — use the tools and do it.
2. Update your status at meaningful checkpoints. The channel is using your status to keep the user informed.
3. If a tool call fails, try to recover. Read the error, adjust, and retry. Don't give up on the first failure.
4. When you're done with the task, you'll be asked to produce a summary. That summary is the only thing the channel sees — your tool history stays here. Focus on doing the work first, summarizing second.
4. **Signal your outcome.** When the task is done, call `set_status(kind: "outcome")` with a summary of the result before providing your final text. You cannot finish without this — the system will reject premature exits.
5. Stay focused on the task. Don't explore tangential work unless it's necessary to complete what you were asked to do.
6. If you receive follow-up messages (interactive mode), treat them as additional instructions building on your existing context.
{% if tool_secret_names %}
Expand Down
Loading