Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/runbooks/memory-health-and-evals.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,11 +44,15 @@ sqlite3 "$HOME/.netclaw/memory/netclaw-memory.db" \
Subagent-originated memory candidates are surfaced as session `SubAgentOutput`
completion events with:

- `outcome` (`completed`, `partial`, `failed`)
- `outcomeReason` (when the run ended for a machine-readable non-completed reason)
- `memoryDecision` (`accepted`, `deferred`, `rejected`)
- `memoryDecisionReason` (when decision is not accepted)
- `findingsCount`

Only `accepted` findings are enqueued into the memory checkpoint pipeline.
Partial runs can still produce accepted findings; failed runs are treated as
operator-visible diagnostics rather than durable-memory evidence by default.

## Eval Execution

Expand Down
15 changes: 12 additions & 3 deletions docs/runbooks/subagents.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,8 +93,11 @@ first user message is just the raw task, identical to the pre-context protocol.
lifecycle-managed — stops when the session stops).
4. The subagent runs an autonomous LLM loop: call tools, process results, repeat.
5. After at most 30 tool iterations, a final response, or an inactivity timeout,
the subagent returns its final text response.
6. The main agent receives this response as the `spawn_agent` tool result.
the subagent returns a terminal run result.
6. The main agent receives the `spawn_agent` tool result as an explicit text
envelope: agent name, run id, outcome (`completed`, `partial`, or `failed`),
optional reason, diagnostics pointer, and either a `Summary:` or `Error:`
section containing the subagent's final text.

Child creation is marshaled back onto the session actor thread, so supervision
stays within Akka's actor-thread rules. If the parent tool call is cancelled or
Expand All @@ -117,10 +120,16 @@ are suppressed in Slack.
Completion events are emitted for every finished subagent run, even when the
subagent returns no structured findings. In that case `FindingsCount` is `0`
and the memory-decision fields are empty because there was nothing to review.
The completion event carries the same terminal outcome and reason used by the
tool-result envelope, so operators can distinguish a useful partial summary from
a failed run.

Structured findings are conservative, parent-reviewed durable-memory candidates.
They should be emitted as explicit conclusion envelopes with review metadata,
not inferred from free-form work logs or tool transcripts.
not inferred from free-form work logs or tool transcripts. They are not the
parent-facing `spawn_agent` result; they exist so accepted subagent conclusions
can enter the memory checkpoint pipeline without asking the parent model to parse
free-form work logs.

## Defining subagents

Expand Down
7 changes: 4 additions & 3 deletions docs/spec/SPEC-016-tool-liveness-and-stall-detection.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,8 +150,9 @@ Expected stream shape:
longer applied to this call.
4. The child watchdog governs prefill, model deltas, keepalive-only wedges,
tool-loop progress, approval waits, cancellation, and iteration exhaustion.
5. The terminal `ToolCompletedUpdate` carries either a successful sub-agent result
or a failed sub-agent result produced by the child.
5. The terminal `ToolCompletedUpdate` carries an explicit sub-agent run envelope
produced by the child: run id, outcome, optional reason, diagnostics pointer,
and the final summary or error text.

## Validation Strategy

Expand Down Expand Up @@ -204,7 +205,7 @@ without the parent killing it, while opaque tools remain bounded.
### Diagnostics And Manual Repro

- Logs SHALL correlate parent session id, sub-agent run id, child watchdog
reason, and terminal `spawn_agent` result.
reason, terminal outcome, and terminal `spawn_agent` result.
- Manual repro should run parallel `spawn_agent` calls where one child opens a
quiet window longer than the parent tool timeout. The expected result is no
parent `produced no activity` failure; either the child completes or the child
Expand Down
8 changes: 7 additions & 1 deletion feeds/skills/.system/files/subagent-authoring/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: subagent-authoring
description: "How to create and troubleshoot file-defined subagents in ~/.netclaw/agents. Load when the user asks to add, edit, or debug subagent definitions, or when a skill routes via metadata.subagent."
metadata:
author: netclaw
version: "1.3.1"
version: "1.3.2"
---

# Subagent Authoring
Expand Down Expand Up @@ -136,6 +136,12 @@ nudges, duplicate-call nudges, and force-no-tools wrap-up. The current
subagent budget is 30 tool iterations per run, where one LLM response with any
number of parallel tool calls counts as one iteration.

The parent-facing `spawn_agent` result is an explicit terminal text envelope:
agent name, run id, outcome (`completed`, `partial`, or `failed`), optional
reason, diagnostics pointer, and a `Summary:` or `Error:` section. Structured
findings, when enabled, are a separate parent-reviewed memory-candidate path;
do not rely on them as the visible result returned to the parent model.

## Fail-loud loader behavior

On the next turn or subagent lookup, invalid files are skipped with warnings.
Expand Down
4 changes: 3 additions & 1 deletion src/Netclaw.Actors.Tests/Sessions/TurnStateTrackerTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,9 @@ public void ReachingIterationCap_ReturnsExhausted()

// The cap-th iteration hits the limit.
var capped = tracker.RecordToolCompletion(resultCount: 1, maxToolIterationsPerTurn: cap);
Assert.IsType<ToolBudgetStatus.Exhausted>(capped);
var exhausted = Assert.IsType<ToolBudgetStatus.Exhausted>(capped);
Assert.Contains("executive summary", exhausted.NudgeText, StringComparison.OrdinalIgnoreCase);
Assert.Contains("Partial or Unknown", exhausted.NudgeText, StringComparison.Ordinal);
Assert.Equal(cap, tracker.ToolIterationCount);
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,9 @@ public async Task Spawn_agent_streams_activity_through_executor_dispatch_to_watc
// not a "Subagent '...' failed: ..." message from FormatResult.
Assert.NotNull(result);
Assert.NotEmpty(result);
Assert.Contains("Subagent run finished.", result, StringComparison.Ordinal);
Assert.Contains("Outcome: completed", result, StringComparison.OrdinalIgnoreCase);
Assert.Contains("Summary:", result, StringComparison.Ordinal);
Assert.DoesNotContain("failed", result, StringComparison.OrdinalIgnoreCase);
}

Expand Down Expand Up @@ -207,6 +210,8 @@ public async Task Spawn_agent_self_monitoring_survives_quiet_window_after_first_

var result = await drain.WaitAsync(TimeSpan.FromSeconds(3), TestContext.Current.CancellationToken);
Assert.NotNull(result);
Assert.Contains("Subagent run finished.", result, StringComparison.Ordinal);
Assert.Contains("Outcome: completed", result, StringComparison.OrdinalIgnoreCase);
Assert.Contains("summary complete", result);
}

Expand Down
14 changes: 12 additions & 2 deletions src/Netclaw.Actors.Tests/SubAgents/SubAgentActorTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,8 @@ public async Task Text_response_returns_success_result()
TimeSpan.FromSeconds(5), TestContext.Current.CancellationToken);

Assert.True(result.Success);
Assert.Equal(SubAgentRunOutcome.Completed, result.Outcome);
Assert.Null(result.OutcomeReason);
Assert.Contains("Response #1", result.Output);
Assert.Equal("test-agent", result.AgentName.Value);
Assert.Empty(result.Findings);
Expand Down Expand Up @@ -1021,6 +1023,8 @@ public async Task Max_iterations_forces_text_response()

// After the configured tool budget, force a no-tools call which returns text.
Assert.True(result.Success);
Assert.Equal(SubAgentRunOutcome.Partial, result.Outcome);
Assert.Equal(SubAgentOutcomeReason.ToolIterationBudgetExhausted, result.OutcomeReason);
Assert.Equal(4, fakeClient.CallCount);
Assert.NotNull(fakeClient.LastReceivedMessages);
Assert.Contains(fakeClient.LastReceivedMessages,
Expand Down Expand Up @@ -1270,11 +1274,14 @@ public async Task Long_text_response_does_not_emit_findings_by_default()
}

[Fact]
public async Task Long_text_response_emits_findings_when_enabled()
public async Task Long_text_response_emits_untruncated_findings_when_enabled()
{
var longSummary = string.Concat(
new string('a', 1900),
"\nTAIL_CONCLUSION: preserve this final conclusion and citation.");
var fakeClient = new FakeChatClient
{
ResponseText = "This is a durable subagent summary with enough detail to be considered a memory candidate for parent-session checkpoint review."
ResponseText = longSummary
};
var definition = CreateDefinition() with { EmitStructuredFindings = true };
var agent = Sys.ActorOf(SubAgentActor.CreateProps(definition, fakeClient));
Expand All @@ -1285,11 +1292,14 @@ public async Task Long_text_response_emits_findings_when_enabled()

Assert.True(result.Success);
Assert.Single(result.Findings);
Assert.Equal(longSummary, result.Findings[0].Content);
Assert.Contains("TAIL_CONCLUSION", result.Findings[0].Content, StringComparison.Ordinal);
Assert.Equal(SubAgentFindingShape.Conclusion, result.Findings[0].Shape);
Assert.Equal("subagent:test-agent", result.Findings[0].Title);
Assert.Equal(SubAgentFindingDurability.Durable, result.Findings[0].Durability);
Assert.Equal(SubAgentFindingReusability.Reusable, result.Findings[0].Reusability);
Assert.Equal(SubAgentFindingRecallMode.Searchable, result.Findings[0].RecallMode);
Assert.Contains("subagent_outcome:completed", result.Findings[0].Evidence);
}

[Fact]
Expand Down
2 changes: 2 additions & 0 deletions src/Netclaw.Actors/Protocol/SessionOutputDto.cs
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,8 @@ public sealed record SessionOutputDto
public string? Phase { get; init; }
public int? ToolCountSub { get; init; }
public bool? SubAgentSuccess { get; init; }
public string? SubAgentOutcome { get; init; }
public string? SubAgentOutcomeReason { get; init; }
public double? DurationMs { get; init; }
public string? MemoryDecision { get; init; }
public string? MemoryDecisionReason { get; init; }
Expand Down
59 changes: 44 additions & 15 deletions src/Netclaw.Actors/Protocol/SessionOutputDtoMapper.cs
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
// -----------------------------------------------------------------------
using Netclaw.Actors.Reminders;
using Netclaw.Media;
using Netclaw.Tools;
using static Netclaw.Actors.Sessions.SessionProtocol;

namespace Netclaw.Actors.Protocol;
Expand Down Expand Up @@ -133,6 +134,12 @@ public static class SessionOutputDtoMapper
Phase = msg.Phase.ToString().ToLowerInvariant(),
ToolCountSub = msg.ToolCount,
SubAgentSuccess = msg.Success,
SubAgentOutcome = msg.Phase == SubAgents.SubAgentPhase.Completed

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This intentionally emits outcome metadata only for completed subagent events. A started event has no terminal outcome yet, so leaving the DTO fields absent avoids sending a misleading completed default over SignalR.

? msg.Outcome.ToString().ToLowerInvariant()
: null,
SubAgentOutcomeReason = msg.Phase == SubAgents.SubAgentPhase.Completed
? msg.OutcomeReason?.Value
: null,
DurationMs = msg.Duration.TotalMilliseconds,
MemoryDecision = msg.MemoryDecision,
MemoryDecisionReason = msg.MemoryDecisionReason,
Expand Down Expand Up @@ -294,21 +301,7 @@ public static SessionOutput FromDto(SessionOutputDto dto)
FileName = dto.FileName ?? "file",
MimeType = new MimeType(dto.MimeType)
},
SessionOutputTypes.SubAgent => new SubAgentOutput
{
SessionId = sessionId,
TimestampMs = dto.TimestampMs,
AgentName = new SubAgents.AgentName(dto.AgentName ?? "unknown"),
Phase = dto.Phase?.Equals("completed", StringComparison.OrdinalIgnoreCase) == true
? SubAgents.SubAgentPhase.Completed
: SubAgents.SubAgentPhase.Started,
ToolCount = dto.ToolCountSub ?? 0,
Success = dto.SubAgentSuccess ?? false,
Duration = TimeSpan.FromMilliseconds(dto.DurationMs ?? 0),
MemoryDecision = dto.MemoryDecision,
MemoryDecisionReason = dto.MemoryDecisionReason,
FindingsCount = dto.FindingsCount ?? 0
},
SessionOutputTypes.SubAgent => MapSubAgentOutput(dto, sessionId),
SessionOutputTypes.BufferFlush => new BufferFlush
{
SessionId = sessionId,
Expand Down Expand Up @@ -362,6 +355,42 @@ public static SessionOutput FromDto(SessionOutputDto dto)
TimestampMs = dto.TimestampMs,
Message = $"Unknown output type from daemon: {dto.Type}"
}
};
}

private static SubAgentRunOutcome ParseSubAgentOutcome(string? value, bool? success)
{
if (!string.IsNullOrWhiteSpace(value)
&& Enum.TryParse<SubAgentRunOutcome>(value, ignoreCase: true, out var parsed))
return parsed;

return success == false ? SubAgentRunOutcome.Failed : SubAgentRunOutcome.Completed;
}

private static SubAgentOutput MapSubAgentOutput(SessionOutputDto dto, SessionId sessionId)
{
var phase = dto.Phase?.Equals("completed", StringComparison.OrdinalIgnoreCase) == true
? SubAgents.SubAgentPhase.Completed
: SubAgents.SubAgentPhase.Started;

return new SubAgentOutput
{
SessionId = sessionId,
TimestampMs = dto.TimestampMs,
AgentName = new SubAgents.AgentName(dto.AgentName ?? "unknown"),
Phase = phase,
ToolCount = dto.ToolCountSub ?? 0,
Success = dto.SubAgentSuccess ?? false,
Outcome = phase == SubAgents.SubAgentPhase.Completed
? ParseSubAgentOutcome(dto.SubAgentOutcome, dto.SubAgentSuccess)
: SubAgentRunOutcome.Completed,
OutcomeReason = phase == SubAgents.SubAgentPhase.Completed && !string.IsNullOrWhiteSpace(dto.SubAgentOutcomeReason)
? new SubAgentOutcomeReason(dto.SubAgentOutcomeReason)
: null,
Duration = TimeSpan.FromMilliseconds(dto.DurationMs ?? 0),
MemoryDecision = dto.MemoryDecision,
MemoryDecisionReason = dto.MemoryDecisionReason,
FindingsCount = dto.FindingsCount ?? 0
};
}
}
6 changes: 3 additions & 3 deletions src/Netclaw.Actors/Sessions/Handlers/TurnStateTracker.cs
Original file line number Diff line number Diff line change
Expand Up @@ -124,9 +124,9 @@ public ToolBudgetStatus RecordToolCompletion(int resultCount, int maxToolIterati
return new ToolBudgetStatus.Exhausted(
$"You have reached the tool iteration limit for this turn. "
+ "Do NOT request any more tools. "
+ "Summarize the work you completed and produce your final response "
+ "based on the information you have gathered so far. "
+ "If you could not complete the task, explain what you found and what remains.");
+ "Produce a concise final executive summary based only on the information gathered so far. "
+ "Use this format: Summary, Completed, Partial or Unknown, Caveats, Useful Evidence. "
+ "Clearly state that the result is partial when work remains or evidence is incomplete.");
}

var budgetThreshold = (int)(maxToolIterationsPerTurn * BudgetNudgeRatio);
Expand Down
6 changes: 4 additions & 2 deletions src/Netclaw.Actors/Sessions/LlmMessages.cs
Original file line number Diff line number Diff line change
Expand Up @@ -100,9 +100,11 @@ internal sealed record ToolExecutionBatchCompleted : INoSerializationVerificatio

internal sealed record CompletedSubAgentRun : INoSerializationVerificationNeeded
{
public required string RunId { get; init; }
public required SubAgentRunId RunId { get; init; }
public required SubAgents.AgentName AgentName { get; init; }
public required bool Success { get; init; }
public required SubAgentRunOutcome Outcome { get; init; }
public SubAgentOutcomeReason? OutcomeReason { get; init; }
public required TimeSpan Duration { get; init; }
public int FindingsCount { get; init; }
public string? MemoryDecision { get; init; }
Expand All @@ -111,7 +113,7 @@ internal sealed record CompletedSubAgentRun : INoSerializationVerificationNeeded

internal sealed record AcceptedSubAgentFinding : INoSerializationVerificationNeeded
{
public required string RunId { get; init; }
public required SubAgentRunId RunId { get; init; }
public required SubAgents.AgentName AgentName { get; init; }
public required TimeSpan Duration { get; init; }
public required SubAgentFindingShape Shape { get; init; }
Expand Down
16 changes: 12 additions & 4 deletions src/Netclaw.Actors/Sessions/LlmSessionActor.cs
Original file line number Diff line number Diff line change
Expand Up @@ -811,13 +811,13 @@ private void HandleToolExecutionCompleted(ToolExecutionCompleted msg)
foreach (var startedJob in msg.StartedBackgroundJobs)
TrackStartedBackgroundJob(startedJob);

var emittedRunIds = new HashSet<string>(StringComparer.Ordinal);
var emittedRunIds = new HashSet<SubAgentRunId>();
foreach (var finding in msg.AcceptedSubAgentFindings)
{
if (emittedRunIds.Add(finding.RunId))
{
var runSummary = msg.CompletedSubAgentRuns
.FirstOrDefault(x => string.Equals(x.RunId, finding.RunId, StringComparison.Ordinal));
.FirstOrDefault(x => x.RunId == finding.RunId);

EmitOutput(new SubAgentOutput
{
Expand All @@ -826,6 +826,8 @@ private void HandleToolExecutionCompleted(ToolExecutionCompleted msg)
AgentName = finding.AgentName,
Phase = Netclaw.Actors.SubAgents.SubAgentPhase.Completed,
Success = true,
Outcome = runSummary?.Outcome ?? SubAgentRunOutcome.Completed,
OutcomeReason = runSummary?.OutcomeReason,
Duration = finding.Duration,
MemoryDecision = finding.Decision.ToWireValue(),
MemoryDecisionReason = finding.DecisionReason,
Expand Down Expand Up @@ -860,6 +862,8 @@ private void HandleToolExecutionCompleted(ToolExecutionCompleted msg)
AgentName = run.AgentName,
Phase = Netclaw.Actors.SubAgents.SubAgentPhase.Completed,
Success = run.Success,
Outcome = run.Outcome,
OutcomeReason = run.OutcomeReason,
Duration = run.Duration,
MemoryDecision = run.MemoryDecision,
MemoryDecisionReason = run.MemoryDecisionReason,
Expand Down Expand Up @@ -4339,13 +4343,13 @@ private void ProcessToolCallResult(Pipelines.ToolCallResult result)
{
TrackStartedBackgroundJob(result.StartedBackgroundJob);

var emittedRunIds = new HashSet<string>(StringComparer.Ordinal);
var emittedRunIds = new HashSet<SubAgentRunId>();
foreach (var finding in result.AcceptedSubAgentFindings)
{
if (emittedRunIds.Add(finding.RunId))
{
var runSummary = result.CompletedSubAgentRuns
.FirstOrDefault(x => string.Equals(x.RunId, finding.RunId, StringComparison.Ordinal));
.FirstOrDefault(x => x.RunId == finding.RunId);

EmitOutput(new SubAgentOutput
{
Expand All @@ -4354,6 +4358,8 @@ private void ProcessToolCallResult(Pipelines.ToolCallResult result)
AgentName = finding.AgentName,
Phase = Netclaw.Actors.SubAgents.SubAgentPhase.Completed,
Success = true,
Outcome = runSummary?.Outcome ?? SubAgentRunOutcome.Completed,
OutcomeReason = runSummary?.OutcomeReason,
Duration = finding.Duration,
MemoryDecision = finding.Decision.ToWireValue(),
MemoryDecisionReason = finding.DecisionReason,
Expand Down Expand Up @@ -4388,6 +4394,8 @@ private void ProcessToolCallResult(Pipelines.ToolCallResult result)
AgentName = run.AgentName,
Phase = Netclaw.Actors.SubAgents.SubAgentPhase.Completed,
Success = run.Success,
Outcome = run.Outcome,
OutcomeReason = run.OutcomeReason,
Duration = run.Duration,
MemoryDecision = run.MemoryDecision,
MemoryDecisionReason = run.MemoryDecisionReason,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -318,6 +318,8 @@ public static async Task<ToolCallResult> ExecuteSingleToolAsync(
RunId = info.RunId,
AgentName = new SubAgents.AgentName(info.AgentName),
Success = info.Success,
Outcome = info.Outcome ?? (info.Success ? SubAgentRunOutcome.Completed : SubAgentRunOutcome.Failed),

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the handoff from subagent runtime metadata into the session event model. It keeps completion events aligned with the terminal tool envelope, while still defaulting legacy success/failure notifications when older paths do not provide an explicit outcome.

OutcomeReason = info.OutcomeReason,
Duration = info.Duration,
FindingsCount = info.Findings.Count,
MemoryDecision = decision,
Expand Down
Loading
Loading