Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions docs/runbooks/background-jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,19 @@ is notified** with the log path so the agent can relaunch. Notification volume
is bounded by design: passivated sessions have no live jobs, so only sessions
that were warm at crash time appear here.

### Terminal artifact retention

The manager retains each terminal definition and its output directory for 24
hours after completion. The policy applies to `Completed`, `Failed`,
`Cancelled`, `TimedOut`, `Lost`, and `Reaped` jobs.

The manager starts an hourly cleanup sweep after startup reconciliation. The
sweep deletes the output directory before it deletes the definition. A file
error keeps the definition and lets the next sweep retry the cleanup.

A terminal definition without `CompletedAtMs` is corrupt. The sweep reports
the problem and retains its artifacts instead of inferring a deletion time.

## Monitoring

### Active jobs in context
Expand Down Expand Up @@ -111,6 +124,9 @@ Returns: status, elapsed time, rationale, exit code (if finished), and the
live output tail (last 2000 chars, read from the streaming log). Only
accessible from the same session/audience/boundary that submitted the job.

The tool can query a terminal job during its 24-hour retention window. After
that window, the tool reports that it cannot find the job.

This tool is only available when shell execution is granted (same `shell` grant
category as `shell_execute`).

Expand Down
3 changes: 2 additions & 1 deletion feeds/skills/.system/files/netclaw-operations/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: netclaw-operations
description: "REQUIRED when the user asks about scheduling, reminders, cron jobs, timers, background jobs, diagnostics, troubleshooting, MCP tools, daemon health, identity updates, or Netclaw capabilities and self-maintenance."
metadata:
author: netclaw
version: "2.42.0"
version: "2.44.0"
---

# Netclaw Operations
Expand Down Expand Up @@ -104,6 +104,7 @@ view inline plus a pointer to the full output — not the whole thing:
`shell_execute` to get around it — that just spills again.
- **`background_job`** output goes to `~/.netclaw/jobs/{id}/output.log` (bounded);
`check_background_job` returns a tail, and you can `file_read`/`grep` the log for the rest.
Netclaw deletes a terminal job's definition and logs 24 hours after completion.

Reading a targeted range or grepping is always cheaper than re-running a command or
re-reading a whole file. Secret-bearing values are redacted from all tool output.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -66,10 +66,11 @@ A known execution or delivery failure starts the Akka.Reminders retry policy.
The retry uses bounded backoff and the same durable occurrence identity. A
successful attempt resets the consecutive failure count.

A one-shot reminder stays enabled while an occurrence can retry. A successful
one-shot becomes disabled with a `Completed` outcome. A poison one-shot becomes
disabled with a `Failed` outcome. Both definitions and their history remain
available until an operator uses the permanent delete command.
A one-shot reminder stays enabled while an occurrence can retry. After a
successful acknowledgement, Netclaw deletes its definition and history. A poison
one-shot becomes disabled with a `Failed` outcome. Its definition and history
remain available until an operator uses the permanent delete command.
Startup reconciliation also removes completed one-shots from prior versions.

Each attempt has a 20-minute inactivity limit and a one-hour absolute limit.
The durable acknowledgement lease is 70 minutes. A daemon crash therefore lets
Expand Down Expand Up @@ -254,6 +255,8 @@ Lifecycle:
notification with the log path — relaunch if still needed.
- Process exit (success or failure) delivers a result turn with exit code,
output tail, and log path — even if the session was passivated mid-flight.
- Netclaw retains every terminal job definition and its logs for 24 hours after
completion. The hourly cleanup sweep then deletes both artifact types.

Monitoring a running job (e.g. waiting for a dev server to come up):

Expand Down Expand Up @@ -282,7 +285,8 @@ Rules:
model server that takes minutes to respond) should run as background jobs.
- The user must approve the command before it starts running in the background.
- Maximum 5 concurrent background jobs; overflow queues FIFO.
- Job definitions persist to `~/.netclaw/jobs/{id}.json`.
- Job definitions persist to `~/.netclaw/jobs/{id}.json` until 24 hours after
the job reaches a terminal state.

`check_background_job` is only available when shell execution is granted (same
`shell` grant category). It validates that the requesting session matches the
Expand Down
185 changes: 185 additions & 0 deletions src/Netclaw.Actors.Tests/Jobs/BackgroundJobDefinitionStoreTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,191 @@ public void Current_job_with_trust_fields_roundtrips_exact_values()
Assert.Equal("C0ABC/1712000000.000001", loaded.SessionId.Value);
}

/// <summary>
/// Terminal-job cleanup: <see cref="BackgroundJobDefinitionStore.DeleteJobArtifacts"/>
/// removes BOTH the definition file and the job's output-log directory, so the
/// store cannot grow without bound after a job's retention window elapses.
/// </summary>
[Fact]
public void DeleteJobArtifacts_removes_definition_and_output_directory()
{
var store = new BackgroundJobDefinitionStore(_paths);
var jobId = new BackgroundJobId("cleanup-job-001");

store.Save(new BackgroundJobDefinition
{
Id = jobId,
Command = "dotnet test",
SessionId = new Netclaw.Actors.Protocol.SessionId("C0ABC/1712000000.000001"),
Rationale = "Run the test suite.",
Status = BackgroundJobStatus.Completed,
TimeoutSeconds = 300,
Audience = TrustAudience.Team,
Boundary = TrustBoundary.Team,
OriginChannelType = Netclaw.Actors.Channels.ChannelType.Slack
});

// Simulate a real job's output log on disk.
var outputLogPath = store.GetOutputLogPath(jobId);
File.WriteAllText(outputLogPath, "build output");

Assert.NotNull(store.Get(jobId));
Assert.True(File.Exists(outputLogPath));

var removed = store.DeleteJobArtifacts(jobId);

Assert.True(removed);
Assert.Null(store.Get(jobId));
Assert.False(File.Exists(outputLogPath));
Assert.False(Directory.Exists(Path.GetDirectoryName(outputLogPath)));
}

/// <summary>
/// Idempotent cleanup: deleting artifacts for an already-removed job reports
/// false and does not throw.
/// </summary>
[Fact]
public void DeleteJobArtifacts_missing_job_returns_false_without_throwing()
{
var store = new BackgroundJobDefinitionStore(_paths);

var removed = store.DeleteJobArtifacts(new BackgroundJobId("never-existed"));

Assert.False(removed);
}

[Fact]
public void DeleteJobArtifacts_keeps_definition_when_output_cleanup_fails_then_retries()
{
var store = new BackgroundJobDefinitionStore(_paths);
var jobId = new BackgroundJobId("cleanup-retry-001");
store.Save(new BackgroundJobDefinition
{
Id = jobId,
Command = "dotnet test",
SessionId = new Netclaw.Actors.Protocol.SessionId("C0ABC/1712000000.000001"),
Rationale = "Run the test suite.",
Status = BackgroundJobStatus.Completed,
TimeoutSeconds = 300,
Audience = TrustAudience.Team,
Boundary = TrustBoundary.Team,
OriginChannelType = Netclaw.Actors.Channels.ChannelType.Slack
});

var outputLogPath = store.GetOutputLogPathOnly(jobId);
var outputDirectory = Path.GetDirectoryName(outputLogPath)!;
File.WriteAllText(outputDirectory, "path collision");

var error = Assert.Throws<IOException>(() => store.DeleteJobArtifacts(jobId));

Assert.Contains("is not a directory", error.Message);
Assert.NotNull(store.Get(jobId));
Assert.True(File.Exists(outputDirectory));

File.Delete(outputDirectory);
File.WriteAllText(store.GetOutputLogPath(jobId), "build output");

Assert.True(store.DeleteJobArtifacts(jobId));
Assert.Null(store.Get(jobId));
Assert.False(Directory.Exists(outputDirectory));
}

/// <summary>
/// Traversal guard (adversarial review, HIGH): Uri.EscapeDataString does not
/// escape dots, so a dot-only id must NOT resolve to the jobs directory's
/// parent and delete it recursively. The store rejects such ids at
/// deserialization, and DeleteJobArtifacts contains the delete to the jobs
/// directory as belt-and-braces.
/// </summary>
[Fact]
public void DeleteJobArtifacts_dot_only_id_cannot_escape_jobs_directory()
{
var store = new BackgroundJobDefinitionStore(_paths);
var jobsDir = _paths.JobsDirectory;
var parentDir = Path.GetDirectoryName(jobsDir.TrimEnd(Path.DirectorySeparatorChar))!;
var sentinel = Path.Combine(parentDir, "sentinel-file.txt");
File.WriteAllText(sentinel, "do not delete");

// ".." would resolve to the parent of the jobs directory.
var removed = store.DeleteJobArtifacts(new BackgroundJobId(".."));

// The delete must be refused (nothing removed) — the parent survives.
Assert.False(removed);
Assert.True(File.Exists(sentinel));
Assert.True(Directory.Exists(jobsDir));

File.Delete(sentinel);
}

/// <summary>
/// Traversal guard (adversarial review, HIGH): a persisted definition whose
/// id is "." or ".." must be rejected at load — it never appears in
/// List(), so the sweep can never act on it.
/// </summary>
[Theory]
[InlineData(".")]
[InlineData("..")]
public void Definition_with_dot_only_id_is_rejected_at_load(string unsafeId)
{
var logger = new CapturingJobLogger<BackgroundJobDefinitionStore>();
var store = new BackgroundJobDefinitionStore(_paths, logger);

// File name mirrors what GetPath would produce for this id:
// Uri.EscapeDataString leaves dots unescaped, so the file is "..json".
var filePath = Path.Combine(_paths.JobsDirectory, $"{Uri.EscapeDataString(unsafeId)}.json");
File.WriteAllText(filePath, $$"""
{
"id": "{{unsafeId}}",
"command": "echo pwn",
"sessionId": "C0TEST/1712000000.000001",
"rationale": "test",
"status": "Completed",
"timeoutSeconds": 600,
"startedAtMs": 0,
"completedAtMs": 0,
"audience": "Personal",
"boundary": "Personal"
}
""");

Assert.Empty(store.List());
Assert.Null(store.Get(new BackgroundJobId(unsafeId)));
Assert.Contains(logger.Errors, e => e.Contains("unsafe id"));
}

[Fact]
public void Definition_with_id_that_does_not_match_file_name_is_rejected_at_load()
{
var logger = new CapturingJobLogger<BackgroundJobDefinitionStore>();
var store = new BackgroundJobDefinitionStore(_paths, logger);
var victimId = new BackgroundJobId("victim-job");
store.Save(new BackgroundJobDefinition
{
Id = victimId,
Command = "dotnet test",
SessionId = new Netclaw.Actors.Protocol.SessionId("C0ABC/1712000000.000001"),
Rationale = "Run the test suite.",
Status = BackgroundJobStatus.Completed,
StartedAtMs = 1,
CompletedAtMs = 2,
Audience = TrustAudience.Team,
Boundary = TrustBoundary.Team,
OriginChannelType = Netclaw.Actors.Channels.ChannelType.Slack
});

var victimPath = Path.Combine(_paths.JobsDirectory, $"{Uri.EscapeDataString(victimId.Value)}.json");
var aliasPath = Path.Combine(_paths.JobsDirectory, "stale-alias.json");
File.Copy(victimPath, aliasPath);

var loaded = store.List();

var definition = Assert.Single(loaded);
Assert.Equal(victimId, definition.Id);
Assert.Null(store.Get(new BackgroundJobId("stale-alias")));
Assert.Contains(logger.Errors, error =>
error.Contains("does not match its canonical file name", StringComparison.Ordinal));
}

/// <summary>
/// Byte-equality gate for issue #994 Pass 7b. Wrapping <c>BackgroundJobDefinition.Id</c>
/// in <see cref="BackgroundJobId"/> and <c>SessionId</c> in
Expand Down
Loading
Loading