fix(reminders): retain failed one-shot executions

Directory.Packages.props

@@ -5,7 +5,7 @@
     <!-- Nuget package versions -->
     <AkkaHostingVersion>1.5.70</AkkaHostingVersion>
     <AkkaPersistenceSqlHostingVersion>1.5.70</AkkaPersistenceSqlHostingVersion>
-    <AkkaRemindersVersion>0.6.0</AkkaRemindersVersion>
+    <AkkaRemindersVersion>0.7.0</AkkaRemindersVersion>
     <OpenTelemetryVersion>1.17.0</OpenTelemetryVersion>
     <SlackNetVersion>0.17.10</SlackNetVersion>
     <DiscordNetVersion>3.20.1</DiscordNetVersion>

feeds/skills/.system/files/netclaw-operations/SKILL.md

@@ -3,7 +3,7 @@ name: netclaw-operations
 description: "REQUIRED when the user asks about scheduling, reminders, cron jobs, timers, background jobs, diagnostics, troubleshooting, MCP tools, daemon health, identity updates, or Netclaw capabilities and self-maintenance."
 metadata:
   author: netclaw
-  version: "2.40.0"
+  version: "2.42.0"
 ---
 
 # Netclaw Operations

feeds/skills/.system/files/netclaw-operations/references/scheduling.md

@@ -62,27 +62,42 @@ Reminders that hit 5 consecutive execution failures are auto-disabled with a
 `ReminderAutoDisabled` critical alert. The definition stays on disk so the
 operator can diagnose and re-enable after fixing the root cause.
 
+A known execution or delivery failure starts the Akka.Reminders retry policy.
+The retry uses bounded backoff and the same durable occurrence identity. A
+successful attempt resets the consecutive failure count.
+
+A one-shot reminder stays enabled while an occurrence can retry. A successful
+one-shot becomes disabled with a `Completed` outcome. A poison one-shot becomes
+disabled with a `Failed` outcome. Both definitions and their history remain
+available until an operator uses the permanent delete command.
+
+Each attempt has a 20-minute inactivity limit and a one-hour absolute limit.
+The durable acknowledgement lease is 70 minutes. A daemon crash therefore lets
+Akka.Reminders retry the occurrence after the lease expires.
+
 **Failure visibility.** When a reminder execution fails for any reason — including
 the 20-minute stall backstop that recovers a wedged run — the failure is posted
 as a plain-language notice to the reminder's **destination channel** (for
 `channel`-delivery reminders), so the operator sees it where they expect that
 reminder's output. This is bounded by the auto-disable threshold (at most a few
 notices plus the disabled notice), not the unbounded skip stream.
 
-A *skipped* fire (one that arrives while the prior execution is still running) is
-**not** posted to the channel — it would be too noisy — but it is counted and
-surfaced by the status command:
+A one-shot that cannot start receives a negative acknowledgement. Akka.Reminders
+then controls its retry delay. Netclaw acknowledges and skips a blocked recurring
+occurrence. It does not keep a stale catch-up queue. The status command shows the
+skip count:
 
 ```
 netclaw reminder status <id>
 ```
 
-`status` shows, per reminder: whether it's enabled, whether an execution is in
-flight right now, when it next fires, the consecutive-failure count, the
-skipped-fire count (since daemon start), and recent run history. Reach for it
-when a reminder seems to have silently stopped doing its job — a high skip count
-means a prior run is wedged (it should self-recover within ~20 minutes), and a
-rising failure count points at a misconfigured or broken reminder.
+`status` shows the enabled state, the terminal outcome, and current execution
+state. It also shows the next fire, consecutive failures, skipped occurrence count,
+and recent history. For one-shots, it shows the durable occurrence state, attempt
+count, next retry time, and last failure reason.
+
+Use this command when a reminder stops its expected work. A failure count that
+increases usually means that the reminder or its delivery target is not healthy.
 
 If `audience` is omitted during conversational scheduling, the reminder inherits
 the audience of the channel/session that created it. A reminder cannot be

openspec/changes/reliable-one-shot-reminder-retry/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-08-07

openspec/changes/reliable-one-shot-reminder-retry/design.md

@@ -0,0 +1,94 @@
+## Context
+
+Akka.Reminders already persists each occurrence attempt, deadline, failure reason, and terminal state. Netclaw acknowledges channel and no-delivery reminders before their LLM session completes.
+
+A failed one-shot then has no active Akka occurrence. Reconciliation treats that absence as completion and deletes the definition and history.
+
+## Goals / Non-Goals
+
+**Goals:**
+
+- Use Akka.Reminders as the occurrence retry source of truth.
+- Preserve failed one-shots until retry success or a terminal failure.
+- Preserve the separate Netclaw poison threshold for the complete reminder.
+- Keep old reminder JSON files and the Akka.Reminders 0.6.0 schema compatible.
+
+**Non-Goals:**
+
+- Add a recurring catch-up queue.
+- Add a durable ingress queue for all session messages.
+- Change reminder trust or tool-policy derivation.
+
+## Decisions
+
+### Akka.Reminders owns occurrence retry state
+
+Netclaw uses the entity-bound `IReminderClient.NackAsync` method for a known failure. It uses `GetOccurrenceStatusAsync` for retry and terminal diagnostics.
+
+Netclaw does not copy an occurrence attempt count or retry timestamp into its definition file.
+
+### Netclaw owns reminder-level poison state
+
+Netclaw persists `ConsecutiveFailures` in each reminder definition. Each failed execution attempt increments this value, and a success resets it.
+
+This value spans recurring occurrences. Akka.Reminders resets its attempt count for each new occurrence.
+
+### The reminder manager coordinates settlement
+
+`ReminderManagerActor` passes the envelope to `ReminderExecutionActor` for all delivery kinds. The child reports its outcome and waits for manager acceptance.
+
+The manager saves the history and reminder state before it settles a known failure. It then sends the negative acknowledgement.
+
+The manager resets the reminder failure count before it acknowledges a success. It records one-shot completion only after a successful acknowledgement.
+
+The manager replies to the child after settlement. The child stops only after this reply, so DeathWatch cannot replace an accepted result.
+
+An actor crash before an outcome leaves the occurrence unacknowledged. The manager records the crash and attempts a negative acknowledgement without risking its own lifecycle.
+
+### Capacity does not transfer occurrence ownership
+
+Netclaw does not retain a blocked Akka.Reminders envelope in an in-memory queue. A queue wait could consume the 70-minute acknowledgement lease.
+
+Netclaw negatively acknowledges a blocked one-shot. Akka.Reminders then owns its retry delay and attempt budget.
+
+Netclaw acknowledges and skips a blocked reminder-series occurrence. This rule prevents a catch-up queue and preserves the latest-only series policy.
+
+Netclaw ignores an exact duplicate of the active occurrence. The active execution remains the sole settlement owner.
+
+### One-shot completion uses a soft delete
+
+A successful one-shot sets `Enabled` to false and records `TerminalOutcome.Completed`. A poison or terminal one-shot records `TerminalOutcome.Failed`.
+
+The explicit delete command remains the only normal hard-delete path. It also deletes the history file.
+
+### Reconciliation uses durable state
+
+Reconciliation never uses a past fire time or a missing active schedule as proof of success. It retains disabled one-shots and restores an enabled one-shot when durable state permits another attempt.
+
+### Timeouts remain bounded
+
+Netclaw sets the Akka acknowledgment timeout to 70 minutes. The execution actor applies a one-hour absolute attempt limit and keeps its 20-minute inactivity limit.
+
+Known failures use negative acknowledgement and do not wait for the acknowledgment timeout.
+
+Netclaw starts an attempt only when the remaining envelope lease exceeds the maximum attempt duration plus a settlement margin.
+
+## Risks / Trade-offs
+
+- **A daemon crash can delay retry for 70 minutes.** The long lease prevents duplicate LLM work during a valid one-hour attempt.
+- **At-least-once delivery can duplicate work.** The occurrence identity remains `(Entity, Key, DueTimeUtc)` and session identifiers use that stable due time.
+- **Netclaw and Akka.Reminders use separate stores.** Ordered writes and reconciliation provide convergence without a cross-store transaction.
+- **A custom Akka storage provider can lack status queries.** Netclaw uses the official SQLite provider and fails loudly if the capability is absent.
+- **Old JSON files lack the new fields.** Serializer defaults preserve active state and a zero failure count.
+
+## Migration Plan
+
+1. Release Akka.Reminders 0.7.0 with the additive delivery-control API.
+2. Upgrade Netclaw to both 0.7.0 packages.
+3. Load old reminder JSON files with default values.
+4. Keep the existing SQLite schema without a migration.
+5. Roll back Netclaw by restoring the prior binary. The added JSON fields remain harmless to older readers.
+
+## Open Questions
+
+None.

openspec/changes/reliable-one-shot-reminder-retry/proposal.md

@@ -0,0 +1,34 @@
+## Why
+
+PRD-008 requires durable failure records and an automatic pause after repeated failures. Netclaw now deletes a failed one-shot before Akka.Reminders can retry it.
+
+## What Changes
+
+- Netclaw will acknowledge every reminder only after successful execution and required delivery.
+- Netclaw will report a known failure through the Akka.Reminders negative acknowledgement API.
+- A failed one-shot will remain enabled while another occurrence attempt is pending.
+- A completed or terminally failed one-shot will use a soft delete.
+- Netclaw will persist its reminder-level consecutive failure count.
+- The reminder manager will coordinate local state and Akka occurrence settlement.
+- Netclaw will not keep Akka.Reminders envelopes in an in-memory catch-up queue.
+- Reconciliation will use durable occurrence state and will never infer success from a past due time.
+- Reminder status output will show the durable occurrence attempt and terminal outcome.
+
+## Capabilities
+
+### New Capabilities
+
+None.
+
+### Modified Capabilities
+
+- `netclaw-scheduling`: Change acknowledgement, retry, poison-reminder, one-shot retention, and reconciliation requirements.
+- `reminder-execution-history`: Retain execution history for soft-deleted one-shot reminders.
+
+## Impact
+
+- **In scope:** PRD-008 reminder execution, reminder status, reconciliation, the definition store, and Akka.Reminders 0.7.0 integration.
+- **Out of scope:** A catch-up queue for recurring occurrences and a general durable ingress queue for all sessions.
+- **Security:** The change keeps the current trust context and tool policy for every retry.
+- **Operations:** Operators can inspect failed one-shots and retry state after a daemon restart.
+- **Compatibility:** Existing reminder JSON files load with default values. The Akka.Reminders 0.6.0 database schema remains valid.