fix(reminders): retain failed one-shot executions by Aaronontheweb · Pull Request #1812 · netclaw-dev/netclaw

Aaronontheweb · 2026-08-08T00:00:56Z

Summary

Delay the occurrence acknowledgement until execution and required delivery succeed.
Send a negative acknowledgement after a known execution or delivery failure.
Keep the Akka.Reminders retry state as the durable occurrence source of truth.
Preserve the Netclaw consecutive failure count across restarts.
Disable completed or terminal one-shot reminders without removal.
Expose retry and terminal details through reminder status.
Use a 70-minute acknowledgement lease for a one-hour execution limit.

Dependency

Aaron.Akka.Reminders 0.7.0
Aaron.Akka.Reminders.Sqlite 0.7.0
Akka.Reminders PR: Add negative acknowledgement and occurrence status APIs Aaronontheweb/akka-reminders#135

Validation

Public NuGet restore passed with a fresh package directory and no HTTP cache.
Netclaw.Actors.Tests: 2,860 passed.
Full solution: 6,418 passed and 14 opt-in tests skipped.
Slopwatch found no issues.
The file-header check passed.
Strict OpenSpec validation passed.
The eval suite could not start because the required NETCLAW_EVAL provider credentials were absent.

Closes #1803

Aaronontheweb · 2026-08-08T13:52:12Z

+        if (_activeExecutions.TryGet(reminderId, out var activeExecution))
+        {
+            RecordSkippedDuplicate(reminderId, definition.Title, "active");
+            if (IsSameDeliveryAttempt(activeExecution.Envelope, envelope))


An exact redelivery has the same key, due time, and deadline. The active child still owns settlement, so this handler sends no Ack or Nack.

Aaronontheweb · 2026-08-08T13:52:50Z

    <AkkaHostingVersion>1.5.70</AkkaHostingVersion>
    <AkkaPersistenceSqlHostingVersion>1.5.70</AkkaPersistenceSqlHostingVersion>
-    <AkkaRemindersVersion>0.6.0</AkkaRemindersVersion>
+    <AkkaRemindersVersion>0.7.0</AkkaRemindersVersion>


Akka.Reminders 0.7.0 supplies NackAsync and the durable occurrence-status query. Netclaw uses both APIs for retry control and reconciliation.

Aaronontheweb · 2026-08-08T13:52:52Z

+    /// Number of consecutive failed execution attempts for this reminder.
+    /// A successful attempt resets this value.
+    /// </summary>
+    public int ConsecutiveFailures { get; set; }


These fields use the existing JSON reminder definition. Missing fields keep safe defaults, so old definitions load without a database migration.

Aaronontheweb · 2026-08-08T13:52:53Z

            return new ReminderStateResponse(id, Found: false, Enabled: false, ErrorMessage: "Reminder not found.");

-        definition = definition with
+        var candidate = definition with


This candidate prevents a failed enable request from erasing stored diagnostics. The store changes only after Akka.Reminders accepts the schedule.

Aaronontheweb · 2026-08-08T13:52:54Z

+            await SettleBlockedOccurrenceAsync(
+                definition,
+                envelope,
+                nack: definition.Schedule.Type == ReminderScheduleType.OneShot || sameOccurrence,


A one-shot or the same occurrence receives a Nack for retry. A later recurring occurrence receives an Ack and is skipped.

Aaronontheweb · 2026-08-08T13:52:55Z

+            await SettleBlockedOccurrenceAsync(
+                definition,
+                envelope,
+                nack: definition.Schedule.Type == ReminderScheduleType.OneShot,


Netclaw does not keep a deferred queue. A blocked one-shot returns to Akka.Reminders, while a recurring occurrence is skipped.

Aaronontheweb · 2026-08-08T13:52:56Z

+        ReminderEnvelope<ReminderPayload> envelope,
+        DateTimeOffset now) =>
+        envelope.Deadline.IsInfinite
+        || envelope.Deadline.UtcDateTime - now >= ReminderExecutionActor.ExecutionAttemptTimeout + SettlementMargin;


The Ack lease must contain the one-hour attempt and the settlement margin. This guard rejects work that cannot settle safely.

Aaronontheweb · 2026-08-08T13:52:57Z

+        finally
+        {
+            _activeExecutions.TryRemove(outcome.Id, outcome.ExecutionId, out _);
+            replyTo.Tell(new ReminderExecutionAccepted(outcome.ExecutionId));


This reply completes the settlement handshake. The child stays alive until the manager handles persistence and Ack or Nack.

Aaronontheweb · 2026-08-08T13:52:58Z

+    private async Task HandleExecutionTerminatedAsync(ReminderExecutionTerminated terminated)
    {
-        if (!_activeExecutions.Remove(completed.Id))
+        if (!_activeExecutions.TryRemove(terminated.Id, terminated.ExecutionId, out var execution))


DeathWatch covers a child that exits before the handshake. The execution ID prevents a stale notice from settling a newer attempt.

Aaronontheweb · 2026-08-08T13:52:59Z

+        {
+            try
+            {
+                _definitionStore.Save(definition with


The manager writes local success state before external settlement. This order prevents an Ack from hiding a later definition-store failure.

Aaronontheweb · 2026-08-08T13:53:01Z

-                        ["title"] = title,
-                        ["failureCount"] = count.ToString(),
-                    }));
+                _definitionStore.Save(definition with


The one-shot becomes disabled only after Ack returns success. Its definition and history remain available for status queries.

Aaronontheweb · 2026-08-08T13:53:02Z

+                    TerminalOutcome = thresholdReached ? ReminderTerminalOutcome.Failed : definition.TerminalOutcome,
+                    UpdatedAtMs = _timeProvider.GetUtcNow().ToUnixTimeMilliseconds()
+                };
+                _definitionStore.Save(definition);


The manager saves the failed attempt before it sends the Nack. This preserves diagnostics if Akka.Reminders settlement fails.

Aaronontheweb · 2026-08-08T13:53:03Z

+                EmitSettlementFailure(definition, ex.Message, ex);
+        }
+
+        var occurrenceTerminal = nack?.ResponseCode is ReminderNackResponseCode.Failed


Failed and Expired mean that the occurrence exhausted its Akka.Reminders budget. Netclaw then disables the definition.

Aaronontheweb · 2026-08-08T13:53:04Z

            {
-                await DeleteReminderInternalAsync(definition.Id);
-                deletedOneShots++;
+                var occurrence = await GetOccurrenceStatusAsync(definition);


Reconciliation now checks the durable occurrence result. A past due time and an absent active schedule no longer prove success.

Aaronontheweb · 2026-08-08T13:53:05Z

            var schedulesTask = ListScheduledRemindersAsync();
            var historyTask = _historyStore.ReadAsync(query.Id, RecentHistoryCount);
-            await Task.WhenAll(schedulesTask, historyTask);
+            var occurrenceTask = GetOccurrenceStatusAsync(definition);


Status combines the Netclaw definition, recent history, and the Akka.Reminders occurrence. Operators can distinguish retries from terminal failure.

Aaronontheweb · 2026-08-08T13:53:07Z


-        // Drain stream stages before stopping so they complete gracefully
-        // rather than being abruptly terminated as actor children.
        RunTask(async () =>


The child drains and stops only after manager acceptance. This handshake makes DeathWatch a fallback for an unexpected exit.

Aaronontheweb · 2026-08-08T13:53:09Z

            {
+                reminders.WithSettings(new ReminderSettings
+                {
+                    AckTimeout = ReminderAckTimeout


The 70-minute lease exceeds the one-hour execution limit. The manager reserves one additional minute before it starts an attempt.

Aaronontheweb · 2026-08-08T13:53:25Z

            _executionId,
            _definition.Id,
            success,
+            history,


The child reports history and outcome but sends no Ack or Nack. The singleton manager owns every settlement decision.

Aaronontheweb

LGTM - no more hard deletes of completed reminders.

Aaronontheweb · 2026-08-08T14:46:42Z

+The retry uses bounded backoff and the same durable occurrence identity. A
+successful attempt resets the consecutive failure count.
+
+A one-shot reminder stays enabled while an occurrence can retry. A successful


Aaronontheweb · 2026-08-08T14:50:01Z

+        var loaded = new ReminderDefinitionStore(_paths).Get(definition.Id);
+
+        Assert.NotNull(loaded);
+        Assert.Equal(5, loaded.ConsecutiveFailures);


Demonstrates that we save the failure execution data

Aaronontheweb · 2026-08-08T14:50:13Z

+    }
+
+    [Fact]
+    public void Definition_without_failure_fields_loads_with_active_defaults()


backwards compat test

Aaronontheweb · 2026-08-08T14:58:02Z

    }

+    [Fact]
+    public async Task Execution_waits_for_manager_acceptance_before_stop()


Aaronontheweb · 2026-08-08T15:03:17Z

Removed the duplicate SourceWithQueue Akka.Streams setup here - not necessary to have.

Aaronontheweb · 2026-08-08T15:04:14Z

 }
+
+internal sealed record ActiveReminderExecution(
+    Guid ExecutionId,


set by the ReminderManagerActor when execution begins

Aaronontheweb · 2026-08-08T15:10:39Z

        _deliveryTimeoutCancelable?.Cancel();
        _deliveryTimeoutCancelable = null;
-
-        if (_pendingHistory is not null)


Execution history is now owned by the ReminderManagerActor and not the ReminderExecutionActor

Aaronontheweb · 2026-08-08T15:16:45Z

    private IReminderClient? _client;

    private readonly ActiveExecutionTracker _activeExecutions = new();
-    private readonly Queue<ReminderId> _deferredQueue = new();


Essentially, we were competing with Akka.Reminders' own queue management for reminder deliveries here. We are now delegating more of that TCO to Akka.Reminders directly now.

Aaronontheweb · 2026-08-08T15:34:00Z

+        AkkaReminderProtocol.ReminderNackResponse? nack = null;
+        try
+        {
+            nack = await _client!.NackAsync(execution.Envelope, reason);


proactively let Akka.Reminders know that this failed reminder needs re-delivery

…ity (#1812 regression) (#1839) The capacity gate (MaxConcurrentExecutions = 3) settled blocked occurrences instead of deferring them, and when the settlement ack raced with the cron reschedule it produced a bogus reminder.settlement.failed alert. Worse, it skipped real reminders (e.g. the daily vendor-price audit) when other reminders were parked on long-running executions. Every execution already has a one-hour absolute timeout and Akka.Reminders owns failure retry, so unbounded scheduling pressure on the LLM is acceptable. Remove the cap entirely - no capacity skip, no settle, no alert.

fix(reminders): retain failed one-shot executions

52aea2f

Aaronontheweb added reminders Reminder scheduling, execution, and history reliability Retries, resilience, graceful degradation labels Aug 8, 2026

Aaronontheweb added 3 commits August 8, 2026 01:09

fix(reminders): coordinate occurrence settlement

c207cec

Merge branch 'dev' into fix/one-shot-retry-retention

2c64957

Merge branch 'dev' into fix/one-shot-retry-retention

22a0d10

Aaronontheweb commented Aug 8, 2026

View reviewed changes

Comment thread src/Netclaw.Actors/Channels/SessionPipelineHandle.cs Outdated

Aaronontheweb commented Aug 8, 2026

View reviewed changes

Aaronontheweb marked this pull request as ready for review August 8, 2026 13:54

Aaronontheweb added 2 commits August 8, 2026 14:18

refactor(sessions): use channel input for executions

ea764a0

fix(reminders): clarify delayed occurrence logs

b6d909f

Aaronontheweb commented Aug 8, 2026

View reviewed changes

Aaronontheweb enabled auto-merge (squash) August 8, 2026 15:34

Aaronontheweb merged commit 1fef636 into netclaw-dev:dev Aug 8, 2026
15 checks passed

This was referenced Aug 8, 2026

Clean up completed background jobs and one-shot reminders that are no longer needed #1820

Closed

fix(reminders): remove execution cap so reminders never skip on capacity #1839

Merged

Aaronontheweb mentioned this pull request Aug 20, 2026

Archive completed OpenSpec changes and sync their specs #2024

Merged

Uh oh!

Conversation

Aaronontheweb commented Aug 8, 2026

Summary

Dependency

Validation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Aaronontheweb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant