actor: durable mailbox backpressure watermarks + depth metrics - #1121
actor: durable mailbox backpressure watermarks + depth metrics#1121Roasbeef wants to merge 12 commits into
Conversation
In this commit, we give the durable mailbox the capacity signal it never had. A channel mailbox bounds its queue with ErrMailboxFull, but a durable mailbox lands every Tell as a database row, so a consumer that stops draining grows its backlog for as long as the outage lasts with nothing pushing back on producers. Two thresholds on DurableMailboxConfig (flowing through DurableActorConfig) now bound the persistent backlog. Past the soft watermark the mailbox logs one warning per breach episode; at or past the hard watermark, Send refuses the message with the new ErrMailboxSaturated sentinel. The check runs before encoding or promise registration, so a refusal needs no cleanup and TrySend/TryTell inherit it for free. Messages with priority >= RestartPriority are always exempt: the RestartMessage that would un-wedge a stuck actor must not be refused by the very backlog it exists to drain. Depth is read through the new MailboxDepthStore surface, a narrow optional interface discovered by type assertion on the DeliveryStore (the same pattern keeps test doubles small). The read is TTL-cached at one second with a local count of sends accepted since the last probe added on top, so the common send path pays no extra query. The estimate is deliberately one-sided: local sends push it up immediately while acks only surface at the next probe, and overshooting is the safe direction for an admission check. A failed probe fails OPEN, since a broken monitoring read must not become message loss. Both thresholds default to zero (disabled), so existing actors are byte-for-byte unchanged until a site opts in.
In this commit, we add the two backlog reads behind the new watermark admission check and the depth scrape gauges. CountMailboxMessages is a COUNT(*) over one mailbox's rows, leased or not: rows are deleted on ack, so the count is exactly the undelivered backlog, and the prefix of idx_mailbox_messages_available covers the equality scan. CountMailboxMessagesByMailbox is the GROUP BY variant for scrape time; mailboxes with an empty backlog produce no row, so the result stays bounded by the number of backed-up actors rather than the number of actors that have ever existed.
In this commit, we regenerate the actor delivery query layer via make sqlc to pick up CountMailboxMessages and CountMailboxMessagesByMailbox. No handwritten changes.
In this commit, we wire the depth queries into the Store as the actor.MailboxDepthStore surface: MailboxDepth for the watermark admission check and MailboxDepths for the scrape gauges, both plain read transactions mapped through the widened ActorDeliveryQueries interface. A compile-time assertion pins the implementation so the durable mailbox's type assertion can never silently stop matching. The tests pin the two facts the watermark check depends on: a leased (in-flight) row still counts toward depth until its ack deletes it, and a fully drained mailbox drops out of the grouped listing entirely.
In this commit, we surface the backlog the watermarks bound. The
SystemCollector gains two scrape-time gauges read off the delivery
store's MailboxDepthStore surface: waved_mailbox_backlog, an unlabelled
total that emits an explicit zero when every mailbox is drained (so
"all clear" and "scrape broke" stay distinguishable), and
waved_mailbox_depth{mailbox_id}, one series per mailbox currently
holding messages. Reporting only backed-up mailboxes keeps cardinality
proportional to live trouble: per-session actor IDs never accumulate
as permanent series, the same posture the OOR/round gauges take.
The waved systemStatsAdapter resolves the depth surface with a type
assertion, so a delivery store without it skips the gauges rather than
failing the scrape. The README gains the reference rows plus an
alerting recipe keyed to the soft/hard watermark defaults, and the
ingress deferral counter's description now points at waved_mailbox_depth
to tell a durable saturation refusal apart from a full in-memory
mailbox.
In this commit, we teach the ingress dispatch path that a durable target can refuse for want of room. deliverToActor now classifies actor.ErrMailboxSaturated exactly like actor.ErrMailboxFull: the envelope comes back as ErrDispatchDeferred, unacknowledged, and the loop re-pulls it after a backoff. A durable actor whose backlog crossed its hard watermark therefore stalls the cursor and exerts backpressure on the operator stream instead of deepening its own backlog, which is precisely the behavior the watermark asks for. The deferral machinery downstream needs no change: the episode logging, the deferred counter, and the redrive loop all key off the ErrDispatchDeferred sentinel. We update the dispatch contract docs (doc.go, EnvelopeDispatcher) to name the second refusal shape.
In this commit, we opt the six durable actor construction sites into the shared default watermarks (soft 1000, hard 10000): the serverconn egress sender, the OOR registry and per-session actors, and the ledger, credit, and unroll actors. The framework default stays disabled, so the diff is the complete inventory of which mailboxes now shed load. The consequences differ by who produces into each mailbox. Ingress-fed actors (the OOR registry and sessions) get deferral semantics: the serverconn cursor stalls at the refused envelope and re-pulls, so saturation becomes backpressure on the operator stream. Locally-fed actors surface ErrMailboxSaturated to their producers, which at ten thousand parked messages is the only honest answer. For egress, an operator unreachable long enough to park that many events now fails producers loudly instead of accreting an unbounded replay queue.
In this commit, we add a Backpressure Watermarks section to the durable actor architecture doc covering the semantics (soft warns, hard refuses with ErrMailboxSaturated, restart priority exempt), the TTL-cached probe and its one-sided estimate, who refuses and what each caller class does with it, and the deliberate exemption of the OutboxPublisher folded delivery path. The per-package CLAUDE/AGENTS pairs for baselib/actor, serverconn, metrics, and db/actordelivery pick up the new surfaces.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 765d424678
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if err := m.checkWatermarks(ctx, priority); err != nil { | ||
| return err |
There was a problem hiding this comment.
When an outbox targets any actor that opted into the hard watermark, OutboxPublisher.deliverMessage still calls ref.Tell, so the folded path reaches this check and returns ErrMailboxSaturated. Each retry increments DeliveryAttempts, and after the default 10 attempts the publisher dead-letters the event; a temporary backlog can therefore permanently lose a durable outbox message instead of using the deliberately unthrottled folded enqueue. Exempt folded sends (for example via their outbox context) from this admission check.
AGENTS.md reference: baselib/actor/AGENTS.md:L73-L74
Useful? React with 👍 / 👎.
| // watermarks so a wedged consumer sheds load at the producer instead | ||
| // of growing its backlog without bound. | ||
| durableCfg.SoftHighWatermark = actor.DefaultSoftHighWatermark | ||
| durableCfg.HardHighWatermark = actor.DefaultHardHighWatermark |
There was a problem hiding this comment.
Once the ledger backlog reaches 10,000, this setting makes new accounting Tells fail with ErrMailboxSaturated, but several existing producers intentionally only log Tell failures and continue, including wallet UTXO confirmations and round ledger emissions. Those entries are never retried, so after the ledger recovers its audit trail and balances remain permanently incomplete; the ledger needs either no hard shedding or a durable redrive path for every producer.
AGENTS.md reference: ledger/AGENTS.md:L154-L161
Useful? React with 👍 / 👎.
| mailboxDepthDesc, prometheus.GaugeValue, | ||
| float64(row.Depth), row.MailboxID, | ||
| ) |
There was a problem hiding this comment.
Using the full mailbox ID as a Prometheus label creates a new time series for every distinct durable actor whose queue is observed nonempty. OOR session mailbox IDs embed a unique session hash (oor-session-<txid>), and Prometheus retains a series after it disappears from later scrapes, so completed sessions accumulate cardinality over the retention window rather than being removed as the comment claims. Aggregate dynamic mailboxes by actor type or expose per-ID depth through a non-Prometheus diagnostic surface.
Useful? React with 👍 / 👎.
In this commit, we address three findings from the adversarial review of the watermark admission check. First, outbox-propagated deliveries are now exempt. The CDC hand-off was NOT bypassing DurableMailbox.Send as the original docs claimed: the publisher's deliver path goes ref.Tell -> mailbox.Send, so a saturated target refused committed outbox rows. Because ClaimOutboxBatch bumps delivery attempts in its own transaction, roughly ten refused claims would dead-letter the row (and any DurableAsk response it carries) instead of exerting backpressure. The publisher stamps the outbox ID into the context on every folded delivery, so the check keys off exactly that marker: an outbox-propagated send is admitted uncheck. Second, the exemption threshold drops from RestartPriority to the new ControlPriority (MaxInt32 - 1), making room for domain-level control messages, boot restores and resumes, that the daemon treats as fatal on failure. Without this, a backlog pinned at the hard mark at boot would refuse the restore Ask and turn a wedged consumer into a daemon-wide restart crash loop, with the durable backlog guaranteeing every subsequent boot fails the same way. Third, the probe itself is hardened: it now runs single-flighted outside the mutex (concurrent senders use the cached estimate instead of stacking behind one COUNT), and outside the sender's ambient transaction via WithoutTx. TransactionExecutor.ExecTx joins any ambient tx and ignores the read-only option, so the old probe executed its whole-mailbox COUNT inside the sender's SERIALIZABLE writer, taking predicate locks that manufacture rw-conflicts with the consumer's acks at precisely the moment the system is contended. A probe failure now falls back to the cached estimate rather than admitting unchecked, and only fails open when no baseline exists at all. The soft-watermark episode is also evaluated before the hard refusal, so a backlog that enters saturation within one probe window still fires the operator's early-warning line instead of failing sends silently first.
In this commit, we stamp actor.ControlPriority onto the three boot-path control messages that flow into watermarked durable mailboxes: RestoreNonTerminalRequest (OOR registry), ResumeUnrollRequest (per-target unroll actors), and ResumeCreditOpRequest (per-operation credit actors). All three were priority 0, and the first two are delivered as Asks whose failure waved treats as fatal at startup, so a backlog past the hard watermark at boot would have refused the very message that exists to work through that backlog and crash-looped the daemon. Control priority both bypasses the admission check and claims ahead of the redelivered backlog, mirroring RestartMessage's restart-first semantics. For OOR, restore-before-backlog is safe because routing and restore converge on ensureChild's dedup either way.
In this commit, we fix the architecture doc's false claim that the OutboxPublisher folded path bypasses DurableMailbox.Send (it does not; the exemption is now real and implemented via the outbox-ID context marker), document the control-priority exemption and the boot messages that carry it, describe the hardened probe (single-flight, ambient-tx stripping), and state the known residual honestly: an in-turn Tell into a saturated peer fails the sender's turn and burns the inbound message's delivery attempts, with postpone semantics as the planned structural fix and the dead-letter tooling from #1119 as the interim recovery path.
|
An adversarial audit pass ran against the original 8 commits; the three commits on top address what it found. Dispositions: Fixed:
Acknowledged, deferred with rationale:
|
|
@claude review this |
|
Claude finished @Roasbeef's task in 2m 54s —— View job Review: durable mailbox backpressure watermarks (#1121)I read the watermark core, the depth store/queries, the serverconn deferral path, the metrics collector, and the fleet opt-in sites, and cross-checked the three follow-up commits against the earlier Codex pass.
This is a careful, well-documented change. The exemption model ( Codex comment dispositions (verified against HEAD)1. Outbox folded delivery bypass (P1) — resolved. The fold path stamps the outbox ID into 2. Metrics label cardinality (P2) — accurate as written, mechanism note. The collector emits Open concern3. Hard-shedding the ledger silently loses audit entries (Codex P1 #2 — still unaddressed). This is distinct from the acknowledged residual #4 (in-turn Tells that nack and eventually dead-letter — those are at least visible and requeueable via the #1119 DLQ tooling). A background confirmation-poll emission that log-and-drops leaves no dead-letter row and no trace beyond one WarnS line. For this actor the watermark converts a temporarily deep but eventually-complete durable backlog into permanent silent financial data loss, which is arguably a worse failure mode than the unbounded-growth problem the PR is solving everywhere else. The ledger is also different in kind from the other opt-in sites: serverconn egress and the OOR/round actors have a real producer-side backpressure path (the cursor stalls, the turn nacks), so shedding there genuinely pushes back. The ledger has no such path — its producers can only drop. Worth reconsidering one of:
Given the audit-trail invariant, soft-only is the smaller, safer step for this PR. Fix this → Minor observations (non-blocking)a. Probe reset can undercount concurrent in-flight sends ( b. Overall: solid change. The ledger opt-in is the one thing I'd want resolved before merge — everything else is either already fixed or cosmetic. |
In this commit, we address the two findings from the bot review pass. The ledger actor drops its hard watermark and keeps only the soft one. Its producers are fire-and-forget Tells that log-and-continue on failure with no redrive path, so a hard ErrMailboxSaturated refusal there is not backpressure but a permanently missing accounting leg: unlike the in-turn transport sends (which nack and eventually surface in the dead-letter table), a refused ledger emission leaves no trace beyond one warning line, and the double-entry balance stays silently incomplete after the backlog drains. A deep-but-eventually-complete backlog is the better failure mode for an audit trail; the soft warning and the depth gauge keep it observable. The probe reset also stops discarding racing sends: senders that increment the delta while the single-flighted COUNT runs enqueue their rows after the count, so zeroing sentSinceProbe on completion dropped them from the estimate, an undershoot in the unsafe direction. The reset now subtracts a pre-probe snapshot instead, keeping the racing increments and preserving the local no-undershoot property across the probe window.
|
Both review items addressed in 8c98bce: Ledger hard-shed (open concern): fixed as suggested. The ledger actor now carries the soft watermark only; the hard mark is disabled with a comment (and a Probe reset undercount (minor a): fixed via the snapshot approach. The reset now subtracts the pre-probe delta instead of zeroing, so sends that race the single-flighted COUNT keep their increments and the local no-undershoot property holds across the probe window. Minor (b) needed no change, as noted. CI re-running on the new head. |
📚 Doc advisoryThis PR's code changes leave the per-package docs for Proposed changes (click to expand)diff --git a/credit/AGENTS.md b/credit/AGENTS.md
index 13ce4f8a..3d95fbc2 100644
--- a/credit/AGENTS.md
+++ b/credit/AGENTS.md
@@ -56,6 +56,15 @@ credit redemptions against the swap-server credit ledger, as a crash-safe
boot-time reconcile); `triggerRedeem` fires only after the settled receive's
terminal snapshot commits, so a crash before that leaves no half-applied
redeem.
+- The per-operation durable mailbox opts into the shared backlog watermarks
+ (`actor.DefaultSoftHighWatermark` / `DefaultHardHighWatermark`), so a
+ wedged operation sheds load at the producer with
+ `actor.ErrMailboxSaturated` instead of growing its backlog without bound.
+ `ResumeCreditOpRequest` is exempt: it declares
+ `Priority() == actor.ControlPriority`, because a per-operation backlog past
+ the hard watermark is exactly the condition the boot-time resume exists to
+ work through, and refusing it would silently strand the in-flight
+ operation.
## Deep Docs
diff --git a/credit/CLAUDE.md b/credit/CLAUDE.md
index 13ce4f8a..3d95fbc2 100644
--- a/credit/CLAUDE.md
+++ b/credit/CLAUDE.md
@@ -56,6 +56,15 @@ credit redemptions against the swap-server credit ledger, as a crash-safe
boot-time reconcile); `triggerRedeem` fires only after the settled receive's
terminal snapshot commits, so a crash before that leaves no half-applied
redeem.
+- The per-operation durable mailbox opts into the shared backlog watermarks
+ (`actor.DefaultSoftHighWatermark` / `DefaultHardHighWatermark`), so a
+ wedged operation sheds load at the producer with
+ `actor.ErrMailboxSaturated` instead of growing its backlog without bound.
+ `ResumeCreditOpRequest` is exempt: it declares
+ `Priority() == actor.ControlPriority`, because a per-operation backlog past
+ the hard watermark is exactly the condition the boot-time resume exists to
+ work through, and refusing it would silently strand the in-flight
+ operation.
## Deep Docs
diff --git a/db/actordelivery/sqlc/AGENTS.md b/db/actordelivery/sqlc/AGENTS.md
index 35036803..6776e286 100644
--- a/db/actordelivery/sqlc/AGENTS.md
+++ b/db/actordelivery/sqlc/AGENTS.md
@@ -11,7 +11,15 @@ by `sqlc` from `db/actordelivery/queries/mailbox.sql`; regenerate with
- `Queries` / `Querier` — generated query struct and interface (enqueue,
lease, peek, ack/nack, extend, expire, outbox claim/complete/fail,
- dedup, FSM checkpoints, dead letters).
+ dedup, FSM checkpoints, dead letters, backlog depth counts).
+- `CountMailboxMessages` / `CountMailboxMessagesByMailbox` —
+ `COUNT(*)` of `mailbox_messages` for one mailbox and, grouped, for
+ every mailbox currently holding at least one message. Leased rows
+ count and acked rows are deleted, so the count is exactly the
+ undelivered backlog; these back `actor.MailboxDepthStore`, the durable
+ mailbox's watermark admission check, and the `waved_mailbox_depth`
+ gauges. The grouped form emits no row for a drained mailbox, keeping
+ the result bounded by the number of backed-up actors.
- `MailboxMessage`, `OutboxMessage`, `AskResult`, `FsmCheckpoint`,
`DeadLetter`, `ProcessedMessage` — row models for the actor-delivery
tables.
diff --git a/db/actordelivery/sqlc/CLAUDE.md b/db/actordelivery/sqlc/CLAUDE.md
index 35036803..6776e286 100644
--- a/db/actordelivery/sqlc/CLAUDE.md
+++ b/db/actordelivery/sqlc/CLAUDE.md
@@ -11,7 +11,15 @@ by `sqlc` from `db/actordelivery/queries/mailbox.sql`; regenerate with
- `Queries` / `Querier` — generated query struct and interface (enqueue,
lease, peek, ack/nack, extend, expire, outbox claim/complete/fail,
- dedup, FSM checkpoints, dead letters).
+ dedup, FSM checkpoints, dead letters, backlog depth counts).
+- `CountMailboxMessages` / `CountMailboxMessagesByMailbox` —
+ `COUNT(*)` of `mailbox_messages` for one mailbox and, grouped, for
+ every mailbox currently holding at least one message. Leased rows
+ count and acked rows are deleted, so the count is exactly the
+ undelivered backlog; these back `actor.MailboxDepthStore`, the durable
+ mailbox's watermark admission check, and the `waved_mailbox_depth`
+ gauges. The grouped form emits no row for a drained mailbox, keeping
+ the result bounded by the number of backed-up actors.
- `MailboxMessage`, `OutboxMessage`, `AskResult`, `FsmCheckpoint`,
`DeadLetter`, `ProcessedMessage` — row models for the actor-delivery
tables.
diff --git a/metrics/README.md b/metrics/README.md
index 366f3570..0800e34d 100644
--- a/metrics/README.md
+++ b/metrics/README.md
@@ -183,8 +183,6 @@ page, because at that depth something downstream has stopped, not slowed.
## gRPC Client Metrics
-## gRPC Client Metrics
-
Per-method **client-side** metrics for calls `waved` makes to the ark
operator, via `go-grpc-middleware/providers/prometheus` `ClientMetrics`,
installed as unary + stream interceptors on the operator connection
diff --git a/oor/AGENTS.md b/oor/AGENTS.md
index bb983c14..39618423 100644
--- a/oor/AGENTS.md
+++ b/oor/AGENTS.md
@@ -122,6 +122,18 @@ For field-level detail, use `go doc github.com/lightninglabs/wavelength/oor.<Sym
- Server-side lineage-cap rejection surfaces as a typed `*ErrLineageTooLarge`
via `ClassifySubmitError`, so wallet callers can switch on the cause
without depending on the `oorpb` proto type.
+- The registry and every session actor opt into the shared backlog
+ watermarks (`actor.DefaultSoftHighWatermark` /
+ `DefaultHardHighWatermark`). `serverconn`'s ingress dispatch classifies
+ the resulting `actor.ErrMailboxSaturated` as a deferral, so a backed-up
+ registry stalls the operator cursor instead of deepening its own backlog.
+ `RestoreNonTerminalRequest` is exempt via
+ `Priority() == actor.ControlPriority`: boot treats a failed restore `Ask`
+ as fatal, so refusing it for the very backlog it exists to work through
+ would turn a backed-up registry into a daemon-wide restart crash loop.
+ Control priority also claims the restore ahead of the redelivered
+ backlog; both orders are safe, since routing and restore converge on
+ `ensureChild`'s dedup.
## Deep Docs
diff --git a/oor/CLAUDE.md b/oor/CLAUDE.md
index bb983c14..39618423 100644
--- a/oor/CLAUDE.md
+++ b/oor/CLAUDE.md
@@ -122,6 +122,18 @@ For field-level detail, use `go doc github.com/lightninglabs/wavelength/oor.<Sym
- Server-side lineage-cap rejection surfaces as a typed `*ErrLineageTooLarge`
via `ClassifySubmitError`, so wallet callers can switch on the cause
without depending on the `oorpb` proto type.
+- The registry and every session actor opt into the shared backlog
+ watermarks (`actor.DefaultSoftHighWatermark` /
+ `DefaultHardHighWatermark`). `serverconn`'s ingress dispatch classifies
+ the resulting `actor.ErrMailboxSaturated` as a deferral, so a backed-up
+ registry stalls the operator cursor instead of deepening its own backlog.
+ `RestoreNonTerminalRequest` is exempt via
+ `Priority() == actor.ControlPriority`: boot treats a failed restore `Ask`
+ as fatal, so refusing it for the very backlog it exists to work through
+ would turn a backed-up registry into a daemon-wide restart crash loop.
+ Control priority also claims the restore ahead of the redelivered
+ backlog; both orders are safe, since routing and restore converge on
+ `ensureChild`'s dedup.
## Deep Docs
diff --git a/unroll/AGENTS.md b/unroll/AGENTS.md
index f7ed9727..ed5566ae 100644
--- a/unroll/AGENTS.md
+++ b/unroll/AGENTS.md
@@ -305,6 +305,15 @@ For field-level detail, use `go doc github.com/lightninglabs/wavelength/unroll.<
operator-sourced OOR artifacts flow into proof assembly, so a
zero- or short-output node maps to a retryable error rather than
a goroutine panic.
+- **Backlog watermarks bound the actor, never the resume.** The
+ unroll actor's durable mailbox opts into the shared defaults
+ (`actor.DefaultSoftHighWatermark` /
+ `DefaultHardHighWatermark`), so a wedged consumer sheds load at
+ the producer with `actor.ErrMailboxSaturated`. `ResumeUnrollRequest`
+ declares `Priority() == actor.ControlPriority` and bypasses the
+ admission check: boot-time restore treats a failed resume `Ask` as
+ fatal, and the backlog it would be refused for is the one the
+ resume exists to drain.
## Deep Docs
diff --git a/unroll/CLAUDE.md b/unroll/CLAUDE.md
index f7ed9727..ed5566ae 100644
--- a/unroll/CLAUDE.md
+++ b/unroll/CLAUDE.md
@@ -305,6 +305,15 @@ For field-level detail, use `go doc github.com/lightninglabs/wavelength/unroll.<
operator-sourced OOR artifacts flow into proof assembly, so a
zero- or short-output node maps to a retryable error rather than
a goroutine panic.
+- **Backlog watermarks bound the actor, never the resume.** The
+ unroll actor's durable mailbox opts into the shared defaults
+ (`actor.DefaultSoftHighWatermark` /
+ `DefaultHardHighWatermark`), so a wedged consumer sheds load at
+ the producer with `actor.ErrMailboxSaturated`. `ResumeUnrollRequest`
+ declares `Priority() == actor.ControlPriority` and bypasses the
+ admission check: boot-time restore treats a failed resume `Ask` as
+ fatal, and the backlog it would be refused for is the one the
+ resume exists to drain.
## Deep Docs
diff --git a/waved/AGENTS.md b/waved/AGENTS.md
index b778f21c..c3538f36 100644
--- a/waved/AGENTS.md
+++ b/waved/AGENTS.md
@@ -138,6 +138,11 @@ For field-level detail, use `go doc github.com/lightninglabs/wavelength/waved.<S
so the bound survives restarts.
- `operatorTermsFromResponse` and daemon `GetInfo` must preserve
`FreeRefreshWindowBlocks` end to end.
+- `systemStatsAdapter.GetMailboxDepths` (`waved/metrics.go`) backs the
+ `waved_mailbox_depth` / `waved_mailbox_backlog` scrape gauges by type
+ asserting the delivery store to `actor.MailboxDepthStore`. That surface is
+ optional, so a store without it yields an error and the collector skips
+ those gauges for the scrape rather than failing it.
- The VTXO manager reads `FreeRefreshWindowBlocks` from the latest cached
operator terms on each expiry check. It delays automatic refresh to the
window boundary only when the local dynamic critical threshold plus retry
diff --git a/waved/CLAUDE.md b/waved/CLAUDE.md
index b778f21c..c3538f36 100644
--- a/waved/CLAUDE.md
+++ b/waved/CLAUDE.md
@@ -138,6 +138,11 @@ For field-level detail, use `go doc github.com/lightninglabs/wavelength/waved.<S
so the bound survives restarts.
- `operatorTermsFromResponse` and daemon `GetInfo` must preserve
`FreeRefreshWindowBlocks` end to end.
+- `systemStatsAdapter.GetMailboxDepths` (`waved/metrics.go`) backs the
+ `waved_mailbox_depth` / `waved_mailbox_backlog` scrape gauges by type
+ asserting the delivery store to `actor.MailboxDepthStore`. That surface is
+ optional, so a store without it yields an error and the collector skips
+ those gauges for the scrape rather than failing it.
- The VTXO manager reads `FreeRefreshWindowBlocks` from the latest cached
operator terms on each expiry check. It delays automatic refresh to the
window boundary only when the local dynamic critical threshold plus retryHow to apply: save the diff above to a file and
Advisory only — this check never fails CI. Posted by the per-PR doc-gardening advisor. Run link omitted: this session's sandbox blocked environment-variable expansion, so |
2 participants
In this PR, we give the durable mailbox the backpressure signal it never had. A channel mailbox bounds its queue with
ErrMailboxFull, but a durable mailbox lands everyTellas a database row: a consumer that stops draining grows its backlog for as long as the outage lasts, with nothing pushing back on producers and nothing on a dashboard showing the hole getting deeper. This is the second PR in the durable-actor hardening series (after #1119), covering the backpressure gap called out in the DLQ/backpressure review.Watermarks
Two thresholds on
DurableMailboxConfig(flowing throughDurableActorConfig) bound the persistent backlog. Past the soft watermark the mailbox logs one warning per breach episode. At or past the hard watermark,Sendrefuses the message with the newErrMailboxSaturatedsentinel, the durable analogue ofErrMailboxFull: the message was not enqueued, and the caller sheds, stashes, or retries after the consumer drains. The check runs before encoding or promise registration, so a refusal needs no cleanup andTrySend/TryTellinherit it for free. Messages with priority >=RestartPriorityare always exempt, since theRestartMessagethat would un-wedge a stuck actor must not be refused by the very backlog it exists to drain.Depth is read through a new narrow
MailboxDepthStoresurface (discovered by type assertion on theDeliveryStore, same pattern as keeping test doubles small elsewhere), backed by aCOUNT(*)that the prefix ofidx_mailbox_messages_availablecovers. The read is TTL-cached at one second with a local sent-since-probe delta on top, so the common send path pays no extra query. The estimate is deliberately one-sided (local sends push it up immediately, acks only surface at the next probe): overshooting is the safe direction for an admission check. A failed probe fails open, because a broken monitoring read must not become message loss.Both thresholds default to zero (disabled). The
multi:commit is the complete inventory of which mailboxes opt in: the serverconn egress sender, the OOR registry and per-session actors, and the ledger, credit, and unroll actors, all at the shared defaults (soft 1000, hard 10000).serverconn classification
deliverToActornow classifiesErrMailboxSaturatedexactly likeErrMailboxFull: the envelope defers, the cursor stalls, and the ingress loop re-pulls after a backoff. A backed-up durable actor therefore exerts backpressure on the operator stream instead of deepening its own backlog, riding the deferral machinery #1093 put in place (episode logging, deferred counter, redrive) with no changes downstream of the sentinel check.The OutboxPublisher's folded delivery path is deliberately unthrottled: it enqueues into the target mailbox inside the publisher's own write transaction, bypassing
DurableMailbox.Send, and throttling it would only move the backlog into the outbox table while breaking the claim-expiry retry contract. This is documented in the new architecture section.Observability
Two scrape-time gauges read the same store surface:
waved_mailbox_backlog(unlabelled total, explicit zero when drained so "all clear" and "scrape broke" stay distinguishable) andwaved_mailbox_depth{mailbox_id}(one series per mailbox currently holding messages, so per-session actor IDs never accumulate as permanent series). The metrics README gains the reference rows and an alerting recipe keyed to the watermark defaults.See each commit message for a detailed description w.r.t the incremental changes.