Skip to content

actor: wire up dead-letter handling for all actors (durable and otherwise) #705

Description

@Roasbeef

Summary

We have a full dead-letter store but no dead-letter handling. When a
durable actor exhausts its TellRetryPolicy, the framework writes the message
to the dead_letters table and deletes it from the mailbox — and then nothing
ever looks at it again. No actor consumes it, no monitor surfaces it, no metric
or log-at-the-right-level alerts on it, no RPC exposes it, and no path retries
or recovers it. A dead-lettered message is silently and permanently stuck.

This was found while reviewing the OOR per-session refactor, but it is not
OOR-specific — it affects every durable actor in the daemon.

What exists vs. what's missing

Exists (db/actordelivery/store_impl.go):

  • MoveToDeadLetter, GetDeadLetter, ListDeadLettersByActor,
    DeleteDeadLetter over a dead_letters table.
  • The durable-actor framework calls MoveToDeadLetter + DeleteMessage when a
    Tell's retry policy gives up (baselib/actor/durable_actor.go handleResult
    / handleResultInTx).

Missing:

  • No production caller of ListDeadLetters / GetDeadLetter (only the store
    impl and tests reference them).
  • No dead-letter office (DLO) actor wired at the actor-system level; OOR's
    registry and per-session actors set no DLO either. (Note: the cfg.DLO
    ActorRef is a separate path — only used by trySendToDLO on
    ErrActorTerminated during Send, not by the retry-exhaustion path, which
    always lands in the store table.)
  • No monitoring, alerting, RPC surface, or operator-facing visibility for
    dead-lettered messages.

Why it matters (concrete OOR cases, generalizes to all actors)

  • Over-cap incoming hint (oor/registry.go errIncomingAdmissionCapped):
    a ResolveIncomingTransferRequest rejected at the concurrency cap is
    Nack-retried by the registry's durable mailbox ~5 times (~31s,
    DefaultTellRetryPolicy) and then dead-lettered. The serverconn ingress
    already acked the operator envelope when the Tell durably enqueued, so there
    is no transport re-send. Result: under sustained over-cap a legitimate
    incoming transfer hint is dropped, the VTXO is never materialized/monitored
    locally, and there is no dead-letter handling to recover it.
  • More generally, any durable-actor Tell that exhausts its retry policy
    (transient dependency failures, a poison message, a bug) vanishes into
    dead_letters with no operator signal.

For value-bearing subsystems (OOR, rounds, unroll, ledger) a silently
dead-lettered message can mean stuck or invisible funds, hence the safety
label.

Proposed work

  1. Decide the policy per actor class. Some messages should hard-fail loudly
    (poison), some should be parked for manual/automated replay, some are safe to
    drop. Make this an explicit choice, not an accident of defaults.
  2. A dead-letter consumer. Either a DLO actor or a daemon-owned monitor that
    periodically ListDeadLettersByActors, surfaces them (structured log at an
    appropriate level + metric/health signal), and supports requeue where the
    message is idempotent.
  3. Operator visibility. An RPC / CLI (or at minimum a health/metrics field)
    to see dead-letter counts and inspect entries, instead of needing to query
    the DB by hand.
  4. Wire it for all durable actors, with a sane default, and document the
    contract in baselib/actor docs (when a message dead-letters, who owns it,
    how it is recovered).
  5. Non-durable actors: confirm/define the analogous behavior (currently the
    DLO ref + trySendToDLO on ErrActorTerminated), and make it consistent
    with the durable policy.

Acceptance criteria

  • A dead-lettered message for any durable actor is observable (log + metric
    / health surface), not silent.
  • A documented, per-actor-class policy for what dead-lettering means and how
    recovery happens.
  • At minimum, an operator-facing way to list/inspect dead letters.
  • Idempotent messages have a defined requeue path.
  • baselib/actor docs describe the dead-letter contract.

References

  • db/actordelivery/store_impl.go — dead-letter store CRUD.
  • baselib/actor/durable_actor.gohandleResult / handleResultInTx
    retry-exhaustion -> MoveToDeadLetter.
  • oor/registry.goerrIncomingAdmissionCapped over-cap path that
    dead-letters with no recovery (motivating case).

Metadata

Metadata

Assignees

Labels

P1Priority 1 — highactorenhancementNew feature or requestgapmailboxDurable mailbox transportsafetyFund-safety: stuck, lost, or mis-counted funds

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions