Summary
We have a full dead-letter store but no dead-letter handling. When a
durable actor exhausts its TellRetryPolicy, the framework writes the message
to the dead_letters table and deletes it from the mailbox — and then nothing
ever looks at it again. No actor consumes it, no monitor surfaces it, no metric
or log-at-the-right-level alerts on it, no RPC exposes it, and no path retries
or recovers it. A dead-lettered message is silently and permanently stuck.
This was found while reviewing the OOR per-session refactor, but it is not
OOR-specific — it affects every durable actor in the daemon.
What exists vs. what's missing
Exists (db/actordelivery/store_impl.go):
MoveToDeadLetter, GetDeadLetter, ListDeadLettersByActor,
DeleteDeadLetter over a dead_letters table.
- The durable-actor framework calls
MoveToDeadLetter + DeleteMessage when a
Tell's retry policy gives up (baselib/actor/durable_actor.go handleResult
/ handleResultInTx).
Missing:
- No production caller of
ListDeadLetters / GetDeadLetter (only the store
impl and tests reference them).
- No dead-letter office (DLO) actor wired at the actor-system level; OOR's
registry and per-session actors set no DLO either. (Note: the cfg.DLO
ActorRef is a separate path — only used by trySendToDLO on
ErrActorTerminated during Send, not by the retry-exhaustion path, which
always lands in the store table.)
- No monitoring, alerting, RPC surface, or operator-facing visibility for
dead-lettered messages.
Why it matters (concrete OOR cases, generalizes to all actors)
- Over-cap incoming hint (
oor/registry.go errIncomingAdmissionCapped):
a ResolveIncomingTransferRequest rejected at the concurrency cap is
Nack-retried by the registry's durable mailbox ~5 times (~31s,
DefaultTellRetryPolicy) and then dead-lettered. The serverconn ingress
already acked the operator envelope when the Tell durably enqueued, so there
is no transport re-send. Result: under sustained over-cap a legitimate
incoming transfer hint is dropped, the VTXO is never materialized/monitored
locally, and there is no dead-letter handling to recover it.
- More generally, any durable-actor Tell that exhausts its retry policy
(transient dependency failures, a poison message, a bug) vanishes into
dead_letters with no operator signal.
For value-bearing subsystems (OOR, rounds, unroll, ledger) a silently
dead-lettered message can mean stuck or invisible funds, hence the safety
label.
Proposed work
- Decide the policy per actor class. Some messages should hard-fail loudly
(poison), some should be parked for manual/automated replay, some are safe to
drop. Make this an explicit choice, not an accident of defaults.
- A dead-letter consumer. Either a DLO actor or a daemon-owned monitor that
periodically ListDeadLettersByActors, surfaces them (structured log at an
appropriate level + metric/health signal), and supports requeue where the
message is idempotent.
- Operator visibility. An RPC / CLI (or at minimum a health/metrics field)
to see dead-letter counts and inspect entries, instead of needing to query
the DB by hand.
- Wire it for all durable actors, with a sane default, and document the
contract in baselib/actor docs (when a message dead-letters, who owns it,
how it is recovered).
- Non-durable actors: confirm/define the analogous behavior (currently the
DLO ref + trySendToDLO on ErrActorTerminated), and make it consistent
with the durable policy.
Acceptance criteria
References
db/actordelivery/store_impl.go — dead-letter store CRUD.
baselib/actor/durable_actor.go — handleResult / handleResultInTx
retry-exhaustion -> MoveToDeadLetter.
oor/registry.go — errIncomingAdmissionCapped over-cap path that
dead-letters with no recovery (motivating case).
Summary
We have a full dead-letter store but no dead-letter handling. When a
durable actor exhausts its
TellRetryPolicy, the framework writes the messageto the
dead_letterstable and deletes it from the mailbox — and then nothingever looks at it again. No actor consumes it, no monitor surfaces it, no metric
or log-at-the-right-level alerts on it, no RPC exposes it, and no path retries
or recovers it. A dead-lettered message is silently and permanently stuck.
This was found while reviewing the OOR per-session refactor, but it is not
OOR-specific — it affects every durable actor in the daemon.
What exists vs. what's missing
Exists (
db/actordelivery/store_impl.go):MoveToDeadLetter,GetDeadLetter,ListDeadLettersByActor,DeleteDeadLetterover adead_letterstable.MoveToDeadLetter+DeleteMessagewhen aTell's retry policy gives up (
baselib/actor/durable_actor.gohandleResult/
handleResultInTx).Missing:
ListDeadLetters/GetDeadLetter(only the storeimpl and tests reference them).
registry and per-session actors set no
DLOeither. (Note: thecfg.DLOActorRefis a separate path — only used bytrySendToDLOonErrActorTerminatedduringSend, not by the retry-exhaustion path, whichalways lands in the store table.)
dead-lettered messages.
Why it matters (concrete OOR cases, generalizes to all actors)
oor/registry.goerrIncomingAdmissionCapped):a
ResolveIncomingTransferRequestrejected at the concurrency cap isNack-retried by the registry's durable mailbox ~5 times (~31s,
DefaultTellRetryPolicy) and then dead-lettered. The serverconn ingressalready acked the operator envelope when the Tell durably enqueued, so there
is no transport re-send. Result: under sustained over-cap a legitimate
incoming transfer hint is dropped, the VTXO is never materialized/monitored
locally, and there is no dead-letter handling to recover it.
(transient dependency failures, a poison message, a bug) vanishes into
dead_letterswith no operator signal.For value-bearing subsystems (OOR, rounds, unroll, ledger) a silently
dead-lettered message can mean stuck or invisible funds, hence the
safetylabel.
Proposed work
(poison), some should be parked for manual/automated replay, some are safe to
drop. Make this an explicit choice, not an accident of defaults.
periodically
ListDeadLettersByActors, surfaces them (structured log at anappropriate level + metric/health signal), and supports requeue where the
message is idempotent.
to see dead-letter counts and inspect entries, instead of needing to query
the DB by hand.
contract in
baselib/actordocs (when a message dead-letters, who owns it,how it is recovered).
DLOref +trySendToDLOonErrActorTerminated), and make it consistentwith the durable policy.
Acceptance criteria
/ health surface), not silent.
recovery happens.
baselib/actordocs describe the dead-letter contract.References
db/actordelivery/store_impl.go— dead-letter store CRUD.baselib/actor/durable_actor.go—handleResult/handleResultInTxretry-exhaustion ->
MoveToDeadLetter.oor/registry.go—errIncomingAdmissionCappedover-cap path thatdead-letters with no recovery (motivating case).