actor: Add a supervision kernel for the durable actor runtime

baselib/actor/AGENTS.md

@@ -30,6 +30,11 @@ crash-safe at-least-once delivery with exactly-once deduplication.
 - `DurableActor` — Actor variant with crash-safe mailbox backed by SQL persistence. Provides `Wait(ctx)` to block until the actor stops and `StopAndWait(ctx)` to request a graceful shutdown and then wait.
 - `DurableActorConfig[M, R]` — Configuration struct for `DurableActor`: behavior, store, codec, clock, DLO, WaitGroup, `TellRetryPolicy`, lease/heartbeat/poll durations, max attempts, cleanup timeout, deduplication TTL, and `NumWorkers`.
 - `DurableActorConfig.NumWorkers` — How many concurrent worker loops drain the actor's single mailbox. Default and any value `<= 1` is one worker (strictly-sequential processing). A value `> 1` turns the actor into a competing-consumer pool: that many goroutines each lease distinct messages via `LeaseNextMailboxMessage`, so independent messages run in parallel while per-correlation-key FIFO still keeps same-key messages ordered. Only for behaviors whose handlers are concurrency-safe and hold no writer across their side effects (e.g. the serverconn egress sender on the Read/Commit path). `NewDurableActor` **fails closed** with `ErrConcurrentClassicBehavior` when `NumWorkers > 1` is paired with a classic (`Left`) `ActorBehavior`, since the classic path wraps the whole `Receive` in one write transaction and assumes sequential delivery; pools are only valid on the Read/Commit (`TxBehavior`) path. The test-only `DurableActorConfig.AllowConcurrentClassicBehavior()` escape hatch bypasses the guard for the egress benchmark that measures the forbidden config; production code must never call it.
+- `DurableActorConfig.MaxRestarts` / `RestartWindow` — BEAM-style restart intensity budget for the supervision kernel. It defaults OFF: `DefaultMaxRestarts` is `UnlimitedRestarts` (-1) and a zero `MaxRestarts` normalizes to it, so a panicking actor restarts for as long as it keeps panicking. That is deliberate. Restarting forever is no worse than the nack-and-continue loop supervision replaces (both are rate-limited by the nack backoff), whereas a finite budget adds a failure mode the runtime did not have: the actor dies permanently and keeps looking alive to anyone who is not watching. Set a finite budget ONLY where the owner wires `Watch` and reacts to `TerminationRestartIntensityExceeded`; `RecommendedMaxRestarts` (5) over `DefaultRestartWindow` (60s) is the value to reach for when you do. Restart timestamps are tracked in a sliding window off the config's injected clock.
+- `(*DurableActor).Watch(ctx) <-chan TerminationInfo` — Registers a terminal lifecycle watcher. The returned channel receives exactly one `TerminationInfo` and is then closed. Delivery is non-blocking by construction (a single-use buffer-of-one channel written once), so a slow or absent watcher can never park the actor's shutdown path (#1093 invariant). Registering after termination returns a channel already loaded with the notification, so there is no race with a stopping actor; cancelling `ctx` deregisters the watcher and closes its channel with no notification. The notification is published when the supervision loop exits, or by `Stop` for an actor that was never started; an actor that is neither started nor stopped never publishes one.
+- `TerminationInfo` / `TerminationReason` — What a watcher observes: `TerminationStopped` (Stop/StopAndWait), `TerminationContextCancelled` (lifetime context died without a Stop; reserved for a future externally-owned-context constructor), `TerminationRestartIntensityExceeded` (restart budget spent, `Err` carries the panic, `RestartsExhausted` is true), `TerminationRestartFailed` (checkpoint reload or RestartMessage enqueue failed). `Restarts` counts restarts over the actor's whole lifetime.
+- `MessageCodec.Supports(typeID)` — Reports whether the codec has a constructor registered for a TLV type. The supervision path uses it to decide, BEFORE tearing a generation down, whether the checkpoint hand-off is even possible; an actor whose codec never registered a `RestartMessage` degrades to cycling its worker generation with no `OnStop` and no restore, since a teardown it cannot be rebuilt from is strictly worse than leaving the behavior running.
+- `PrependRestartMessageWithID` — `PrependRestartMessage` with the enqueued row ID returned. Supervision deletes the row it enqueued last time before writing the next, so a run of restarts leaves at most one restart row in the mailbox. A restart row carries `MaxAttempts` 1 and the runtime never nacks one (a nacked row at `attempts == max_attempts` is neither leasable nor reapable, so it would strand): a failed restart turn dead-letters instead, which makes restore handlers responsible for their own idempotency.
 - `DefaultDurableActorConfig[M, R]()` — Constructor returning a `DurableActorConfig` with safe defaults (30s lease, 10 max attempts, 1s poll floor / 30s poll ceiling, DefaultTellRetryPolicy).
 - `DurableActorConfig.PollInterval` / `MaxPollInterval` — Floor and ceiling of the idle mailbox poll backoff (defaults 1s / 30s). The fallback poll is NOT the delivery path: a same-process enqueue signals the mailbox's wake channel, and the store's post-commit `RegisterMailboxWake` callback rouses the exact mailbox a committed transaction enqueued into, so delivery latency is unaffected by how far the backoff has decayed. Each consecutive empty poll roughly doubles the wait from `PollInterval` up to `MaxPollInterval`; any wake or successfully claimed message snaps it back to the floor. This matters at scale because on a Postgres-backed store every empty poll is a full SERIALIZABLE write transaction that updates no rows, so thousands of resident-but-idle actors polling at a fixed 1Hz become a pure transaction tax. The timer is never stopped: since `RegisterMailboxWake` is same-process only, the poll remains the sole discovery mechanism for a row enqueued by another process or replica, which makes `MaxPollInterval` the worst-case cross-process/cross-replica delivery latency. A zero value normalizes to the default and a ceiling below the floor is raised to the floor (constant cadence, never a shrinking wait).
 - `TellRetryPolicy` — Function type `func(attempts int, lastErr error) (bool, time.Duration)` determining retry behavior for failed Tell messages. Return `(false, _)` to dead-letter immediately.
@@ -93,6 +98,14 @@ crash-safe at-least-once delivery with exactly-once deduplication.
   decisions must use `Delivery.EffectiveAttempts()` so the in-flight peeked
   attempt is counted before a nack can raise the row to `max_attempts`.
 - `Tell` with a `DurableActor` persists the message before returning (crash-safe enqueue).
+- **Panic means restart, not redeliver.** A behavior that *returns* an error is an ordinary message failure: it nacks and retries per `TellRetryPolicy`. A behavior that *panics* is treated as corrupted in-memory state: the recovered value becomes a `behaviorPanic`, the delivery's normal ack/nack/dead-letter bookkeeping runs first (so the poison message burns its attempt), and only then does the worker hand the panic to supervision. Supervision cancels the current worker generation (draining ALL workers, not just the panicking one), runs the behavior's `OnStop` bounded by `CleanupTimeout`, reloads the persisted FSM checkpoint, prepends a `RestartMessage` at `RestartPriority`, and starts a fresh generation. The nack-before-restart ordering is load-bearing: it is what makes a deterministic poison message climb to `max_attempts` and dead-letter instead of crash-looping the actor forever.
+- **A restart reuses the behavior INSTANCE; the clean slate is the handler's job.** The framework does not rebuild the behavior. It stops the workers, optionally calls `OnStop`, and redelivers a `RestartMessage`; the same Go value keeps serving afterwards with whatever fields the panic left behind. The actor is therefore clean exactly when its `RestartMessage` handler rebuilds every piece of in-memory state from the durable row, and stale otherwise. This is not hypothetical: the behaviors that adopt supervision (`credit.opBehavior`, `oor.sessionBehavior`, `oor.oorRegistryBehavior`, `unroll.behavior`) all carry a reload seam, and a handler that returns Ok without using it leaves the actor exactly as far ahead of durable truth as the panic left it. A behavior with no in-memory turn state (the `serverconn` egress sender) may consume the message as a no-op, but it should say so and say why.
+- **A restart message is not retried.** It is enqueued with `MaxAttempts` 1, and the runtime dead-letters (rather than nacks) a restart turn that fails, because a nacked row at `attempts == max_attempts` strands forever. Restore handlers get exactly one shot per restart and must be idempotent.
+- **`OnStop` may run mid-life and more than once.** A supervised restart calls it before the rebuild, so implementations must be idempotent and must leave the behavior able to serve a new generation rather than assuming it is being discarded. A panic escaping `OnStop` is recovered (it is invoked precisely when the behavior's invariants are broken) and terminates the actor with `TerminationRestartFailed` rather than taking the process down.
+- **A panicking turn's own writes are rolled back.** On the classic path the whole `Receive` runs inside one framework transaction, so supervision returns the panic from that transaction to force a rollback and redoes the message's ack/nack bookkeeping outside it. Committing the partial writes alongside the nack would persist exactly the torn state the restart exists to escape, and the checkpoint reload would then hand it straight back.
+- **Restart preserves public identity.** The restart runs on an internal generation context derived from the actor's lifetime context, deliberately bypassing the `Once`-guarded `Start`/`Stop`. The actor keeps its ID, its `DurableMailbox` (so senders keep enqueueing across the restart gap, and the mailbox's promise registry survives), and its cached `Ref`. Callers holding an `ActorRef` observe nothing beyond a pause in processing.
+- **In-flight Ask promises across a restart.** The panicking turn's promise is completed with the panic error by the normal result handling. A sibling worker's turn sees its generation context cancelled, returns a context error, and has its promise completed with that error; its durable bookkeeping still runs on a detached context. A message not yet handed to the behavior is simply redelivered afterwards and its caller still gets the eventual result. `DurableAsk` responses travel through the outbox, so a restart only delays them.
+- **Exceeding the restart budget is terminal, which is why it is off by default.** Once a finite `MaxRestarts` is exhausted inside `RestartWindow`, supervision logs at error level (an internal-bug class, so the level rule allows it), cancels the actor's lifetime context so further sends fail fast rather than piling into a mailbox nothing will drain, tears the actor down, and publishes `TerminationRestartIntensityExceeded` to watchers. Nothing else notices: an actor that dies this way still holds its ID and its mailbox rows, so a finite budget without a `Watch` observer converts a visible crash loop into invisible permanent death.
 - Outbox messages are dispatched only after state is persisted (outbox pattern).
 - **Outbox fold p-model.** For tx-aware stores, outbox delivery is
   `claim -> (target mailbox enqueue + CompleteOutbox) in one write tx`. If the
@@ -101,7 +114,7 @@ crash-safe at-least-once delivery with exactly-once deduplication.
   transaction failure even when the inner Tell/Complete operations returned nil,
   because begin/commit failures happen outside those operation-level logs.
 - `ServiceKey` lookup via `Receptionist` is type-safe: mismatched types return `ErrServiceKeyTypeMismatch`.
-- `RestartMessage` has `RestartPriority` (MaxInt32) ensuring it is processed before all other messages on recovery.
+- **`RestartMessage` ordering, and how it holds under a pool.** `RestartMessage` carries `RestartPriority` (MaxInt32), which makes the claim query hand it out before every other row. Under `NumWorkers > 1` that orders the CLAIMS but not the TURNS: launching the whole pool at once lets one worker take the restart while a sibling takes the row behind it, so a normal turn runs against a behavior instance that is still rebuilding itself from the checkpoint. The guarantee is therefore enforced by a **single-worker warm-up barrier**: a generation launches one worker first, and the rest of the pool waits until that worker has resolved the restart hand-off. The barrier holds unconditionally for a row supervision enqueued itself (a supervised restart), and for the boot hand-off, which an owner prepends before `Start` and which the actor cannot see, it orders the first claim and releases on an idle tick. It cannot wedge a pool: a first claim of anything other than a restart releases it before that message is processed, a restore that fails or panics releases it, whatever ends the warm-up worker releases it, and a `Stop` mid-barrier releases it through the generation context. A single-worker actor is already strictly sequential and gets no barrier at all.
 - Transaction context (`WithTx`/`RequireTx`) enables same-DB-transaction joining between actors and their callers.
 - `Mailbox.Send` returns the exact failure error (`ErrMailboxClosed`, `ErrActorTerminated`, `context.Canceled`, `context.DeadlineExceeded`) rather than a boolean; `Tell` and `Ask` propagate this directly to callers.
 - **Never `Tell` from inside a receive goroutine without a bound.** A blocking