feat(gateway): PendingCompletionRegistry — explicit state machine for completion delivery

[handnewb]

Two approaches are offered for #70300 — maintainers pick one.

• fix(gateway): coalesce concurrent process completions and watch events #72675 — pointwise patches. Smaller diff, keeps the existing delivery shape, carries
  one documented residual (see Declared cost in that PR).
• feat(gateway): PendingCompletionRegistry — explicit state machine for completion delivery #73427 — root-cause fix. Replaces the split pending-state with an explicit registry
  and state machine. Larger diff, removes the old mechanism, no residual.

Both close the reported flood. They are independent branches, not a stack — reviewing either
one alone is enough. I am not advocating for one over the other; #73427 is the better code
and #72675 is the smaller risk, and that trade is yours to make.
Root-cause writeup: #70300.

---

Summary

Multiple background process completions arriving in the same tick flooded the session with
one synthetic turn each (#70300). This PR makes them arrive as a single coalesced turn.

The delivery path is reworked around an explicit pending-completion registry rather than
patching the individual failure modes, because two review rounds on #72675 established that
they share one cause. Root-cause writeup and maintainer discussion: #70300.

Root cause

Pending-completion state was split across three structures that could not observe each other:

• _completion_notification_batches — a dict of lists with no per-entry state
• one Future per waiter, carrying Optional[bool]
• one asyncio.Task per route, whose death was indistinguishable from success

Every finding from both review rounds follows from that split: cancelling destroyed the only
record that anything was pending; claiming and delivering were the same call, so siblings
could not be claimed ahead of the adapter await; failed and unknown shared one value; and
there was nowhere to hold an attempt counter.

Design

PendingCompletionRegistry, owned by the gateway lifecycle, one entry per completion
identity.

PENDING --claim(batch_id)--> CLAIMED --delivered--> DELIVERED   (terminal)
   ^                            |
   |                            +-- transient failure --> PENDING     (attempts+1, backoff)
   |                            +-- attempts exhausted --> FAILED      (terminal, logged)
   |                            +-- already delivered --> SUPERSEDED  (terminal)
   |                            +-- no route -----------> UNROUTABLE  (terminal)
   +---- cancel -----------------+-- gateway stopping --> STOPPING    (terminal, counted)

Entry fields: identity, route_key, state, attempts, next_attempt_at, payload,
batch_id.

Invariants, each covered by a test:

• No entry leaves a non-terminal state except through a declared transition. There is no
  path on which a pending completion disappears.
• claim(batch_id) transitions every fresh sibling of the batch atomically, before the
  adapter await. Failure releases all of them, never a subset.
• Dedupe reads the registry, so "claimed by this flush" and "claimed by another path" are
  distinct states rather than the same boolean.
• Cancellation is never terminal by omission: in-window cancellation returns the entry to
  PENDING; cancellation at stop moves it to STOPPING.

Shutdown contract

On stop, pending completions transition to STOPPING. They are not delivered, not
retried, and the count is logged. Shutdown stays bounded.

Delivering a synthetic turn into a session that is being torn down surfaces it to nobody, so
there is no "last delivery chance" worth buying with an unbounded shutdown. STOPPING is
terminal by declared contract, counted, and tested — not a silent drop.

An in-memory registry does not survive SIGKILL. That is a pre-existing property of the
delivery path rather than a behaviour change here, and out of scope for #70300; see the
discussion in that issue.

Status — work in progress

The registry and state machine exist (34f19f7), but migration of the delivery path onto them is
not finished. _flush_process_completion_batch and
_enqueue_process_completion_notification still run on the previous batch-dict mechanism, so
both mechanisms currently coexist. That is worse than either alone, and it is what I am fixing
next.

Still open:

• _completion_notification_batches and _completion_notification_batch_tasks are still live
  alongside the registry
• the CompletionDisposition enum from 9a6f2ef sits on the old path rather than on registry
  state, and will be replaced by the state field
• commits de9acba through 9a6f2ef are the pointwise series carried over from fix(gateway): coalesce concurrent process completions and watch events #72675; they
  belong to that PR and are being removed from this branch, not kept
• no property tests over the state machine, and no deterministic concurrency harness, so the
  cancellation and shutdown paths are currently argued rather than proven

Please do not review the delivery path yet. I will convert this out of draft when the old
structures are gone and the invariants in Design are covered by tests.

Verification still owed

The issue body diagnosed the flood as the post-turn drain loop injecting one notification per
event. A review comment on #72675 observed that _drain_gateway_watch_events() forwards only
watch_match/watch_disabled, which would mean standard completions never reach that loop.
Both cannot be true.

Both PRs coalesce at both seams, so the reported behaviour is fixed either way — but I have not
proven which seam produced the six injections in the log. I am instrumenting that against
main before finishing the migration, and I will correct the issue body with the result.

Separately, the log in #70300 shows a second defect that coalescing masks rather than fixes:
Persisted transcript lagged live cached history (disk=154, memory=157). Reducing six
injections to one removes the trigger, not the cause. Filing that separately.

Relationship to #72675

#72675 contains pointwise patches for the same five findings — useful if maintainers prefer
a minimal change. This PR replaces the patches with a root-cause fix. Both
are offered; maintainers choose one.

Relationship to #71898 and #70319

The original batching core and its delivery tests carry forward from #71898. That groundwork
is preserved in git history via co-author trailers on the relevant commits, and I am happy to
land this in whatever shape the maintainers prefer — including as a PR stacked on #71898's
branch so those commits stay first. See #70300 for that discussion.

Supersedes #71898 and #72675

[handnewb]

@yuzilongleif-collab — this is the PendingCompletionRegistry approach discussed in #70300. It replaces the pointwise patches in #72675 with an explicit state machine that eliminates the shared root cause of all five review findings. Both PRs are offered; maintainers choose one. HEAD: 34f19f7ac10ed050cfab6915bb22add40890b112

[handnewb]

Closed in favour of #73469.

This branch carried the four pointwise commits from #72675 on top of the registry work, which
meant the two approaches were not independent — reviewing this one implied reviewing that one
plus more. #73469 is rebuilt from main and contains only the registry replacement, so the two
offers are now genuinely disjoint: #72675 is the pointwise patch set, #73469 is the root-cause
fix. GitHub does not allow repointing a PR's head branch, hence the new number rather than an
update here.