Extended progression batches amortize the connection, not the transaction (2.39.2)

[jeremydmiller]

Follow-up to #622, driven by the lock analysis behind marten#5167.

The finding

The report that the extended-progression telemetry UPDATE increases database lock waits is correct, and the mechanism is not write volume. Marten implemented the batched overload as one UPDATE … FROM unnest(…) covering the whole flush. A multi-row statement takes a row lock on every shard in the batch and holds all of them until it commits — so one slow projection batch sitting on one progression row stalls the telemetry write of every other shard on that database, and transitively whatever those shards were waiting to do.

Reproduced against PostgreSQL with three sessions (P = a projection batch holding one row, H = the batched telemetry UPDATE over six, Q = an unrelated projection committing progress for a row P never touched):

  pid  | state  | wait_event_type |  wait_event   | blocked_by
 11762 | active | Lock            | transactionid | {11763}     Q: UPDATE ... WHERE name = 'shard_a'
 11763 | active | Lock            | transactionid | {11765}     H: the batched telemetry UPDATE
 11765 | active | Timeout         | PgSleep       | {}          P: the projection batch on shard_d

Q timed out after 4s. It was never contending with P — it was queued behind H, which had already locked shard_a on its way to shard_d and then stalled there. Replacing H with six single-row autocommit UPDATEs, Q clears in ~1ms and only shard_d — the genuinely contended row — waits.

This also explains the part of #5167 the reporter could not account for: that the blocker and the blocked statement were both the heartbeat UPDATE. pg_blocking_pids() reports direct blockers only; the projection transaction that is the real root is one hop further down the chain.

What changed

The contract is now explicit (IEventDatabase.WriteExtendedProgressionAsync(IReadOnlyList<ShardState>, …)). It previously asked for "as few round-trips as the store can manage — ideally one" and said nothing about transaction scope, so folding the batch into one statement was a reasonable reading. It now states that one row per transaction is required, that what a batch amortizes is the connection — which is all #553 ever needed, since N single-row statements on one rented connection cost one rent — and that implementations should skip rows whose telemetry is unchanged rather than hand every matched row a new tuple version per flush.

Batches flush ordered by shard name. With one row per transaction a single writer never holds more than one row lock, but Tracker is per-database and shared and building a daemon does not go through a cache, so two writers can race over the same rows. The pending dictionary's enumeration order is an implementation detail, not an agreement between them; sorting removes that deadlock hazard for every store at zero cost. Pinned by a_flushed_batch_is_ordered_by_shard_name.

No behavior change for stores that never implemented the batched overload — the default interface implementation loops the single-state write, which already satisfies the rule. The Marten-side rewrite is marten#5186.

Verification

Full EventTests suite green (725 tests) on net9.0.

🤖 Generated with Claude Code

https://claude.ai/code/session_017CTtw2kVRSZKp1p5RTxgAy