The extended-progression batch write no longer builds a lock convoy (#5167)

[jeremydmiller]

Closes the lock-wait half of #5167. #622 (adopted in #5176, documented in #5183) removed the steady-state cost by turning the periodic beat off; this removes the blast radius of the writes that remain.

The finding

The production report that this statement increases database lock waits is correct, and the mechanism is not write volume — it is that a single statement takes row locks on every shard in the batch at once and holds them until it completes.

Three sessions against a stand-in for mt_event_progression — P a projection batch transaction holding one row (shard_d), H the batched telemetry UPDATE covering all six, Q an unrelated projection committing progress for shard_a:

  pid  | state  | wait_event_type |  wait_event   | blocked_by |  query
 11762 | active | Lock            | transactionid | {11763}    | UPDATE ... WHERE name = 'shard_a'      <- Q
 11763 | active | Lock            | transactionid | {11765}    | UPDATE ... SET heartbeat = t.heartbeat <- H
 11765 | active | Timeout         | PgSleep       | {}         | (the projection batch holding shard_d) <- P

Q — which touches a shard the telemetry statement has no business serializing — blocked and timed out after 4s. It was never contending with P; it was queued behind H, which had already locked shard_a on its way to shard_d and then stalled there.

With H replaced by six single-row autocommit UPDATEs:

H2: starting per-row writes   14:57:25.008
H2: a,b,c committed           14:57:25.009      <-- 1 ms
Q:  updating shard_a          14:57:28.011
Q:  SUCCEEDED                 14:57:28.012      <-- 1 ms, no wait
H2: d finally done            14:57:37.013      <-- only the genuinely contended row waited

This also explains the observation in #5167 that the reporter could not account for — "the blocker and the blocked statement were both this heartbeat UPDATE, within the same database". pg_blocking_pids() reports direct blockers only. The heartbeat UPDATE genuinely is the blocker; the projection transaction that is the real root is one hop further down the chain and invisible unless you walk it recursively. No second concurrent writer is required to produce that picture.

Why #622 alone was not enough. It made this rarer without making it milder. The transitions that still reach the database are precisely the correlated ones — node start, node stop, rollout, agent rebalance — when many shards on a database transition together, which is when a batch is at its widest and when projection batches are being cancelled and re-established on those same rows.

What changed

One row per transaction. The flush is now one single-row UPDATE per shard on the same rented connection, each in its own implicit transaction. That is the load-bearing property — six single-row statements inside one transaction would reproduce the convoy exactly, which is why they must stay separate round trips rather than one batched Npgsql command (several statements in one command share an implicit transaction). What a batch amortizes is the connection, which is all #553 ever needed: N single-row statements on one rented connection still cost one rent. Writes go in shard-name order so two writers racing over the same rows cannot take their locks in opposite orders.

A change guard. The SET list was unconditional, so every flush gave every matched row a new tuple version whether anything had changed or not — on a small, hot table (two byte-identical replays of a 6-row batch left 31 dead tuples). An is distinct from guard makes the replay an UPDATE 0. This is secondary to the convoy, but each avoided rewrite is also an avoided row lock.

Semantics are otherwise unchanged: update-only, never INSERT, never touches last_seq_id / last_updated, missing rows skipped silently, and the #5048 failure_* columns keep their write / clear-on-Started / leave-alone rule.

Verification

DaemonTests green on net9.0 (300 tests). Two new tests, both falsified — each fails against the shape it guards:

test	falsified against	failure
a_contended_row_does_not_hold_the_locks_of_the_rows_already_written	the same loop wrapped in an explicit transaction	first row's telemetry never becomes visible while the batch is parked
replaying_identical_telemetry_does_not_rewrite_the_row	the is distinct from guard neutralized	xmin changes on a byte-identical replay

The lock test holds a real row lock from a second connection, then asserts the earlier row's write is visible from a third connection while the batch is still parked — and that an unrelated writer touching that row completes under a 2s lock_timeout rather than queuing behind the batch.

Companion contract change in JasperFx.Events: JasperFx/jasperfx#630.

🤖 Generated with Claude Code

https://claude.ai/code/session_017CTtw2kVRSZKp1p5RTxgAy