Bound the node record table by row count, not just age (GH-3701)

[jeremydmiller]

Closes #3701.

The report

wolverine_node_records reached 36,135,221 rows / 16 GB (15 GB heap + 867 MB index) in five days on a production cluster — a diagnostic table with no foreign keys whose only reader is FetchRecentRecordsAsync, and which nothing in Wolverine ever asks for more than ~100 rows from.

Two defects

1. The row cap had no call site. INodeAgentPersistence.DeleteOldNodeRecordsAsync(int retainCount) was implemented for the relational stores, tested, and never invoked outside those tests — exactly as reported.

2. The pruning that did run only bounds by age. DeleteOldNodeEventRecords deletes against NodeEventRecordExpirationTime (5 days). Age is no ceiling at all when the write rate is high: one AssignmentChanged row per agent per assignment decision, on a cluster with thousands of distributed agents, is millions of rows a day — all of them inside the window. Every one of those 36M rows was younger than the cutoff.

And a third, found on the way in

The age sweep was appended to buildOperationBatch behind this guard:

#pragma warning disable CS0649 // Field is never assigned to
    private DateTimeOffset? _lastNodeRecordPruneTime;
#pragma warning restore CS0649

The suppression says it outright — the field is never assigned, so isTimeToPruneNodeEventRecords() always returned true and the intended hourly throttle never engaged. The delete went out on every recovery cycle, i.e. every 5 seconds by default. On the reporting cluster that is a full scan of a 16 GB table every five seconds, on the main Wolverine database, while it was already unhealthy for an unrelated reason (#3698).

The fix

• Durability.NodeRecordRetention — a hard row cap, default 10,000, applied by a new TrimNodeRecordsCommand. Zero or negative keeps the old age-only behavior.
• Durability.NodeRecordPruningPeriod — default hourly. Both deletes now run on a dedicated timer against the Main store instead of riding the 5-second recovery batch, which is what the dead throttle was supposed to achieve.
• MultiTenantedMessageStore now delegates DeleteOldNodeRecordsAsync to Main.Nodes, matching LogRecordsAsync / FetchRecentRecordsAsync. It was inheriting the interface's no-op default, so a multi-tenanted store would never have trimmed even once a call site existed — and one main store with hundreds of tenant databases is precisely the reporter's shape.
• Sqlite, MySQL and Oracle implement the row cap; they had also been falling through to the no-op default. MySQL rejects a LIMIT inside a subquery over the table being deleted, and Oracle pays for a NOT IN over a FETCH FIRST subquery, so both resolve a floor id first and delete as a primary-key range scan.

On the write volume itself

The reporter offered to split off the "one AssignmentChanged row per agent per decision is very fine-grained" point. That is a fair separate issue and this PR does not touch it — but with a row cap in place, that volume is now a bounded cost rather than an unbounded one.

Tests

• PostgresqlTests/Agents/node_record_retention.cs — seeds 50 rows, starts a real host, and asserts the table drains to the cap with no test-side call to DeleteOldNodeRecordsAsync. Red baseline verified: 54 rows remain without the fix.
• Store-level retention tests for Sqlite, MySQL and Oracle, mirroring the existing Postgres and SQL Server ones (newest rows retained, zero-retain no-op, under-cap no-op, empty table).
• Full wolverine.slnx Release build clean.

🤖 Generated with Claude Code