Ask for the active node numbers once per node, not once per database (GH-3846)

[erdtsieck]

Closes #3846.

DurabilityAgent's recovery timer wants the active node numbers. The only call that yields them, LoadAllNodesAsync, also selects the entire assignment table to populate ActiveAgents — which this caller never reads — and there is one durability agent per message database. So a per-node fact was being fetched per database.

Measured on a 512-database, five-node, ~10,000-agent cluster (details in the issue): 76 calls and 772,000 rows a second against the main store, with Client:ClientWrite at 164 of 170 average active sessions and Lock:transactionid at 0.04. Each call also held a pooled connection, so the heartbeat writes timed out, the leader declared healthy nodes stale, and reassigning their agents churned the very table being read.

The change

ActiveNodeNumberCache fetches the numbers at most once per ScheduledJobPollingTime for the whole node, and DurabilityAgent asks it instead of the store. Same shape as PersistenceMetricsSweeper for the metrics polling in GH-3375 — a ConditionalWeakTable keyed on the runtime, so one instance per node and nothing to dispose.

Behaviour is unchanged: the caller already tolerates data one interval old, since that is how often it looks. Failures are deliberately not cached and not swallowed — DurabilityAgent keeps its own try/catch and decides what a failed lookup means, and a failure leaves the previous value in place rather than remembering an empty one as the truth.

The double-check inside the gate is what does the collapsing: the databases whose timers queue up behind the first caller are served by its fetch.

Tests

active_node_number_cache_tests — five tests, green:

• returns the assigned node numbers
• 50 concurrent callers, standing in for the node's databases on one timer tick, produce one underlying fetch
• refetches once the polling interval has passed, and sees the new numbers
• a failed lookup reaches the caller and is not cached
• one cache per runtime

Not in scope

LoadAllNodesAsync itself still returns assignments for every caller, including the ones that only want nodes. Narrowing that means an interface change across four persistence implementations, so it seemed worth keeping separate from the fix for the hot path.

https://claude.ai/code/session_01LZkaXgbeEgiT9BrZR4aXsb

[Copilot]

Pull request overview

This PR reduces persistence load in clustered/sharded deployments by introducing a node-wide cache for active Wolverine node numbers, so DurabilityAgent no longer hits LoadAllNodesAsync() once per database per polling tick.

Changes:

• Added ActiveNodeNumberCache (per-runtime, polling-interval TTL) to collapse active-node lookups across all databases on the same node.
• Updated DurabilityAgent recovery polling to use the cache instead of querying node persistence directly.
• Added CoreTests coverage for correctness, concurrency collapsing, refetch behavior, failure behavior, and per-runtime scoping.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
src/Wolverine/Persistence/ActiveNodeNumberCache.cs	Introduces the per-runtime, node-wide cache for active node numbers.
src/Testing/CoreTests/Persistence/active_node_number_cache_tests.cs	Adds tests validating cache behavior and concurrency collapsing.
src/Persistence/Wolverine.RDBMS/DurabilityAgent.cs	Switches recovery polling to use ActiveNodeNumberCache instead of querying persistence directly.

Suppressed comments (2)

src/Wolverine/Persistence/ActiveNodeNumberCache.cs:75

• Update the cache via a single snapshot assignment (and publish it with Volatile.Write) so readers never see mismatched numbers/expiry values.

            var nodes = await _runtime.Storage.Nodes.LoadAllNodesAsync(token).ConfigureAwait(false);
            var numbers = nodes.Select(x => x.AssignedNodeNumber).ToList();

            _numbers = numbers;
            _staleAt = DateTimeOffset.UtcNow.Add(_runtime.DurabilitySettings.ScheduledJobPollingTime);

            return numbers;

src/Wolverine/Persistence/ActiveNodeNumberCache.cs:93

• Use Volatile.Read when checking cache state on the fast path to guarantee safe publication of the snapshot across threads.

    private bool fresh(out IReadOnlyList<int> numbers)
    {
        var current = _numbers;
        if (current != null && DateTimeOffset.UtcNow < _staleAt)
        {
            numbers = current;
            return true;
        }

        numbers = Array.Empty<int>();
        return false;

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.