Skip to content

Stop writing node records that describe the wrong thing (GH-3698) - #3724

Merged
jeremydmiller merged 1 commit into
mainfrom
gh-3698/node-record-accuracy
Jul 29, 2026
Merged

Stop writing node records that describe the wrong thing (GH-3698)#3724
jeremydmiller merged 1 commit into
mainfrom
gh-3698/node-record-accuracy

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Two independent defects found while diagnosing #3698.
Neither affects assignment behaviour, but both made wolverine_node_records actively misleading during the
reported incident — the reporter drew conclusions from these rows.

Independent of #3723 and #3719; this branch is off main.

NodeStarted records carried a random node_number

NodeRecord.For reads Options.Durability.AssignedNodeNumber, but the Balanced start path called the
observer one line before adopting the number PersistAsync handed back:

current.AssignedNodeNumber = await _persistence.PersistAsync(current, _cancellation.Token);

await _observer.NodeStarted();                                        // <-- reads the default

_runtime.Options.Durability.AssignedNodeNumber = current.AssignedNodeNumber;

DurabilitySettings.AssignedNodeNumber defaults to
Guid.NewGuid().ToString().GetDeterministicHashCode(), so every NodeStarted row in a Balanced cluster
described itself with a random value unrelated to the node — the reporter's 1827159984 / -1841074440,
which they attributed to the skeleton-node insert. The Solo path in StartLocally.cs already had the two in
the right order; this makes Balanced match.

While here: "Node {NodeNumber} successfully assumed leadership" was being passed Options.UniqueNodeId, a
Guid. It now logs both, each under its own name.

DormantNodeEjected fired for nodes that were never ejected

ejectStaleNodes continues past DeleteAsync for every node it deliberately spares — this node itself,
a leader a follower may not evict, and above all a node inside the GH-3604 hysteresis window — and then
handed the whole stale list to _observer.StaleNodes, which writes one DormantNodeEjected record per
node it is given.

So a node that blipped stale for a single tick and recovered was recorded as ejected while still
heartbeating. That is exactly the contradiction reported against the production cluster: an "ejected" record
sitting next to a fresh health_check. Only the nodes DeleteAsync actually ran for are reported now.

Tests

Three behavioural tests, all red-checked against the unfixed source:

  • node_reregisters_after_ejection.the_node_started_record_sees_the_assigned_node_number
  • ejection_hysteresis_tests.records_no_ejection_for_a_peer_still_inside_the_hysteresis_window
  • ejection_hysteresis_tests.records_no_ejection_for_a_stale_leader_a_follower_may_not_delete

plus records_an_ejection_for_the_peer_it_actually_deleted guarding the positive case. Full
wolverine.slnx Release build clean; 203/203 in CoreTests.Runtime.Agents.

🤖 Generated with Claude Code

https://claude.ai/code/session_013ojaHNzLcE8m5krwfTNPxw

…H-3698)

Two independent defects found while diagnosing #3698, both of which made the
wolverine_node_records table actively misleading during the reported incident.

NodeRecord.For reads Options.Durability.AssignedNodeNumber, but the Balanced start
path called the observer one line BEFORE adopting the number PersistAsync handed
back, so every NodeStarted row carried the per-process default --
Guid.NewGuid().ToString().GetDeterministicHashCode() -- a random value unrelated to
the node it describes. The reporter attributed those to the skeleton-node insert.
The Solo path in StartLocally.cs already had the two in the right order. While here,
"Node {NodeNumber} successfully assumed leadership" was being passed UniqueNodeId, a
Guid; it now logs both, each under its own name.

ejectStaleNodes continues past DeleteAsync for every node it deliberately spares --
this node itself, a leader a follower may not evict, and above all a node inside the
GH-3604 hysteresis window -- and then reported the whole stale list to the observer,
which writes one DormantNodeEjected record per node. A node that blipped stale for a
single tick and recovered was recorded as ejected while still heartbeating, which is
exactly the contradiction reported: an "ejected" record next to a fresh health_check.
Only the nodes actually deleted are reported now.

All three behavioural tests were red-checked against the unfixed source.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ojaHNzLcE8m5krwfTNPxw
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant