Stamp AssignedNodeNumber on daemon-published ShardState so managed distribution populates running_on_node (marten#5001)

[jeremydmiller]

Summary

Under Balanced, Wolverine-managed subscription distribution with extended-progression tracking on, mt_event_progression.running_on_node stays NULL even though the projection agents are genuinely assigned to a real node. The sibling telemetry (heartbeat, agent_status, last_updated) writes correctly in the same row on the same path — only running_on_node never populates.

Downstream tracking issue: JasperFx/marten#5001. Repro (green, pins the gap with a // FLIP-WHEN-FIXED marker): CritterWatch RunningOnNodeMultiNodeTests.

The load-bearing fix is here in JasperFx.Events (the daemon publish surface). A one-line wire-up in Wolverine (which owns the node assignment) is a companion task, described below. No Marten change is required.

Root cause (traced end to end)

ExtendedProgressionWriter.OnNext already has the carry logic, and its own comment says the value is expected to be "stamped by a distribution layer (e.g. Wolverine-managed subscription distribution)":

// JasperFx.Events/Daemon/ExtendedProgressionWriter.cs
if (value.RunningOnNode == null && value.AssignedNodeNumber != 0)
    value.RunningOnNode = value.AssignedNodeNumber;

But nothing ever stamps AssignedNodeNumber on a runtime-published ShardState. A grep of both trees finds AssignedNodeNumber = … in exactly one place — Marten's ShardStateSelector, which sets it only when reading the assigned_node column back from the DB (and nothing writes that column; it defaults 0). Every ShardState a SubscriptionAgent publishes carries AgentStatus/LastHeartbeat but AssignedNodeNumber == 0, so the carry block is dead and running_on_node persists NULL. The carry logic is correct; its input is never supplied on the runtime publish path.

Proposed fix — Part 1 (JasperFx.Events): one stamp point on ShardStateTracker

This mirrors exactly how DatabaseIdentifier is already stamped for every state the daemon publishes:

// ShardStateTracker.cs
public int AssignedNodeNumber { get; set; } // 0 = unset

public ValueTask PublishAsync(ShardState state)
{
    if (state.ShardName == ShardState.HighWaterMark) HighWaterMark = state.Sequence;

    // existing single stamp point for DatabaseIdentifier:
    state.DatabaseIdentifier ??= DatabaseIdentifier;

    // add: same idea for the assigned node
    if (AssignedNodeNumber != 0 && state.AssignedNodeNumber == 0)
        state.AssignedNodeNumber = AssignedNodeNumber;

    return _block.PostAsync(state);
}

There is one ShardStateTracker per database-daemon, so this single stamp covers the subscription agents' progress publications and the high-water marks alike — and it lands before ExtendedProgressionWriter observes, because the tracker delivers the same ShardState instance to its _listeners in subscription order (the same in-place mechanism Marten's SkippedEventsCountObserver already relies on).

Ordering note (why the tracker, not a late observer)

The stamp must be at the source (the tracker), not a Wolverine-subscribed observer: ExtendedProgressionWriter is subscribed during daemon construction, so any observer Wolverine subscribes afterward runs after the writer and would be too late. Stamping inside PublishAsync sidesteps ordering entirely.

Companion task — Part 2 (Wolverine): set it when the node owns the daemon

Wolverine owns the assignment and knows the local node number (runtime.Options.Durability.AssignedNodeNumber — the value written to wolverine_nodes.node_number). When EventStoreAgents takes ownership of a database's daemon (subscribeObservers / FindDaemonAsync, where it already holds the IProjectionDaemon), set:

daemon.Tracker.AssignedNodeNumber = runtime.Options.Durability.AssignedNodeNumber;

Why the local node number is correct: EventStoreAgents only builds/owns the daemon for databases assigned to this node, and that daemon only runs the agents assigned to this node. So every ShardState flowing through that tracker belongs to an agent running here — the local node number is exactly the "assigned node" for all of them. No per-agent assignment-grid lookup is needed.

With both parts, the existing ExtendedProgressionWriter carry + Marten's WriteExtendedProgressionAsync persist running_on_node. Marten needs no change.

Acceptance

running_on_node reflects the projection agent's assigned node number under Balanced managed distribution, populated on the same heartbeat/transition publications that already write heartbeat/agent_status. CritterWatch RunningOnNodeMultiNodeTests flips its runningOnNode.ShouldBeNull(...) assertion to ShouldBe(assignedNodeNumber).

References

• JasperFx/marten#5001 — downstream tracking issue (full end-to-end trace)
• Wire the extended-progression write path (release 9.17.1) marten#4990 — extended-progression write path (WriteExtendedProgressionAsync)
• Extended progression tracking is write-path inert: heartbeat/agent_status/pause_reason columns are never persisted by any daemon path (never wired, not a regression) #537 / feat(#537): wire the extended progression write path into the async daemon #538 — ExtendedProgressionWriter observer
• CritterWatch #750 — extended progression tracking end-to-end

---

🤖 Drafted with Claude Code