Extended progression: running_on_node stays NULL under managed distribution (AssignedNodeNumber never stamped on published ShardState)

[jeremydmiller]

Summary

The extended-progression running_on_node column stays NULL forever, even under genuine
Balanced multi-node managed distribution where the projection agents are assigned to a real node.
The sibling telemetry columns (heartbeat, agent_status, last_updated) write correctly on the
same code path and in the same row — only running_on_node never populates.

This is the last open gap from the extended-progression write path wired in marten#4990
(MartenDatabase.WriteExtendedProgressionAsync) / #537–#538 (ExtendedProgressionWriter).
Core #750 is otherwise confirmed live on 9.17.1.

Root cause (traced end to end)

The write op consumes state.RunningOnNode:

// Marten: MartenDatabase.EventStorage.cs — WriteExtendedProgressionAsync
.With("node", (object?)state.RunningOnNode ?? DBNull.Value, NpgsqlDbType.Integer)

The JasperFx.Events observer that drives it tries to source that value from the agent's assigned
node number:

// JasperFx.Events: ExtendedProgressionWriter.OnNext (release/2.33.0, commit 33a2f46)
if (value.RunningOnNode == null && value.AssignedNodeNumber != 0)
{
    value.RunningOnNode = value.AssignedNodeNumber;
}

…but nothing ever stamps AssignedNodeNumber onto a runtime-published ShardState. A grep of
the entire JasperFx.Events tree finds AssignedNodeNumber = … in exactly one place — Marten's
ShardStateSelector, which populates it only when reading a row back from the DB (the
assigned_node column). The telemetry ShardState the daemon publishes on the heartbeat timer /
status transitions never has it set, so AssignedNodeNumber == 0, the carry block is dead, and the
write persists NULL.

So the carry logic is correct but its input is never supplied on the runtime publish path. The fix
is to stamp the agent's actual assigned node number onto the ShardState a subscription/projection
agent publishes (the ones ShardStateTracker broadcasts and ExtendedProgressionWriter observes).
Under Wolverine-managed subscription distribution the assignment lives in
wolverine_node_assignments / is known to the coordinator, so this likely spans JasperFx.Events (the
daemon publish surface) and the distribution layer that owns the assignment — happy to split this to
a jasperfx issue if that's the better home.

Reproduction (already written, green, pinning the gap)

CritterWatch src/EventStoreCoordinationTests/RunningOnNodeMultiNodeTests.cs boots two
MartenProjectionService nodes, Balanced + UseWolverineManagedEventSubscriptionDistribution,
extended progression forced on via the CritterWatch #321 instrumentation path. The leader distributes
the Trip/Day/Distance agents across both nodes (logs show node-2 join and the distance agent
redistributing node 1 → node 2). After the daemon advances the Trip shard and a heartbeat tick lands,
projection_tests.mt_event_progression for the Trip row reads:

running_on_node = NULL,  agent_status = Running,  heartbeat = <set>,  last_seq_id = 30

heartbeat + agent_status write under genuine managed distribution; running_on_node stays NULL in
the same telemetry row → this is a write-path/publish gap, not a "single-node, nothing to stamp"
expected-NULL (that interpretation was ruled out — the agents are assigned to node_number 1, confirmed
via wolverine_node_assignments + wolverine_nodes).

The test is a characterization guard: it asserts the working columns and pins the gap with a
// FLIP-WHEN-FIXED marker on runningOnNode.ShouldBeNull(...). When this is fixed, that assertion
flips to ShouldBe(assignedNodeNumber).

Acceptance

running_on_node reflects the projection agent's assigned node number under Balanced managed
distribution, populated on the same heartbeat/transition publications that already write
heartbeat/agent_status.

References

• marten#4990 — extended-progression write path (WriteExtendedProgressionAsync)
• SELECT access no longer required on database tables. #537 / Marten.Embedded #538 — ExtendedProgressionWriter observer
• CritterWatch Multi-Tenancy through "TenantId" column in document tables #750 — extended progression tracking end-to-end

[jeremydmiller]

Traced end to end: this isn't fixable in Marten — Marten has no runtime node-number source, and the carry logic in ExtendedProgressionWriter expects a distribution layer to stamp AssignedNodeNumber. Filed the fix as JasperFx/jasperfx#550: a one-line stamp on ShardStateTracker.PublishAsync (mirroring how DatabaseIdentifier is already stamped), with a companion Wolverine wire-up (daemon.Tracker.AssignedNodeNumber = runtime.Options.Durability.AssignedNodeNumber when EventStoreAgents owns the daemon). No Marten change required.

[jeremydmiller]

Closing as resolved on the Marten side for the 9.19.0 release.

The chain

running_on_node is populated end-to-end only when three links are in place:

1. A distribution layer stamps the owning node onto the daemon's ShardStateTracker.AssignedNodeNumber. This is not Marten's native HotCold daemon (single active node — it legitimately has no managed node assignment). It is a managed-distribution host, i.e. Wolverine-managed subscription distribution, via EventSubscriptionAgentFamily.BuildAgentAsync (wolverine#3578, merged).
2. The tracker carries that number onto every published ShardState — the ShardStateTracker seam added in JasperFx 2.33.1 (PostgreSQL precision quirk surfacing on Duplicate Field #550/Integrate with ASP.NET Core Json Patch support #551). ExtendedProgressionWriter then copies AssignedNodeNumber into ShardState.RunningOnNode when the row would otherwise be null.
3. Marten persists ShardState.RunningOnNode into the running_on_node column in WriteExtendedProgressionAsync, and the async daemon subscribes the extended-progression writer to the tracker so a stamped node actually reaches that write.

Link 3 is the only link Marten owns, and it is implemented and pinned by Bug_5001_running_on_node_persisted (DaemonTests). Both cases are green on JasperFx.Events 2.35.0 (the 9.19.0 pin):

• running_on_node_is_persisted_when_the_tracker_carries_an_assigned_node — stamps Tracker.AssignedNodeNumber exactly as the distribution layer does and asserts the column is written.
• running_on_node_stays_null_when_no_distribution_layer_assigns_a_node — characterizes the reported symptom: with no distribution-layer stamp (single-node / HotCold) the sibling telemetry columns still write and running_on_node correctly stays NULL.

Conclusion

There is no remaining Marten-side defect: with a managed-distribution host stamping the node (shipped in JasperFx 2.33.1 + Wolverine wolverine#3578) running_on_node is written; without one it is correctly NULL. Nothing further ships in Marten 9.19.0 for this beyond adopting JasperFx.Events 2.35.0.

If the end-to-end column is still NULL on a Wolverine-managed multi-node setup running these versions, please reopen with the Wolverine version and daemon mode and it can be chased on the distribution-layer side.