Daemon tryStartAgentAsync swallows a failed/partial shard start, leaving a possible orphan and no propagated cause (wolverine#3519 residual)

[jeremydmiller]

Follow-up from the wolverine#3519 investigation (Wolverine-side recovery landed in wolverine#3550; the de-masking was #536). This tracks the daemon-side root cause that #536 only surfaced a message for.

Where

src/JasperFx.Events/Daemon/JasperFxAsyncDaemon.cs, tryStartAgentAsync (~lines 220-290):

try
{
    ...
    await agent.StartAsync(request).ConfigureAwait(false);   // ~273
    agent.MarkHighWater(highWaterMark);
    _agents = _agents.AddOrUpdate(agent.Name.Identity, agent); // ~276 — only reached on success
    syncTenantPolling();
}
catch (Exception ex)
{
    Logger.LogError(ex, "Error trying to start agent {ShardName}", agent.Name.Identity);
    return false;                                             // ~282 — swallowed
}

Two problems

1. A failed start can leave an orphaned agent that is never stopped.
If agent.StartAsync(request) throws after spinning up any background execution / acquiring the shard's progression or advisory lock, control jumps to the catch and returns false without registering the agent in _agents and without stopping/disposing it. The registry never knows about it, so no later StopAgentAsync can reach it. On multi-store / Wolverine-managed hosts this is a candidate explanation for the permanent wedge in wolverine#3519: the first start loses a race with high-water detection, faults mid-start, and the orphan keeps holding shard state so every subsequent retry (fresh agent object) keeps failing to acquire — the randomized victim never recovers for the process lifetime. startContinuousShardAsync only disposes the agent when tryStartAgentAsync returns false for the idempotent reason, not when it faulted mid-start.

2. The failure cause is swallowed, forcing callers to reverse-engineer it.
tryStartAgentAsync returns a bare bool, collapsing "already running (idempotent skip)", "start faulted", and "positioning failed" into a single false. StartAgentAsync(ShardName) then has to guess the reason via describeStartFailure(name) (added in #536) by re-inspecting daemon state after the fact. The actual caught exception — the real cause — is only logged, never propagated, so the guess can be wrong.

Suggested direction

• On a mid-start fault, stop/dispose the partially-started agent (and reconcile syncTenantPolling / any activated tenant) before returning, so no orphan can hold the shard.
• Consider distinguishing "idempotent skip" from "faulted" in the return (e.g. an enum/result, or rethrow the captured exception through a typed ShardStartException at the tryStartAgentAsync boundary) so StartAgentAsync(ShardName) can attach the real cause instead of reconstructing a probable one.

Repro

JasperFx/CritterWatch MultiStoreHost sample consistently reproduces the wolverine#3519 wedge (main Marten store with UseWolverineManagedEventSubscriptionDistribution = true + two AddMartenStore<T> ancillary stores, three async shards, DurabilityMode.Solo); the randomized victim differs per boot.

Filed from the wolverine#3519 follow-up.

[jeremydmiller]

Fully resolved on main:

• The swallowed-cause half was fixed in JasperFxAsyncDaemon.StartAgentAsync throws a causeless 'Unable to start a subscription agent' — failed-start reasons are swallowed (startContinuousShardAsync bool ignored, silent false paths) #534 — tryStartAgentAsync stashes the caught start exception in _lastStartFailures and StartAgentAsync(ShardName) attaches it to the ShardStartException it throws.
• The orphaned partial start half was fixed in GH-540: tear down a faulted shard start so it can't orphan the shard #541 (commit 6e18200) — a faulted agent.StartAsync is now hard-stopped in the catch so it can't keep holding the shard's execution loop.

Both land in the 2.32.0 release.