GH-3519: Isolate agent start failures so one wedged agent doesn't skip its siblings

[jeremydmiller]

Contributes to #3519

Scope

This PR contains the blast radius of the event-subscription startup race described in #3519. It does not claim to fix the underlying race (see "What this does not fix" below).

The bug

In DurabilityMode.Solo, NodeAgentController.startAllAgentsAsync iterated each agent family's agents and awaited StartAgentAsync per agent inside a single per-family try/catch:

foreach (var controller in _agentFamilies.Values)
{
    try
    {
        var allAgents = await controller.AllKnownAgentsAsync();
        foreach (var uri in allAgents)
        {
            await StartAgentAsync(uri);   // one throw here...
        }
    }
    catch (Exception e) { /* ...aborts the whole family loop */ }
}

When one agent's start throws — e.g. an event-subscription shard that loses a first-assignment startup race with high-water detection, surfacing as AgentStartingException → System.Exception: Unable to start a subscription agent for Identity: … — the throw breaks out of the family loop. Every sibling agent listed after the wedged one is never started that tick, and because the 30s reevaluation re-runs the same ordering, those siblings stay dead for the life of the process.

This matches the field report in #3519: "two of three shards dead on that boot; only one ran … its progression frozen while its store's high-water climbed thousands of events."

The fix

Isolate each agent's start in its own try/catch (log + continue) so a single failing agent only blocks itself:

• AllKnownAgentsAsync() failure still logs and skips to the next family (unchanged semantics).
• Each StartAgentAsync(uri) now catches independently — a wedged agent is logged and skipped, and its healthy siblings in the same family still start.
• The wedged agent is still retried on the next reevaluation tick exactly as before; it just no longer takes its siblings down with it.

Test

CoreTests/Runtime/Agents/agent_start_isolation.cs builds a Solo-mode NodeAgentController with a fake family of three agents where the middle one throws on start, and asserts the sibling listed after the failing one still lands in Agents. Verified red before the change (fails on exactly that assertion) and green after.

What this does not fix

The reason the wedged shard fails to start in the first place is a startup race inside the JasperFx projection daemon (high-water detection not yet primed for a SubscribeFromPresent shard during the first assignment evaluation). That is tracked separately and still requires the CritterWatch multi-store repro to validate a root-cause fix. This PR is the defensive, independently-valuable half: the race can no longer wedge unrelated sibling agents on the node.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FKAxzuZ36VP6UPcTQf3MUs