Daemon block observability: loud Block<T> failures, fault semantics, narrowed teardown catches (GH-506, GH-507)

[jeremydmiller]

Closes #506. Closes #507. Part of the community open-issue sweep; companion to #508 (GH-505) — the three ride the next release together.

Block (GH-506)

• Default OnError is no longer Debug.WriteLine — that call is compiled away in release builds, which made any exception escaping a block action invisible in production (the two-day silent projection outage in Sharded + tenant-partitioned daemon: per-tenant subscription agents silently never start / composite agent silently stalls at a fixed sequence (no log output) marten#4941). The last-resort default now writes to Console.Error + Trace; real hosts should (and the daemon now does) assign OnError to actual logging.
• The error callback can never kill the consumer loop. Per-item action failures go to OnError and the loop keeps consuming. If the callback itself throws, error handling is terminally broken and the block faults.
• Fault semantics — a dead consumer is now observable by producers. On a terminal consumer failure the block latches, the channel is completed with the failure (releasing any producer block-waiting on a full bounded channel with ChannelClosedException instead of hanging forever — the post-2.19.1 backpressure interaction called out on the issue), Post/PostAsync throw InvalidOperationException with the terminal failure as inner exception, and Block<T>.Failure exposes it. WaitForCompletionAsync stays quiet on a faulted block so teardown paths don't blow up.
• OnError moves onto IBlock<T> (virtual default on BlockBase<T>, so downstream BlockBase subclasses — e.g. Wolverine's — compile unchanged); BlockSet delegates to its top block, BatchingChannel to its inner block, which makes PushUpstream chains wireable.
• RetryBlock behavior is unchanged: it remains deliberately unbounded (fix(blocks): Post no longer silently drops items on a full bounded channel #485 self-repost deadlock) and its action-level try/catch means it never faults its inner block.

Daemon OnError wiring (GH-506)

GroupedProjectionExecution (processing + upstream grouping block), SubscriptionExecutionBase, and ProjectionExecution wire OnError to the daemon logger plus ReportCriticalFailureAsync; SubscriptionAgent's command block logs + reports critical failure; ShardStateTracker's publish block logs. Nothing escaping a daemon block can be invisible anymore.

Narrowed teardown catches (GH-507)

The state-based catch when (_cancellation.IsCancellationRequested) clauses (and SubscriptionExecutionBase's unconditional OCE swallow, which had drifted even wider than the issue described) now classify the caught exception via the new CancellationExceptions.IsCancellationLike:

• Silent (genuine teardown side effects): OperationCanceledException incl. wrapped/aggregated, ObjectDisposedException, and DbExceptions whose SqlState is 57014 (query_canceled) or class 08 (connection teardown). Provider-agnostic via DbException.SqlState (.NET 5+), no Npgsql reference needed; wrapped provider exceptions (e.g. MartenCommandException) are handled by the inner-exception walk.
• Logged before discarding: everything else. A 23514 missing-partition failure is a schema problem regardless of the CTS state. Logged at Information per the teardown context — the same failure outside teardown still takes the LogError + critical-failure path.

ProjectionExecution.processRangeAsync had the opposite defect (no teardown clause at all — it critical-failed shards on teardown side effects); it gets the same classified clause.

Tests

• CoreTests/Blocks/BlockErrorHandlingTests (5): OnError receives item+exception and the loop survives; throwing callback faults the block and Post/PostAsync throw; a fault releases a producer parked on a full bounded channel (TCS-gated, no sleeps in the arrange); WaitForCompletionAsync quiet on fault; the default sink survives release builds (Console.Error capture).
• EventTests/Daemon/TeardownExceptionHandlingTests (15): IsCancellationLike classification matrix (OCE/wrapped/aggregated/ODE/SqlState theory incl. the incident's 23514), plus GroupedProjectionExecution end-to-end: a fake DbException("23514") thrown while the shard CTS fires mid-batch-build is logged and not promoted to ReportCriticalFailureAsync; a genuine OCE in the same window stays silent.

Full CoreTests 458/458, EventTests 498/498, EventStoreTests 72/72, solution build clean.

🤖 Generated with Claude Code