feat(#367,#368,#369): JasperFx.Events 2.36.1 — classified shard failures, widened natural key contract, drain-timeout docs

Directory.Packages.props

@@ -36,17 +36,31 @@
              onto every published ShardState, so the ExtendedProgressionWriter can carry the running node
              into WriteExtendedProgressionAsync's running_on_node column under managed distribution.
              JasperFx 2.34.0: jasperfx#555 (tenant-scoped QueryByTagsAsync + EventQuery.TenantId explorer
-             reads, #353) and jasperfx#557 (extended-progression writer shutdown drain). -->
-        <PackageVersion Include="JasperFx" Version="2.34.0" />
-        <PackageVersion Include="JasperFx.Events" Version="2.34.0" />
+             reads, #353) and jasperfx#557 (extended-progression writer shutdown drain).
+             JasperFx 2.36.1: three Polecat issues ride on this line —
+             (a) jasperfx#564: DaemonSettings.StopAndDrainTimeout, the configurable per-shard bound on the
+                 daemon's graceful stop-and-drain (docs, polecat#367);
+             (b) jasperfx#565: ShardFailureCategory / IEventFailureContext / EventFailureDetails /
+                 ShardFailure + ShardState.Failure — the classified reason a shard paused or stopped.
+                 Polecat implements IEventFailureContext on its read-path exceptions and persists the
+                 failure_* extended-progression columns (polecat#368). The daemon has NO fallback
+                 type-name sniffing, so a store that does not implement this classifies every failure as
+                 ShardFailureCategory.Other;
+             (c) jasperfx#569/#571: NaturalKeyEventMapping.Extractor widens from Func<object, object?> to
+                 Func<IEvent, object?> — SOURCE BREAKING, which is why the bump and the call-site change
+                 land together — plus reachable NaturalKeyBuilder / NaturalKeyFor() and loud failure on an
+                 unbindable [NaturalKeySource] (polecat#369).
+             Also carries jasperfx#568/#572 daemon race fixes, picked up for free. -->
+        <PackageVersion Include="JasperFx" Version="2.36.1" />
+        <PackageVersion Include="JasperFx.Events" Version="2.36.1" />
         <!-- Pin the sibling JasperFx packages for parity with Marten 9's matrix
              even though Polecat doesn't currently reference them directly —
              keeps the lockstep matrix coherent when transitive resolution
              surfaces them through JasperFx / JasperFx.Events updates. The
              RuntimeCompiler 5.x line is the active continuation of the 4.x
              lineage; do not pin against the parallel stale 2.0.x series. -->
         <PackageVersion Include="JasperFx.RuntimeCompiler" Version="5.0.0" />
-        <PackageVersion Include="JasperFx.SourceGenerator" Version="2.34.0" />
+        <PackageVersion Include="JasperFx.SourceGenerator" Version="2.36.1" />
         <PackageVersion Include="Microsoft.Data.SqlClient" Version="7.0.0" />
         <PackageVersion Include="Microsoft.Extensions.DependencyInjection" Version="10.0.0" />
         <PackageVersion Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="10.0.0" />
@@ -91,7 +105,7 @@
         <PackageVersion Include="StronglyTypedId" Version="1.0.0-beta08" />
 
         <!-- Source generators (matched to JasperFx.Events above) -->
-        <PackageVersion Include="JasperFx.Events.SourceGenerator" Version="2.34.0" />
+        <PackageVersion Include="JasperFx.Events.SourceGenerator" Version="2.36.1" />
 
         <!-- Build automation -->
         <PackageVersion Include="Nuke.Common" Version="9.0.4" />

docs/events/natural-keys.md

@@ -25,6 +25,40 @@ Every event type that sets or changes the natural key must be declared through t
 
 Events that do not affect the natural key (like `NkOrderItemAdded` in the example above) do not need any mapping.
 
+A `[NaturalKeySource]` method that cannot be bound to an extraction strategy is now a configuration-time
+error — `AssembleAndAssertValidity()` throws an `InvalidProjectionException` naming the method and the
+reason. Previously such a method registered nothing at all, silently, and the lookup table was simply
+never written for that event type.
+
+## Explicit Mappings with `NaturalKeyFor()`
+
+When attribute discovery cannot bind your method — or when you would simply rather be explicit — register
+the mapping directly from the projection:
+
+```cs
+public class OrderProjection: SingleStreamProjection<Order, Guid>
+{
+    public OrderProjection()
+    {
+        NaturalKeyFor(x => x
+            // From the event body
+            .SetBy<ProductRegistered>(e => new ProductCode(e.Code))
+            // From the whole IEvent, when the key depends on event metadata
+            .SetByEvent<ProductCodeChanged>(e => new ProductCode(e.Data.NewCode)));
+    }
+}
+```
+
+An explicit registration replaces any discovered mapping for the same event type, so it always wins over
+the attribute.
+
+::: warning
+The natural key has to be a function of **the event alone**. The lookup table is maintained inline at
+append time, where no prior aggregate exists under an `Async` snapshot lifecycle, so an extraction that
+depends on the current aggregate state has nothing to read. This is the constraint the attribute path now
+enforces as well.
+:::
+
 ## Storage
 
 Polecat automatically creates and manages a lookup table (prefixed with `pc_`) for each aggregate type that has a natural key configured. The table maps natural key values to stream ids and is:

docs/events/projections/async-daemon.md

@@ -61,6 +61,42 @@ Unlike Marten's PostgreSQL `LISTEN/NOTIFY`, Polecat uses **polling** to detect n
 // Default: 500ms
 ```
 
+## Graceful Shutdown and the Drain Timeout
+
+When a projection or subscription shard is stopped, the daemon does not simply cancel it. It first tries to
+*drain* the agent: let the in-flight page of events finish being applied, then flush the shard's progression row
+so the next start picks up exactly where this one left off. `StopAndDrainTimeout` bounds how long the daemon
+waits for that drain on **a single** shard:
+
+```cs
+// The default is 5 seconds
+opts.Projections.StopAndDrainTimeout = TimeSpan.FromSeconds(30);
+```
+
+The bound applies to every stop path: stopping one agent, stopping all agents (the `SIGTERM`/host shutdown
+path), and the internal stop-if-already-running replacement that happens when an agent is reassigned.
+
+**Why you would raise it.** If the drain is cut off before the progression flush lands, the shard restarts
+against a stale progression row and throws `ProgressionProgressOutOfOrderException` on its next start. Raise
+the timeout when in-flight batches legitimately take longer than five seconds — a large `BatchSize`, expensive
+projection code, heavy rebuild load, or a slow or contended SQL Server. This is most visible shutting down a
+host with a large agent universe: a [database-per-tenant](/documents/multi-tenancy) deployment with thousands of
+(projection × tenant) shards all draining inside a Kubernetes termination grace window.
+
+::: tip
+A per-shard bound is only useful if the process lives long enough to spend it. Match a raised
+`StopAndDrainTimeout` with the host's own `HostOptions.ShutdownTimeout` and, on Kubernetes, the pod's
+`terminationGracePeriodSeconds`.
+:::
+
+**Why you would lower it.** A deployment that would rather cut a wedged shard loose quickly and take the
+progression replay hit — to keep node failover and reassignment latency low, for instance — can set it below
+the default.
+
+**Opting out.** `Timeout.InfiniteTimeSpan`, or any non-positive value, removes the separate bound so the drain
+is limited only by the daemon's own cancellation. Be aware that this means a genuinely wedged shard can hold up
+shutdown indefinitely.
+
 ## Waiting for Non-Stale Data
 
 ### CatchUpAsync

src/Polecat.Tests/Daemon/event_loader_resiliency_tests.cs

@@ -2,6 +2,7 @@
 using JasperFx.Events.Projections;
 using Microsoft.Data.SqlClient;
 using Polecat.Events.Daemon;
+using Polecat.Exceptions;
 using Polecat.Tests.Harness;
 
 namespace Polecat.Tests.Daemon;
@@ -73,8 +74,11 @@ await InsertPoisonPillEventAsync("unknown_event_type", "{\"data\": \"poison\"}",
         var request = CreateRequest(0, highWater, batchSize: 100,
             skipUnknown: false, skipSerialization: false);
 
-        await Should.ThrowAsync<InvalidOperationException>(
+        // #368 / jasperfx#565: a typed exception that declares its own ShardFailureCategory. This used to
+        // be a bare InvalidOperationException, which the daemon could only classify as Other.
+        var ex = await Should.ThrowAsync<UnknownEventTypeException>(
             loader.LoadAsync(request, CancellationToken.None));
+        ex.Category.ShouldBe(ShardFailureCategory.UnknownEventType);
     }
 
     // ===== SkipSerializationErrors =====
@@ -122,8 +126,12 @@ await InsertPoisonPillEventAsync(questStartedTypeName, "CORRUPTED",
         var request = CreateRequest(0, highWater, batchSize: 100,
             skipUnknown: false, skipSerialization: false);
 
-        await Should.ThrowAsync<InvalidOperationException>(
+        // #368 / jasperfx#565: EventSerialization is kept distinct from UnknownEventType on purpose —
+        // bad data is a different operator action from a missing registration.
+        var ex = await Should.ThrowAsync<EventDeserializationFailureException>(
             loader.LoadAsync(request, CancellationToken.None));
+        ex.Category.ShouldBe(ShardFailureCategory.EventSerialization);
+        ex.EventTypeName.ShouldBe(questStartedTypeName);
     }
 
     // ===== Both skip options enabled =====

src/Polecat.Tests/Daemon/event_loader_tests.cs

@@ -2,6 +2,7 @@
 using JasperFx.Events.Projections;
 using Microsoft.Data.SqlClient;
 using Polecat.Events.Daemon;
+using Polecat.Exceptions;
 using Polecat.Tests.Harness;
 
 namespace Polecat.Tests.Daemon;
@@ -104,6 +105,74 @@ public async Task load_empty_range_returns_empty_page()
         page.Count.ShouldBe(0);
     }
 
+    // #368 / jasperfx#565: the daemon classifies a paused shard purely from what the store's exception
+    // declares through IEventFailureContext — there is deliberately no fallback type-name sniffing. These
+    // two pin the throw sites that used to raise a bare InvalidOperationException, which classified as
+    // ShardFailureCategory.Other with no event details at all.
+    [Fact]
+    public async Task unresolvable_event_type_throws_a_classified_failure_naming_the_sequence()
+    {
+        await InsertEventsAsync(1);
+        var seqId = (await GetAllSeqIdsAsync()).Single();
+        await CorruptDotNetTypeAsync(seqId, "Nope.NotARealEventType, Nope");
+
+        var loader = CreateLoader();
+        var request = CreateRequest(0, seqId, batchSize: 100);
+
+        var ex = await Should.ThrowAsync<UnknownEventTypeException>(
+            async () => await loader.LoadAsync(request, CancellationToken.None));
+
+        ShardFailure.For(ex, DateTimeOffset.UtcNow).Category.ShouldBe(ShardFailureCategory.UnknownEventType);
+        ex.Sequence.ShouldBe(seqId);
+    }
+
+    [Fact]
+    public async Task corrupted_event_body_throws_a_classified_failure_naming_the_event_type()
+    {
+        await InsertEventsAsync(1);
+        var seqId = (await GetAllSeqIdsAsync()).Single();
+        var alias = theStore.Database.Events.EventMappingFor(typeof(QuestStarted)).EventTypeName;
+        await CorruptEventBodyAsync(seqId);
+
+        var loader = CreateLoader();
+        var request = CreateRequest(0, seqId, batchSize: 100);
+
+        var ex = await Should.ThrowAsync<EventDeserializationFailureException>(
+            async () => await loader.LoadAsync(request, CancellationToken.None));
+
+        // The acceptance case from the issue: EventSerialization, with the failing sequence AND the
+        // store's type alias — the alias a consumer can act on, not the assembly-qualified dotnet_type.
+        var failure = ShardFailure.For(ex, DateTimeOffset.UtcNow);
+        failure.Category.ShouldBe(ShardFailureCategory.EventSerialization);
+        failure.Event.ShouldNotBeNull();
+        failure.Event.Sequence.ShouldBe(seqId);
+        failure.Event.EventTypeName.ShouldBe(alias);
+    }
+
+    private async Task CorruptDotNetTypeAsync(long seqId, string dotNetType)
+    {
+        await using var conn = await OpenConnectionAsync();
+        await using var cmd = conn.CreateCommand();
+        cmd.CommandText = "UPDATE [dbo].[pc_events] SET dotnet_type = @type WHERE seq_id = @seq;";
+        cmd.Parameters.AddWithValue("@type", dotNetType);
+        cmd.Parameters.AddWithValue("@seq", seqId);
+        await cmd.ExecuteNonQueryAsync();
+    }
+
+    private async Task CorruptEventBodyAsync(long seqId)
+    {
+        // Well-formed JSON (so the native `json` column accepts it) that cannot bind to the event type at
+        // all, so the row reads back fine but STJ throws on materialization — the shape of the failure the
+        // issue is about. A JSON array rather than a bad property value: property NAMES are subject to the
+        // serializer's naming policy, so an unmatched name would silently bind to the default instead.
+        await using var conn = await OpenConnectionAsync();
+        await using var cmd = conn.CreateCommand();
+        cmd.CommandText = "UPDATE [dbo].[pc_events] SET data = @data WHERE seq_id = @seq;";
+        cmd.Parameters.AddWithValue("@data", "[1, 2, 3]");
+        cmd.Parameters.AddWithValue("@seq", seqId);
+        (await cmd.ExecuteNonQueryAsync()).ShouldBe(1);
+    }
+
     private PolecatEventLoader CreateLoader()
     {
         return new PolecatEventLoader(theStore.Database.Events, theStore.Options, theStore.Options.ConnectionString);