feat: pluggable binary event serialization (IEventBinarySerializer) (#388)

[jeremydmiller]

Closes #388.

Parity with Marten's IEventBinarySerializer (marten#4515, shipped in 9.20.2). CritterWatch ships Marten/PostgreSQL and Polecat/SQL Server flavors from one source tree, and CritterWatch#896 has already opted its highest-volume internal event into binary on the Marten side — measured at field scale as −97.9% on the wire, −60.4% on disk, −73.4% append p50. Until now the SQL Server flavor kept paying the JSON cost the Marten one shed.

API

Mirrored as-is, so a store-agnostic consumer can wire either flavor:

opts.Events.UseBinarySerializer<TEvent>(serializer);   // explicit, per event type
[BinaryEvent] + opts.Events.DefaultBinarySerializer;   // attribute-driven

The coexistence design, which is the part worth copying

An additive nullable bdata varbinary(max) column beside data, with bdata IS NULL as the per-row discriminator. JSON and binary rows live in the same pc_events table, so the feature switches on for an existing store with no migration of existing event data — and switches back off just as safely.

The column is added unconditionally rather than only when a serializer is configured, so the read projection has one fixed shape. Weasel's additive delta adds it on the next schema apply; rows written before it existed have bdata = NULL and keep reading through the JSON path untouched. data is NOT NULL and typed json on SQL Server 2025, so a binary row carries the '{}' placeholder there rather than NULL.

Quick mode was in scope from the start rather than as a phase 2, per the issue's note. Polecat is QuickAppend-only, so there is no Rich-mode split to sequence and the CritterWatch store's configuration is covered by the first pass.

Implementation notes worth review attention

Every read path dispatches, not just the main one. Polecat has four independent readers over pc_events, each with its own SELECT and its own deserialization: PcEventsRowReader (the canonical one), the two DCB tag-query readers in EventOperations, the LINQ EventListHandler, and the daemon's PolecatEventLoader. bdata is pinned at ordinal 10 — after the previously-locked 0–9 block, before the optional metadata columns, which shift to 11 — so the dispatch reads a stable ordinal everywhere. The two EventOperations projections that duplicated the canonical column list by hand now compose it from PcEventsRowReader; three hand-written copies of one list is how they stop agreeing.

Masking rewrites bdata, not just data. OverwriteEventOperation (behind ApplyEventDataMasking) previously wrote only the JSON column. For a binary event that would have left the original, unmasked payload readable in bdata — for a GDPR masking operation, the entire point missed. There is a test for exactly this. The same reasoning applies to CompletelyReplaceEvent and stream compaction, where the replacement body's type decides the row's format and a stale bdata would keep being read.

A [BinaryEvent] type with no serializer configured throws instead of silently writing JSON, and the append path deliberately does not short-circuit the resolve to make the common case cheaper. A store that quietly ignored the attribute would have write-amplification characteristics that do not match its configuration, which is the problem the feature exists to solve. The resolve is a per-event-type cached ConcurrentDictionary hit.

Tests

12 new tests: the wire format (payload in bdata, '{}' in data, and a magic-header serializer so an assertion can prove the bytes really took the binary path), JSON/binary coexistence within a single stream, reading rows written before the serializer was configured, attribute-vs-explicit precedence, the misconfiguration throw, and the dispatch through each of inline projections, the async daemon, the event LINQ provider, and masking.

Full Polecat.Tests suite locally against dockerized SQL Server 2025, net10.0: 1650 total, 0 failed, 1647 passed, 3 skipped.

🤖 Generated with Claude Code

https://claude.ai/code/session_01G8tN8ApXiKhyVzia4iwmof