Foundation for binary event serialization (#4515 — Phase 1, Rich-mode only)

[jeremydmiller]

Closes part of #4515. Phase 1 ships the foundation and a working vertical slice on the Rich append path; the Quick-mode write paths, BulkEventAppender, and binary upcasters are deferred as explicit, documented scope (see §Limitations).

TL;DR

• Per-event-type binary serialization, coexisting with JSON on the same mt_events table.
• New optional package: Marten.MemoryPack.
• Additive schema change only: one new bdata bytea NULL column. No migration of existing data required.
• 5/5 new MemoryPack integration tests pass; existing EventSourcing tests unaffected.
• Limitation: Rich append mode only in this cut — the Quick modes throw at store-build time with a clear remediation recipe if a binary event type is registered without AppendMode = Rich.

Coexistence design

Schema is purely additive:

column	type	populated when
data	jsonb NOT NULL	JSON event: full payload. Binary event: literal '{}'::jsonb placeholder.
bdata	bytea NULL (new)	Binary event: serialized bytes. JSON event: NULL.

Per-row discriminator: bdata IS NULL. data NOT NULL stays intact, so existing rows in an upgraded store have bdata = NULL and read through the JSON path unchanged.

Registration API — both forms, per the maintainer's recommended option

// 1. Attribute-driven (uses opts.Events.DefaultBinarySerializer as resolver)
[BinaryEvent]
[MemoryPackable]
public partial record TripStarted(Guid TripId, string DriverName, DateTimeOffset StartedAt);

// 2. Fluent (explicit per-type serializer; wins over the default)
opts.Events.UseBinarySerializer<TripStarted>(new MemoryPackEventSerializer());

Resolution order on EventMapping construction:

1. Explicit UseBinarySerializer<T>(...) for that type
2. [BinaryEvent] + opts.Events.DefaultBinarySerializer
3. Otherwise, plain JSON (existing path)

If [BinaryEvent] is set but neither a per-type nor a default serializer is wired, the EventMapping constructor throws with a remediation message naming both registration entry points.

Marten.MemoryPack package

opts.Events.AppendMode = EventAppendMode.Rich;   // Phase 1 limitation
opts.Events.UseMemoryPackSerializer();           // sets DefaultBinarySerializer

Then [BinaryEvent] [MemoryPackable] types just work. The serializer itself is ~5 lines over MemoryPackSerializer.

Surface area touched

Area	What
Marten/Events/{IEventBinarySerializer,BinaryEventAttribute}.cs	New interface + attribute
Marten/Events/Schema/EventBdataColumn.cs	New nullable bytea column
Marten/Events/Schema/EventsTable.cs	Add column; pin at SELECT position 3
Marten/Events/IEventStoreOptions.cs	DefaultBinarySerializer + UseBinarySerializer<T>
Marten/Events/EventGraph.cs	Registry + ResolveBinarySerializerFor
Marten/Events/EventMapping.cs	BinarySerializer + IsBinary
Marten/Events/EventDocumentStorage.cs	Per-row JSON-vs-binary dispatch in Resolve / ResolveAsync
Marten/EventStorage/Rich/RichAppendEventOperation.cs	Bind bdata at column slot 4
Marten/EventStorage/Rich/RichEventStorageDescriptor.cs	SerializeEventBdata closure
Marten/EventStorage/Dialects/PostgresEventStoreDialect.cs	bdata in IsCoreColumn; binary-aware closures; Quick-mode guard
Marten/EventStorage/ClosedShapeEventDocumentStorage.cs	Skip(3) → Skip(4); +4 ordinal offset
Marten.MemoryPack/	New library — MemoryPackEventSerializer + sugar extensions
Marten.MemoryPack.Tests/	5 integration tests
Marten.slnx	New MemoryPack/ folder
build/build.cs	TestMemoryPack target hung off TestExtensions
Directory.Packages.props	MemoryPack 1.21.4

Tests

Marten.MemoryPack.Tests — all 5 pass against local Postgres:

Test	What it pins
can_round_trip_a_single_binary_event	Basic MemoryPack write/read
multiple_binary_events_replay_in_order	Multi-event stream round-trip
json_and_binary_events_coexist_on_one_stream	Mixed stream: JSON and binary events interleaved; both read back correctly
binary_events_land_in_bdata_column_jsoned_events_land_in_data_column	On-disk shape verification — confirms binary rows have data = '{}' + bdata non-null; JSON rows have data = real JSON + bdata = NULL
pre_existing_json_rows_still_read_after_feature_is_in_place	Upgrade-backfill: rows written before the feature still read correctly

Regression check: EventSourcingTests.end_to_end_event_capture_and_fetching 83/83 passing — JSON-only events still flow correctly through the dispatched read path.

Phase 1 limitations (explicit, documented)

These are written up in the new docs/events/binary-serialization.md page; calling them out here so the review knows what's intentionally deferred:

1. EventAppendMode.Rich only. The default QuickWithServerTimestamps and Quick go through the mt_quick_append_events PG function, whose signature needs a parallel bdata bytea[] parameter to carry binary payloads. BuildQuickDescriptor / BuildQuickWithServerTimestampsDescriptor throw at store-build time if a binary event type is registered while AppendMode is non-Rich, with the remediation recipe in the exception message. Quick-mode support is Phase 2.
2. No BulkEventAppender support. Same root cause — COPY column shape needs bdata adding. Follow-up.
3. No binary upcaster support. Marten's JSON upcasters operate on JSON payloads; a separate typed transform shape for binary is needed. Filing as a separate follow-up issue.

Docs

• New page: docs/events/binary-serialization.md (coexistence design, registration, on-disk shape, migration story, full constraints list)
• Wired into the Events sidebar in docs/.vitepress/config.mts
• Link from docs/events/optimizing.md (Performance & Scalability) pointing to the new page

Both pass markdownlint --disable MD009 and cspell locally.

Out of scope (deferred to follow-ups)

• Quick-mode write paths (the PG function mt_quick_append_events needs a bdata bytea[] parameter and matching operation binds)
• BulkEventAppender COPY-path binary support
• Binary event upcasters / downcasters (filed as a separate follow-up issue — see [#4579](https://github.com/JasperFx/marten/issues/4579))
• Integration tests for projections / FetchForWriting / async daemon against binary events — the same Resolve/ResolveAsync dispatch is used everywhere, so the existing per-row tests cover the core read contract; richer integration tests can land in Phase 2

🤖 Generated with Claude Code