Foundation for binary event serialization (#4515 — Phase 1, Rich-mode only)

Directory.Packages.props

@@ -24,6 +24,7 @@
     <PackageVersion Include="Lamar" Version="7.1.1" />
     <PackageVersion Include="Lamar.Microsoft.DependencyInjection" Version="15.0.0" />
     <PackageVersion Include="Marten" Version="8.2.1" />
+    <PackageVersion Include="MemoryPack" Version="1.21.4" />
     <PackageVersion Include="Microsoft.CodeAnalysis.Analyzers" Version="3.3.4" />
     <PackageVersion Include="Microsoft.CodeAnalysis.CSharp" Version="4.8.0" />
     <PackageVersion Include="Meziantou.Analyzer" Version="2.0.257" />

build/build.cs

@@ -44,7 +44,8 @@ class Build : NukeBuild
         .DependsOn(TestNodaTime)
         .DependsOn(TestAspnetcore)
         .DependsOn(TestPostGIS)
-        .DependsOn(TestPgVector);
+        .DependsOn(TestPgVector)
+        .DependsOn(TestMemoryPack);
 
     Target Init => _ => _
         .Executes(() =>
@@ -138,6 +139,18 @@ class Build : NukeBuild
                 .SetFramework(Framework));
         });
 
+    Target TestMemoryPack => _ => _
+        .ProceedAfterFailure()
+        .Executes(() =>
+        {
+            DotNetTest(c => c
+                .SetProjectFile("src/Marten.MemoryPack.Tests")
+                .SetConfiguration(Configuration)
+                .EnableNoBuild()
+                .EnableNoRestore()
+                .SetFramework(Framework));
+        });
+
     Target TestPostGIS => _ => _
         .ProceedAfterFailure()
         .Executes(() =>

docs/.vitepress/config.mts

@@ -223,6 +223,7 @@ const config: UserConfig<DefaultTheme.Config> = {
             { text: 'Natural Keys', link: '/events/natural-keys' },
             { text: 'Dynamic Consistency Boundary', link: '/events/dcb' },
             { text: 'Optimizing Performance', link: '/events/optimizing' },
+            { text: 'Binary Event Serialization', link: '/events/binary-serialization' },
 
             {
               text: 'Projections Overview', link: '/events/projections/', collapsed: true, items: [

docs/events/binary-serialization.md

@@ -0,0 +1,168 @@
+# Binary Event Serialization <Badge type="tip" text="9.3" />
+
+Marten can serialize individual event types to a binary wire format
+([MemoryPack](https://github.com/Cysharp/MemoryPack),
+[MessagePack](https://msgpack.org/), or anything else implementing
+`IEventBinarySerializer`) instead of the default JSON, trading a few of JSON's
+ergonomic wins for a meaningful throughput and storage-size improvement on
+hot streams. See [#4515](https://github.com/JasperFx/marten/issues/4515) for
+the design discussion.
+
+The opt-in is **per event type** — binary-serialized and JSON-serialized events
+coexist in the same `mt_events` table, so the feature can be rolled out on an
+existing store with **no migration of existing data**.
+
+## How it works
+
+A second column, `bdata bytea NULL`, sits alongside the existing `data jsonb NOT NULL`
+on `mt_events`. The row-level discriminator is `bdata IS NULL`:
+
+| When | `data` | `bdata` |
+| --- | --- | --- |
+| Event uses the JSON serializer | full JSON payload | `NULL` |
+| Event uses an `IEventBinarySerializer` | the placeholder `'{}'::jsonb` | the serialized bytes |
+
+On read, Marten inspects `bdata`:
+
+- `NULL` → existing JSON deserialization path. Pre-feature rows continue to work without conversion.
+- non-null → `IEventBinarySerializer.Deserialize(eventType, bytes)`.
+
+Because the discriminator is on the row and the serializer is resolved per
+event type, the same stream can carry rows of either format with no special
+handling at the call site.
+
+## Quick start with `Marten.MemoryPack`
+
+The companion `Marten.MemoryPack` NuGet package ships a ready-to-use
+`IEventBinarySerializer` over MemoryPack:
+
+```shell
+dotnet add package Marten.MemoryPack
+```
+
+Mark event types you want to serialize as binary with both
+`[BinaryEvent]` (so Marten picks them up) and `[MemoryPackable]` (so MemoryPack
+can serialize them):
+
+```csharp
+using Marten.Events;
+using MemoryPack;
+
+[BinaryEvent]
+[MemoryPackable]
+public partial record TripStarted(Guid TripId, string DriverName, DateTimeOffset StartedAt);
+```
+
+Wire MemoryPack as the store-wide fallback for `[BinaryEvent]` types:
+
+```csharp
+using Marten.MemoryPack;
+
+var store = DocumentStore.For(opts =>
+{
+    opts.Connection(connectionString);
+
+    // Phase 1 limitation — see "Constraints" below.
+    opts.Events.AppendMode = EventAppendMode.Rich;
+
+    // Wire MemoryPack as DefaultBinarySerializer. [BinaryEvent]-marked
+    // event types resolve to this serializer on registration.
+    opts.Events.UseMemoryPackSerializer();
+});
+```
+
+Now `TripStarted` writes through MemoryPack to `bdata`; un-marked events
+continue to write JSON to `data`.
+
+## Registration ergonomics
+
+Two equivalent ways to opt an event type in:
+
+```csharp
+// 1. Attribute-driven — uses opts.Events.DefaultBinarySerializer as the resolver.
+[BinaryEvent]
+[MemoryPackable]
+public partial record TripEnded(Guid TripId, DateTimeOffset EndedAt);
+
+// 2. Fluent — wire an explicit per-type serializer (overrides any default).
+opts.Events.UseBinarySerializer<TripEnded>(new MemoryPackEventSerializer());
+```
+
+Resolution order on `EventMapping` construction:
+
+1. Explicit `opts.Events.UseBinarySerializer<TEvent>(...)` for that type.
+2. `[BinaryEvent]` attribute + `opts.Events.DefaultBinarySerializer`.
+3. Otherwise, plain JSON (existing path).
+
+If a type carries `[BinaryEvent]` but no per-type serializer was wired AND
+`DefaultBinarySerializer` is `null`, Marten throws at the first append with a
+remediation message naming both registration entry points.
+
+## Bring your own serializer
+
+`IEventBinarySerializer` is small enough to implement directly against any
+binary format — MessagePack, protobuf, etc.:
+
+```csharp
+public interface IEventBinarySerializer
+{
+    byte[] Serialize(Type type, object data);
+    object Deserialize(Type type, byte[] data);
+}
+```
+
+The serializer is a singleton — keep its state thread-safe.
+
+## On-disk shape
+
+For binary events, `data` holds the literal `{}` placeholder so the existing
+`data jsonb NOT NULL` constraint stays intact (no schema relaxation):
+
+```sql
+-- binary-serialized event
+select type, data::text, bdata is null
+from mt_events where seq_id = 42;
+--    type        | data | bdata is null
+-- --------------|------|---------------
+--  trip_started | {}   | false
+
+-- JSON-serialized event in the same stream
+select type, data::text, bdata is null
+from mt_events where seq_id = 43;
+--    type                | data                            | bdata is null
+-- --------------------- |---------------------------------|---------------
+--  trip_comment_added   | {"comment": "looking good", …}  | true
+```
+
+## Migration
+
+Purely additive: the only schema change is `bdata bytea NULL` on `mt_events`.
+Existing rows have `bdata = NULL` (the column's default for prior data) and
+read through the JSON path. Marten's standard schema migration creates the
+column for existing installations — no event data conversion required.
+
+## Constraints
+
+The 9.3 cut ships with deliberate scope:
+
+- **`EventAppendMode.Rich` only.** The default `QuickWithServerTimestamps` and
+  `Quick` modes go through the `mt_quick_append_events` PostgreSQL function,
+  whose signature would need a parallel `bdata bytea[]` parameter to carry
+  binary payloads. Until that lands, `BuildQuickDescriptor` /
+  `BuildQuickWithServerTimestampsDescriptor` throw at store-build time if any
+  binary event type is registered. Workaround: set
+  `opts.Events.AppendMode = EventAppendMode.Rich;`.
+- **No bulk appender support.** `BulkEventAppender` uses Npgsql `COPY` with
+  the existing column shape; adding the `bdata` column to the COPY format
+  is part of the same Quick-mode follow-up.
+- **No upcaster support.** Marten's
+  [event upcasters](/events/versioning) operate on JSON payloads and don't
+  generalize to a `byte[]` wire form. Binary event upcasters need their own
+  typed transform shape; tracked as a deferred follow-up. For now, design
+  binary event schemas with forward-compatibility in the serializer itself
+  (MemoryPack's `[MemoryPackOrder]` evolution, for example).
+
+## See also
+
+- [Optimizing Event Store Performance and Scalability](/events/optimizing)
+- [Event Versioning](/events/versioning) (JSON upcasters)

docs/events/optimizing.md

@@ -11,6 +11,13 @@ very large data loads.
 Marten has several options to potentially increase the performance and scalability of a system that uses
 the event sourcing functionality:
 
+::: tip
+For hot streams where JSON serialization overhead dominates, see
+[Binary Event Serialization](/events/binary-serialization) — opt individual
+event types into MemoryPack (or any `IEventBinarySerializer`) on a
+per-type basis, with no migration of existing JSON-serialized events.
+:::
+
 <!-- snippet: sample_turn_on_optimizations_for_event_sourcing -->
 <a id='snippet-sample_turn_on_optimizations_for_event_sourcing'></a>
 ```cs

src/Marten.MemoryPack.Tests/Marten.MemoryPack.Tests.csproj

@@ -0,0 +1,22 @@
+<Project Sdk="Microsoft.NET.Sdk">
+    <PropertyGroup>
+        <IsPackable>false</IsPackable>
+        <Nullable>enable</Nullable>
+    </PropertyGroup>
+
+    <ItemGroup>
+        <PackageReference Include="Microsoft.NET.Test.Sdk" />
+        <PackageReference Include="Shouldly" />
+        <PackageReference Include="xunit" />
+        <PackageReference Include="xunit.runner.visualstudio">
+            <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
+            <PrivateAssets>all</PrivateAssets>
+        </PackageReference>
+        <PackageReference Include="MemoryPack" />
+    </ItemGroup>
+
+    <ItemGroup>
+        <ProjectReference Include="..\Marten.MemoryPack\Marten.MemoryPack.csproj" />
+        <ProjectReference Include="..\Marten.Testing\Marten.Testing.csproj" />
+    </ItemGroup>
+</Project>