Add SerializerV2 foundation

openspec/IMPLEMENTATION_ORDER.md

@@ -1,10 +1,14 @@
-# Akka.NET 1.6 Transport & Serialization Epic — Implementation Order
+# Akka.NET 1.6 Transport & Serialization Epic - Implementation Order
 
 ## Overview
 
-Five OpenSpec changes implement the Akka.NET 1.6 transport and serialization overhaul. This file defines the order in which they must be implemented, their dependencies, and the completion criteria for each milestone.
+Akka.NET 1.6 makes `SerializerV2` the canonical serialization abstraction and uses it as the payload contract for both existing compatibility paths and the new high-throughput remoting path.
 
-Each milestone is implemented on its own branch off of `feature/openspec-init`. After a milestone is complete and reviewed, its OpenSpec change is archived via `openspec archive`.
+The important sequencing decision is that `SerializerV2` cannot be implemented as a core-only API change. Once `Serialization.FindSerializerFor()` returns `SerializerV2`, classic Akka.Remote, Akka.Persistence, delivery, DistributedData, and other serializer call sites are affected immediately. Therefore the first active milestone is an atomic foundation: V2 core plus the classic remoting and persistence bridges needed to keep the repository green.
+
+The second sequencing decision is that source-generated MessagePack serialization must validate the V2 API before Artery envelopes are introduced. Artery should not bake in assumptions about size hints, bytes-written reporting, manifests, async behavior, or V1 fallback until generated serializers have passed through `Serialization`, classic remoting, and persistence.
+
+The third sequencing decision is to replace the previous `streams-tcp-transport` plan. The high-throughput path should be an Artery-style remoting stack beside classic remoting, not a retrofit through `EndpointWriter`, `AkkaProtocolTransport`, `AkkaPduCodec`, and `AssociationHandle.Write(ByteString)`.
 
 ## Performance Baseline
 
@@ -20,125 +24,155 @@ Captured on `dev` branch (commit 467cbb510), .NET 10.0, Release, ServerGC, Linux
 | 25 | 670,422 |
 | 30 | 679,964 |
 
-**Peak: ~680K msgs/sec.** The new transport (after Milestone 3) must exceed this.
+**Peak: ~680K msgs/sec.** The new Artery TCP transport must exceed this before it can replace DotNetty as the preferred remoting path.
 
 ## Milestones
 
-### Milestone 1: `modernize-akka-io-tcp` (Spec 1)
+### Milestone 1: `modernize-akka-io-tcp`
 
-**Branch**: `feature/spec1-modernize-akka-io-tcp` (off `feature/openspec-init`)
+**Branch**: already merged via PR #8132
 **OpenSpec change**: `openspec/changes/modernize-akka-io-tcp/`
-**Tasks file**: `openspec/changes/modernize-akka-io-tcp/tasks.md`
-
-**What it does**: Replace ByteString with System.Memory, replace SocketAsyncEventArgs with Stream + Pipe, add IStreamProvider abstraction.
+**Status**: completed baseline work
 
-**Implementation strategy**: Dark period approach. Delete ByteString, use compiler errors as todo list, fix module by module until `dotnet build` succeeds. Then replace SAEA with Stream+Pipe and get tests passing.
+**What it did**: Modernized Akka.IO TCP around `ReadOnlySequence<byte>`, `System.IO.Pipelines`, and `ITransportConnection`.
 
-**Completion criteria**:
-- `dotnet build -warnaserror` passes on net10.0
-- `dotnet test -c Release --framework net10.0` — all Akka.IO TCP tests pass
-- `dotnet test -c Release --framework net10.0` — all Akka.Streams TCP tests pass
-- `dotnet test -c Release --framework net10.0` — all Akka.Remote tests pass (with DotNetty still present but ByteString removed)
-- No ByteString references remain in the codebase
-- IStreamProvider + TcpStreamProvider exist and are used by TcpOutgoingConnection/TcpIncomingConnection
-
-**After completion**: Review with human. Archive via `openspec archive modernize-akka-io-tcp`.
+**Relevant outcome**: This work is useful for Artery and for user-facing Akka.Streams TCP, but the Artery MVP should not route its hot path through materialized Akka.Streams TCP.
 
 ---
 
-### Milestone 2: `serializer-v2` (Spec 4)
+### Milestone 2: `serializer-v2` - Atomic V2 Foundation
 
-**Branch**: `feature/spec4-serializer-v2` (off Milestone 1's merged branch)
+**Branch**: `feature/spec2-serializer-v2-foundation`
 **OpenSpec change**: `openspec/changes/serializer-v2/`
 **Tasks file**: `openspec/changes/serializer-v2/tasks.md`
 
-**What it does**: Add SerializerV2 base class, SerializerV1Adapter, MessagePackSerializer, modify Serialization.cs infrastructure, mechanical port of simple internal Protobuf serializers.
+**What it does**: Makes `SerializerV2` the canonical serialization abstraction and fixes every immediate compatibility bridge required by that API break.
+
+**Must be atomic**:
 
-**Note**: This was originally planned as parallel with Milestone 1 but is sequenced after it to avoid merge conflicts in Serialization.cs and MessageSerializer.cs. The ByteString removal in Milestone 1 also affects serializer code paths.
+- `SerializerV2` core API
+- `SerializerV1Adapter`
+- `Serialization.cs` internal storage and public return-type changes
+- classic Akka.Remote V2 bridge preserving classic wire compatibility
+- Akka.Persistence V2 bridge preserving stored event and snapshot compatibility
+- compile fallout in delivery, DistributedData, tests, and other direct serializer call sites
 
 **Completion criteria**:
-- SerializerV2, SerializerV1Adapter exist in `src/core/Akka/Serialization/`
-- Akka.Serialization.V2 package exists with MessagePackSerializer, AkkaWriter, AkkaReader
-- `Serialization.cs` uses SerializerV2 internally, auto-wraps V1
-- `FindSerializerFor()` returns SerializerV2
-- Hand-written MessagePack serializer round-trips through full pipeline
-- All existing serialization tests pass (V1 auto-wrapped)
-- Simple internal Protobuf serializers ported to V2 base (same IDs, same wire format)
-- `dotnet test -c Release --framework net10.0` passes
 
-**After completion**: Review with human. Archive via `openspec archive serializer-v2`.
+- `SerializerV2` and `SerializerV1Adapter` exist in `src/core/Akka/Serialization/`.
+- `Serialization.cs` stores V2 serializers internally and auto-wraps V1 serializers.
+- `FindSerializerFor()` and `FindSerializerForType()` return `SerializerV2`.
+- V1 serializers continue to work through `SerializerV1Adapter`.
+- classic Akka.Remote sends and receives messages through V2 while preserving existing classic wire format.
+- Akka.Persistence events and snapshots write/read through V2 while preserving existing stored data compatibility.
+- Old journal events and snapshots remain readable.
+- New V2 payloads can be persisted and recovered.
+- Delivery, DistributedData, and other direct serializer call sites compile and preserve behavior.
+- API approval tests are updated for the intentional 1.6 API break.
+- `dotnet build -warnaserror` passes.
+- Akka.Serialization, Akka.Remote, and Akka.Persistence tests pass.
+
+**After completion**: Review with human. Archive via `openspec archive serializer-v2` only after the repository is green across serialization, remoting, and persistence.
 
 ---
 
-### Milestone 3: `akka-io-tls-support` (Spec 2)
+### Milestone 3: `messagepack-sourcegen-validation`
 
-**Branch**: `feature/spec2-akka-io-tls` (off Milestone 2's merged branch)
-**OpenSpec change**: `openspec/changes/akka-io-tls-support/`
-**Tasks file**: `openspec/changes/akka-io-tls-support/tasks.md`
+**Branch**: `feature/spec3-messagepack-sourcegen-validation`
+**OpenSpec change**: `openspec/changes/messagepack-sourcegen-validation/`
+**Tasks file**: `openspec/changes/messagepack-sourcegen-validation/tasks.md`
 
-**What it does**: Add TlsStreamProvider, server-side TLS handshake, TlsSettings config, TlsSetup programmatic API.
+**What it does**: Implements the source-generated MessagePack serializer path and uses it to validate the real `SerializerV2` API before Artery envelopes depend on it.
 
 **Completion criteria**:
-- TlsStreamProvider wraps SslStream, handshake in ConnectAsync
-- Server-side TLS handshake in TcpIncomingConnection with timeout
-- All existing DotNetty TLS HOCON keys parse into TlsSettings
-- TlsSetup programmatic config works and overrides HOCON
-- TLS integration tests pass (self-signed certs, mutual TLS, validation)
-- `dotnet test -c Release --framework net10.0` passes
 
-**After completion**: Review with human. Archive via `openspec archive akka-io-tls-support`.
+- `Akka.Serialization.V2` package exists.
+- Concrete sealed `AkkaWriter` and `AkkaReader` wrap MessagePack directly.
+- Source generator emits serializers for `[AkkaSerializable]` / `[AkkaField]` types.
+- Generated serializers round-trip through `Serialization.cs`.
+- Generated serializers send over classic remoting.
+- Generated serializers persist and recover events.
+- Generated serializers save and load snapshots.
+- V1 and generated V2 serializers coexist.
+- `SizeHint`, unknown-size behavior, manifest handling, bytes-written/result semantics, and oversized payload behavior are validated.
+
+**After completion**: Review with human. Archive via `openspec archive messagepack-sourcegen-validation`.
 
 ---
 
-### Milestone 4: `streams-tcp-transport` (Spec 3)
+### Milestone 4: `artery-tcp-remoting`
+
+**Branch**: `feature/spec4-artery-tcp-remoting`
+**OpenSpec change**: `openspec/changes/artery-tcp-remoting/`
+**Tasks file**: `openspec/changes/artery-tcp-remoting/tasks.md`
+
+**What it does**: Adds a new Artery-style TCP remoting stack beside classic remoting, using the validated `SerializerV2` payload contract.
 
-**Branch**: `feature/spec3-streams-transport` (off Milestone 3's merged branch)
-**OpenSpec change**: `openspec/changes/streams-tcp-transport/`
-**Tasks file**: `openspec/changes/streams-tcp-transport/tasks.md`
+**Implementation strategy**:
 
-**What it does**: Replace DotNetty with Akka.Streams TCP transport. FrameBufferWriter for integrated framing + serialization. Binary PDU codec. Delete DotNetty entirely.
+- Do not route the hot path through classic `EndpointWriter` / `AkkaProtocolTransport` / `AkkaPduCodec`.
+- Start with plaintext TCP and one ordinary stream.
+- Add control stream and reliable system-message delivery before lanes and compression.
+- Preserve classic remoting as a compatibility path.
+- Defer QUIC to Akka.NET 1.7.
 
 **Completion criteria**:
-- StreamsTcpTransport implements Transport abstraction
-- FrameBufferWriter enables single-buffer frame construction
-- BinaryPduCodec replaces Protobuf AkkaPduCodec
-- All `akka.remote.dot-netty.tcp.*` HOCON config works unchanged
-- DotNetty directory and NuGet deps deleted
-- Two ActorSystems communicate via new transport
-- Cluster formation works
-- All non-DotNetty-specific Akka.Remote specs pass
-- `dotnet test -c Release --framework net10.0` passes
-
-**After completion**: Review with human. Archive via `openspec archive streams-tcp-transport`.
+
+- `ArteryRemoting : RemoteTransport` exists and is selected by configuration.
+- Artery TCP listener and outbound association setup work.
+- TCP framing uses `AKKA` magic + 1-byte stream id connection header and 4-byte little-endian frame length.
+- Handshake includes address and UID and is tied to association incarnation state.
+- Basic remote actor messaging works over Artery TCP.
+- Control stream exists for handshake, liveness, quarantine, and system-message ACK/NACK traffic.
+- Reliable system-message delivery preserves remoting correctness.
+- Quarantine semantics are UID-scoped and compatible with existing remote lifecycle expectations.
+- Classic remoting still works independently.
+
+**After completion**: Review with human. Archive via `openspec archive artery-tcp-remoting`.
 
 ---
 
-### Milestone 5: `transport-performance` (Spec 5)
+### Milestone 5: `transport-performance`
 
-**Branch**: `feature/spec5-performance` (off Milestone 4's merged branch)
+**Branch**: `feature/spec5-artery-performance`
 **OpenSpec change**: `openspec/changes/transport-performance/`
 **Tasks file**: `openspec/changes/transport-performance/tasks.md`
 
-**What it does**: Run RemotePingPong, implement flush batching and optimizations, exceed DotNetty baseline.
+**What it does**: Measures and tunes the Artery TCP stack against the DotNetty baseline.
 
 **Completion criteria**:
-- RemotePingPong on new transport exceeds 680K msgs/sec peak
-- Flush batching implemented and tuned
-- Pipe thresholds benchmarked and configured
-- Results documented
+
+- RemotePingPong on Artery TCP exceeds the 680K msgs/sec DotNetty baseline.
+- Artery envelope codec materially outperforms the classic protobuf PDU path in allocation and throughput microbenchmarks.
+- Generated MessagePack serializers show lower allocation and better throughput than V1 adapter fallback.
+- Bounded queue and backpressure tests prove memory cannot grow unbounded under slow receivers.
+- Latency impact of batching is measured and configurable.
+- Results are documented.
 
 **After completion**: Review with human. Archive via `openspec archive transport-performance`.
 
+---
+
+### Milestone 6: `akka-io-tls-support`
+
+**Branch**: `feature/spec6-akka-io-tls`
+**OpenSpec change**: `openspec/changes/akka-io-tls-support/`
+**Tasks file**: `openspec/changes/akka-io-tls-support/tasks.md`
+
+**What it does**: Adds TLS support for the modernized TCP infrastructure and eventually Artery TCP.
+
+**Sequencing note**: TLS is important, but plaintext Artery TCP should prove the envelope, association, system-message, and performance model first. TLS can then wrap the same connection model.
+
 ## Orchestration
 
-Each milestone is executed by an OpenProse orchestrator that:
+Each milestone is executed by an OpenSpec-oriented orchestrator that:
 
-1. Reads the OpenSpec `tasks.md` for the current milestone
-2. Creates a new branch for the milestone
-3. Delegates tasks to worker agents (Sonnet gophers for mechanical changes, Opus for design decisions)
-4. After each task group: attempts `dotnet build`, captures errors, iterates
-5. After all tasks: runs `dotnet test -c Release --framework net10.0`, fixes failures
-6. Commits progress incrementally
-7. Reports completion status to human for review
+1. Reads the OpenSpec `tasks.md` for the current milestone.
+2. Creates a new branch for the milestone.
+3. Delegates mechanical call-site work to focused workers when useful.
+4. Runs focused builds and tests after each task group.
+5. Captures compile fallout instead of hiding it behind compatibility shims.
+6. Runs the milestone completion test suite.
+7. Reports completion status to human for review.
 
-Only one milestone is executed per orchestrator run. Human reviews before proceeding to the next.
+Only one milestone should be executed per implementation run. Human review happens before proceeding to the next milestone.

openspec/changes/akka-io-tls-support/design.md

@@ -1,6 +1,6 @@
 ## Context
 
-Spec 1 (`modernize-akka-io-tcp`) introduces `IStreamProvider` — an abstraction that returns a connected `Stream` from connection parameters. `TcpStreamProvider` returns a plain `NetworkStream`. The TCP connection actor reads/writes the `Stream` via `System.IO.Pipelines.Pipe` and never interacts with the socket directly. This design makes TLS a provider swap: `TlsStreamProvider` wraps the `NetworkStream` in `SslStream` and completes the TLS handshake inside `ConnectAsync`, returning an authenticated `Stream`.
+`modernize-akka-io-tcp` introduces `IStreamProvider`, an abstraction that returns a connected `Stream` from connection parameters. `TcpStreamProvider` returns a plain `NetworkStream`. The TCP connection actor reads/writes the `Stream` via `System.IO.Pipelines.Pipe` and never interacts with the socket directly. This design makes TLS a provider swap: `TlsStreamProvider` wraps the `NetworkStream` in `SslStream` and completes the TLS handshake inside `ConnectAsync`, returning an authenticated `Stream`.
 
 The current DotNetty transport supports TLS via `DotNetty.Handlers.Tls.TlsHandler` which itself wraps `SslStream`. The HOCON configuration covers: file-based certificates (`ssl.certificate.path` + `password`), Windows certificate store lookup (`ssl.certificate.thumbprint` + `store-name` + `store-location`), mutual authentication, hostname validation, and validation suppression for development. All of these must continue to work unchanged.
 
@@ -46,7 +46,7 @@ Reference implementation: TurboMQTT `TlsStreamProvider.cs` (~60 lines) and `Fake
 
 **Decision:** Parse `akka.remote.dot-netty.tcp.ssl.*` keys into a new `TlsSettings` class that produces `SslClientAuthenticationOptions` and `SslServerAuthenticationOptions`.
 
-**Rationale:** Zero user configuration changes. The HOCON key names stay the same even though the underlying transport changes from DotNetty to Akka.Streams. Users migrating to Akka.NET 1.6 don't need to rewrite their TLS config.
+**Rationale:** Zero user configuration changes. The HOCON key names stay the same even though the underlying transport moves away from DotNetty-specific TLS machinery. Users migrating to Akka.NET 1.6 don't need to rewrite their TLS config.
 
 Config mapping:
 - `ssl.certificate.path` + `ssl.certificate.password` → `new X509Certificate2(path, password)`

openspec/changes/akka-io-tls-support/proposal.md

@@ -1,6 +1,6 @@
 ## Why
 
-Akka.Remote requires TLS support for production deployments. The current implementation relies on DotNetty's `TlsHandler`, which is being removed (Spec 3). With Spec 1 (`modernize-akka-io-tcp`) introducing `IStreamProvider` and replacing `SocketAsyncEventArgs` with `Stream` + `Pipe`, TLS becomes a simple `IStreamProvider` implementation — `TlsStreamProvider` wraps `SslStream` around `NetworkStream`, handshake happens inside `ConnectAsync`, and the TCP connection actor never knows it's encrypted. This was previously impossible because SAEA couldn't work with `SslStream`.
+Akka.Remote requires TLS support for production deployments. The current implementation relies on DotNetty's `TlsHandler`, which cannot be reused by the modern Akka.IO / future Artery TCP path. With `modernize-akka-io-tcp` introducing `IStreamProvider` and replacing `SocketAsyncEventArgs` with `Stream` + `Pipe`, TLS becomes a simple provider implementation: `TlsStreamProvider` wraps `SslStream` around `NetworkStream`, handshake happens inside `ConnectAsync`, and the TCP connection actor never knows it is encrypted. This was previously impossible because SAEA could not work with `SslStream`.
 
 ## What Changes