diff --git a/openspec/IMPLEMENTATION_ORDER.md b/openspec/IMPLEMENTATION_ORDER.md index 0c017f40d8e..58cd41769fc 100644 --- a/openspec/IMPLEMENTATION_ORDER.md +++ b/openspec/IMPLEMENTATION_ORDER.md @@ -1,10 +1,14 @@ -# Akka.NET 1.6 Transport & Serialization Epic — Implementation Order +# Akka.NET 1.6 Transport & Serialization Epic - Implementation Order ## Overview -Five OpenSpec changes implement the Akka.NET 1.6 transport and serialization overhaul. This file defines the order in which they must be implemented, their dependencies, and the completion criteria for each milestone. +Akka.NET 1.6 makes `SerializerV2` the canonical serialization abstraction and uses it as the payload contract for both existing compatibility paths and the new high-throughput remoting path. -Each milestone is implemented on its own branch off of `feature/openspec-init`. After a milestone is complete and reviewed, its OpenSpec change is archived via `openspec archive`. +The important sequencing decision is that `SerializerV2` cannot be implemented as a core-only API change. Once `Serialization.FindSerializerFor()` returns `SerializerV2`, classic Akka.Remote, Akka.Persistence, delivery, DistributedData, and other serializer call sites are affected immediately. Therefore the first active milestone is an atomic foundation: V2 core plus the classic remoting and persistence bridges needed to keep the repository green. + +The second sequencing decision is that source-generated MessagePack serialization must validate the V2 API before Artery envelopes are introduced. Artery should not bake in assumptions about size hints, bytes-written reporting, manifests, async behavior, or V1 fallback until generated serializers have passed through `Serialization`, classic remoting, and persistence. + +The third sequencing decision is to replace the previous `streams-tcp-transport` plan. The high-throughput path should be an Artery-style remoting stack beside classic remoting, not a retrofit through `EndpointWriter`, `AkkaProtocolTransport`, `AkkaPduCodec`, and `AssociationHandle.Write(ByteString)`. ## Performance Baseline @@ -20,125 +24,155 @@ Captured on `dev` branch (commit 467cbb510), .NET 10.0, Release, ServerGC, Linux | 25 | 670,422 | | 30 | 679,964 | -**Peak: ~680K msgs/sec.** The new transport (after Milestone 3) must exceed this. +**Peak: ~680K msgs/sec.** The new Artery TCP transport must exceed this before it can replace DotNetty as the preferred remoting path. ## Milestones -### Milestone 1: `modernize-akka-io-tcp` (Spec 1) +### Milestone 1: `modernize-akka-io-tcp` -**Branch**: `feature/spec1-modernize-akka-io-tcp` (off `feature/openspec-init`) +**Branch**: already merged via PR #8132 **OpenSpec change**: `openspec/changes/modernize-akka-io-tcp/` -**Tasks file**: `openspec/changes/modernize-akka-io-tcp/tasks.md` - -**What it does**: Replace ByteString with System.Memory, replace SocketAsyncEventArgs with Stream + Pipe, add IStreamProvider abstraction. +**Status**: completed baseline work -**Implementation strategy**: Dark period approach. Delete ByteString, use compiler errors as todo list, fix module by module until `dotnet build` succeeds. Then replace SAEA with Stream+Pipe and get tests passing. +**What it did**: Modernized Akka.IO TCP around `ReadOnlySequence`, `System.IO.Pipelines`, and `ITransportConnection`. -**Completion criteria**: -- `dotnet build -warnaserror` passes on net10.0 -- `dotnet test -c Release --framework net10.0` — all Akka.IO TCP tests pass -- `dotnet test -c Release --framework net10.0` — all Akka.Streams TCP tests pass -- `dotnet test -c Release --framework net10.0` — all Akka.Remote tests pass (with DotNetty still present but ByteString removed) -- No ByteString references remain in the codebase -- IStreamProvider + TcpStreamProvider exist and are used by TcpOutgoingConnection/TcpIncomingConnection - -**After completion**: Review with human. Archive via `openspec archive modernize-akka-io-tcp`. +**Relevant outcome**: This work is useful for Artery and for user-facing Akka.Streams TCP, but the Artery MVP should not route its hot path through materialized Akka.Streams TCP. --- -### Milestone 2: `serializer-v2` (Spec 4) +### Milestone 2: `serializer-v2` - Atomic V2 Foundation -**Branch**: `feature/spec4-serializer-v2` (off Milestone 1's merged branch) +**Branch**: `feature/spec2-serializer-v2-foundation` **OpenSpec change**: `openspec/changes/serializer-v2/` **Tasks file**: `openspec/changes/serializer-v2/tasks.md` -**What it does**: Add SerializerV2 base class, SerializerV1Adapter, MessagePackSerializer, modify Serialization.cs infrastructure, mechanical port of simple internal Protobuf serializers. +**What it does**: Makes `SerializerV2` the canonical serialization abstraction and fixes every immediate compatibility bridge required by that API break. + +**Must be atomic**: -**Note**: This was originally planned as parallel with Milestone 1 but is sequenced after it to avoid merge conflicts in Serialization.cs and MessageSerializer.cs. The ByteString removal in Milestone 1 also affects serializer code paths. +- `SerializerV2` core API +- `SerializerV1Adapter` +- `Serialization.cs` internal storage and public return-type changes +- classic Akka.Remote V2 bridge preserving classic wire compatibility +- Akka.Persistence V2 bridge preserving stored event and snapshot compatibility +- compile fallout in delivery, DistributedData, tests, and other direct serializer call sites **Completion criteria**: -- SerializerV2, SerializerV1Adapter exist in `src/core/Akka/Serialization/` -- Akka.Serialization.V2 package exists with MessagePackSerializer, AkkaWriter, AkkaReader -- `Serialization.cs` uses SerializerV2 internally, auto-wraps V1 -- `FindSerializerFor()` returns SerializerV2 -- Hand-written MessagePack serializer round-trips through full pipeline -- All existing serialization tests pass (V1 auto-wrapped) -- Simple internal Protobuf serializers ported to V2 base (same IDs, same wire format) -- `dotnet test -c Release --framework net10.0` passes -**After completion**: Review with human. Archive via `openspec archive serializer-v2`. +- `SerializerV2` and `SerializerV1Adapter` exist in `src/core/Akka/Serialization/`. +- `Serialization.cs` stores V2 serializers internally and auto-wraps V1 serializers. +- `FindSerializerFor()` and `FindSerializerForType()` return `SerializerV2`. +- V1 serializers continue to work through `SerializerV1Adapter`. +- classic Akka.Remote sends and receives messages through V2 while preserving existing classic wire format. +- Akka.Persistence events and snapshots write/read through V2 while preserving existing stored data compatibility. +- Old journal events and snapshots remain readable. +- New V2 payloads can be persisted and recovered. +- Delivery, DistributedData, and other direct serializer call sites compile and preserve behavior. +- API approval tests are updated for the intentional 1.6 API break. +- `dotnet build -warnaserror` passes. +- Akka.Serialization, Akka.Remote, and Akka.Persistence tests pass. + +**After completion**: Review with human. Archive via `openspec archive serializer-v2` only after the repository is green across serialization, remoting, and persistence. --- -### Milestone 3: `akka-io-tls-support` (Spec 2) +### Milestone 3: `messagepack-sourcegen-validation` -**Branch**: `feature/spec2-akka-io-tls` (off Milestone 2's merged branch) -**OpenSpec change**: `openspec/changes/akka-io-tls-support/` -**Tasks file**: `openspec/changes/akka-io-tls-support/tasks.md` +**Branch**: `feature/spec3-messagepack-sourcegen-validation` +**OpenSpec change**: `openspec/changes/messagepack-sourcegen-validation/` +**Tasks file**: `openspec/changes/messagepack-sourcegen-validation/tasks.md` -**What it does**: Add TlsStreamProvider, server-side TLS handshake, TlsSettings config, TlsSetup programmatic API. +**What it does**: Implements the source-generated MessagePack serializer path and uses it to validate the real `SerializerV2` API before Artery envelopes depend on it. **Completion criteria**: -- TlsStreamProvider wraps SslStream, handshake in ConnectAsync -- Server-side TLS handshake in TcpIncomingConnection with timeout -- All existing DotNetty TLS HOCON keys parse into TlsSettings -- TlsSetup programmatic config works and overrides HOCON -- TLS integration tests pass (self-signed certs, mutual TLS, validation) -- `dotnet test -c Release --framework net10.0` passes -**After completion**: Review with human. Archive via `openspec archive akka-io-tls-support`. +- `Akka.Serialization.V2` package exists. +- Concrete sealed `AkkaWriter` and `AkkaReader` wrap MessagePack directly. +- Source generator emits serializers for `[AkkaSerializable]` / `[AkkaField]` types. +- Generated serializers round-trip through `Serialization.cs`. +- Generated serializers send over classic remoting. +- Generated serializers persist and recover events. +- Generated serializers save and load snapshots. +- V1 and generated V2 serializers coexist. +- `SizeHint`, unknown-size behavior, manifest handling, bytes-written/result semantics, and oversized payload behavior are validated. + +**After completion**: Review with human. Archive via `openspec archive messagepack-sourcegen-validation`. --- -### Milestone 4: `streams-tcp-transport` (Spec 3) +### Milestone 4: `artery-tcp-remoting` + +**Branch**: `feature/spec4-artery-tcp-remoting` +**OpenSpec change**: `openspec/changes/artery-tcp-remoting/` +**Tasks file**: `openspec/changes/artery-tcp-remoting/tasks.md` + +**What it does**: Adds a new Artery-style TCP remoting stack beside classic remoting, using the validated `SerializerV2` payload contract. -**Branch**: `feature/spec3-streams-transport` (off Milestone 3's merged branch) -**OpenSpec change**: `openspec/changes/streams-tcp-transport/` -**Tasks file**: `openspec/changes/streams-tcp-transport/tasks.md` +**Implementation strategy**: -**What it does**: Replace DotNetty with Akka.Streams TCP transport. FrameBufferWriter for integrated framing + serialization. Binary PDU codec. Delete DotNetty entirely. +- Do not route the hot path through classic `EndpointWriter` / `AkkaProtocolTransport` / `AkkaPduCodec`. +- Start with plaintext TCP and one ordinary stream. +- Add control stream and reliable system-message delivery before lanes and compression. +- Preserve classic remoting as a compatibility path. +- Defer QUIC to Akka.NET 1.7. **Completion criteria**: -- StreamsTcpTransport implements Transport abstraction -- FrameBufferWriter enables single-buffer frame construction -- BinaryPduCodec replaces Protobuf AkkaPduCodec -- All `akka.remote.dot-netty.tcp.*` HOCON config works unchanged -- DotNetty directory and NuGet deps deleted -- Two ActorSystems communicate via new transport -- Cluster formation works -- All non-DotNetty-specific Akka.Remote specs pass -- `dotnet test -c Release --framework net10.0` passes - -**After completion**: Review with human. Archive via `openspec archive streams-tcp-transport`. + +- `ArteryRemoting : RemoteTransport` exists and is selected by configuration. +- Artery TCP listener and outbound association setup work. +- TCP framing uses `AKKA` magic + 1-byte stream id connection header and 4-byte little-endian frame length. +- Handshake includes address and UID and is tied to association incarnation state. +- Basic remote actor messaging works over Artery TCP. +- Control stream exists for handshake, liveness, quarantine, and system-message ACK/NACK traffic. +- Reliable system-message delivery preserves remoting correctness. +- Quarantine semantics are UID-scoped and compatible with existing remote lifecycle expectations. +- Classic remoting still works independently. + +**After completion**: Review with human. Archive via `openspec archive artery-tcp-remoting`. --- -### Milestone 5: `transport-performance` (Spec 5) +### Milestone 5: `transport-performance` -**Branch**: `feature/spec5-performance` (off Milestone 4's merged branch) +**Branch**: `feature/spec5-artery-performance` **OpenSpec change**: `openspec/changes/transport-performance/` **Tasks file**: `openspec/changes/transport-performance/tasks.md` -**What it does**: Run RemotePingPong, implement flush batching and optimizations, exceed DotNetty baseline. +**What it does**: Measures and tunes the Artery TCP stack against the DotNetty baseline. **Completion criteria**: -- RemotePingPong on new transport exceeds 680K msgs/sec peak -- Flush batching implemented and tuned -- Pipe thresholds benchmarked and configured -- Results documented + +- RemotePingPong on Artery TCP exceeds the 680K msgs/sec DotNetty baseline. +- Artery envelope codec materially outperforms the classic protobuf PDU path in allocation and throughput microbenchmarks. +- Generated MessagePack serializers show lower allocation and better throughput than V1 adapter fallback. +- Bounded queue and backpressure tests prove memory cannot grow unbounded under slow receivers. +- Latency impact of batching is measured and configurable. +- Results are documented. **After completion**: Review with human. Archive via `openspec archive transport-performance`. +--- + +### Milestone 6: `akka-io-tls-support` + +**Branch**: `feature/spec6-akka-io-tls` +**OpenSpec change**: `openspec/changes/akka-io-tls-support/` +**Tasks file**: `openspec/changes/akka-io-tls-support/tasks.md` + +**What it does**: Adds TLS support for the modernized TCP infrastructure and eventually Artery TCP. + +**Sequencing note**: TLS is important, but plaintext Artery TCP should prove the envelope, association, system-message, and performance model first. TLS can then wrap the same connection model. + ## Orchestration -Each milestone is executed by an OpenProse orchestrator that: +Each milestone is executed by an OpenSpec-oriented orchestrator that: -1. Reads the OpenSpec `tasks.md` for the current milestone -2. Creates a new branch for the milestone -3. Delegates tasks to worker agents (Sonnet gophers for mechanical changes, Opus for design decisions) -4. After each task group: attempts `dotnet build`, captures errors, iterates -5. After all tasks: runs `dotnet test -c Release --framework net10.0`, fixes failures -6. Commits progress incrementally -7. Reports completion status to human for review +1. Reads the OpenSpec `tasks.md` for the current milestone. +2. Creates a new branch for the milestone. +3. Delegates mechanical call-site work to focused workers when useful. +4. Runs focused builds and tests after each task group. +5. Captures compile fallout instead of hiding it behind compatibility shims. +6. Runs the milestone completion test suite. +7. Reports completion status to human for review. -Only one milestone is executed per orchestrator run. Human reviews before proceeding to the next. +Only one milestone should be executed per implementation run. Human review happens before proceeding to the next milestone. diff --git a/openspec/changes/akka-io-tls-support/design.md b/openspec/changes/akka-io-tls-support/design.md index ac3dbcf640a..5dce51bc0f2 100644 --- a/openspec/changes/akka-io-tls-support/design.md +++ b/openspec/changes/akka-io-tls-support/design.md @@ -1,6 +1,6 @@ ## Context -Spec 1 (`modernize-akka-io-tcp`) introduces `IStreamProvider` — an abstraction that returns a connected `Stream` from connection parameters. `TcpStreamProvider` returns a plain `NetworkStream`. The TCP connection actor reads/writes the `Stream` via `System.IO.Pipelines.Pipe` and never interacts with the socket directly. This design makes TLS a provider swap: `TlsStreamProvider` wraps the `NetworkStream` in `SslStream` and completes the TLS handshake inside `ConnectAsync`, returning an authenticated `Stream`. +`modernize-akka-io-tcp` introduces `IStreamProvider`, an abstraction that returns a connected `Stream` from connection parameters. `TcpStreamProvider` returns a plain `NetworkStream`. The TCP connection actor reads/writes the `Stream` via `System.IO.Pipelines.Pipe` and never interacts with the socket directly. This design makes TLS a provider swap: `TlsStreamProvider` wraps the `NetworkStream` in `SslStream` and completes the TLS handshake inside `ConnectAsync`, returning an authenticated `Stream`. The current DotNetty transport supports TLS via `DotNetty.Handlers.Tls.TlsHandler` which itself wraps `SslStream`. The HOCON configuration covers: file-based certificates (`ssl.certificate.path` + `password`), Windows certificate store lookup (`ssl.certificate.thumbprint` + `store-name` + `store-location`), mutual authentication, hostname validation, and validation suppression for development. All of these must continue to work unchanged. @@ -46,7 +46,7 @@ Reference implementation: TurboMQTT `TlsStreamProvider.cs` (~60 lines) and `Fake **Decision:** Parse `akka.remote.dot-netty.tcp.ssl.*` keys into a new `TlsSettings` class that produces `SslClientAuthenticationOptions` and `SslServerAuthenticationOptions`. -**Rationale:** Zero user configuration changes. The HOCON key names stay the same even though the underlying transport changes from DotNetty to Akka.Streams. Users migrating to Akka.NET 1.6 don't need to rewrite their TLS config. +**Rationale:** Zero user configuration changes. The HOCON key names stay the same even though the underlying transport moves away from DotNetty-specific TLS machinery. Users migrating to Akka.NET 1.6 don't need to rewrite their TLS config. Config mapping: - `ssl.certificate.path` + `ssl.certificate.password` → `new X509Certificate2(path, password)` diff --git a/openspec/changes/akka-io-tls-support/proposal.md b/openspec/changes/akka-io-tls-support/proposal.md index 06df2c548f8..f407fe0db71 100644 --- a/openspec/changes/akka-io-tls-support/proposal.md +++ b/openspec/changes/akka-io-tls-support/proposal.md @@ -1,6 +1,6 @@ ## Why -Akka.Remote requires TLS support for production deployments. The current implementation relies on DotNetty's `TlsHandler`, which is being removed (Spec 3). With Spec 1 (`modernize-akka-io-tcp`) introducing `IStreamProvider` and replacing `SocketAsyncEventArgs` with `Stream` + `Pipe`, TLS becomes a simple `IStreamProvider` implementation — `TlsStreamProvider` wraps `SslStream` around `NetworkStream`, handshake happens inside `ConnectAsync`, and the TCP connection actor never knows it's encrypted. This was previously impossible because SAEA couldn't work with `SslStream`. +Akka.Remote requires TLS support for production deployments. The current implementation relies on DotNetty's `TlsHandler`, which cannot be reused by the modern Akka.IO / future Artery TCP path. With `modernize-akka-io-tcp` introducing `IStreamProvider` and replacing `SocketAsyncEventArgs` with `Stream` + `Pipe`, TLS becomes a simple provider implementation: `TlsStreamProvider` wraps `SslStream` around `NetworkStream`, handshake happens inside `ConnectAsync`, and the TCP connection actor never knows it is encrypted. This was previously impossible because SAEA could not work with `SslStream`. ## What Changes diff --git a/openspec/changes/artery-tcp-remoting/design.md b/openspec/changes/artery-tcp-remoting/design.md new file mode 100644 index 00000000000..8665d7456ea --- /dev/null +++ b/openspec/changes/artery-tcp-remoting/design.md @@ -0,0 +1,91 @@ +## Context + +Scala Akka 2.6 Artery is more than a wire format. It includes association lifecycle, UID-scoped handshake state, control/user/large streams, reliable system-message delivery, bounded queues, compression tables, and lane partitioning. + +Akka.NET classic remoting currently uses: + +- `EndpointWriter` / `EndpointReader` +- `AkkaProtocolTransport` +- `AkkaPduCodec` +- protobuf `Payload` / `AckAndEnvelopeContainer` +- `AssociationHandle.Write(ByteString)` + +That path should remain for compatibility, but it should not be the high-throughput Artery path. + +## Goals / Non-Goals + +**Goals:** + +- Add new Artery-style remoting beside classic remoting. +- Use `SerializerV2` for payloads. +- Start with plaintext TCP. +- Implement handshake, UID tracking, association state, ordinary stream, control stream, and reliable system-message delivery. +- Preserve classic remoting independently. +- Build a protocol that can later host QUIC. + +**Non-Goals:** + +- Classic wire compatibility for Artery. +- Removing classic remoting. +- Implementing QUIC in 1.6. +- Making Akka.Streams TCP the Artery MVP hot path. +- Implementing compression before basic association and system-message correctness. + +## Decisions + +### 1. New RemoteTransport Implementation + +Introduce `ArteryRemoting : RemoteTransport` rather than forcing Artery through `AkkaProtocolTransport`. + +Rationale: Artery needs association state, stream separation, reliable system-message delivery, bounded queues, and compression table lifecycle that do not fit the classic `AssociationHandle.Write(ByteString)` abstraction. + +### 2. Direct TCP/Pipelines Hot Path + +Use dedicated TCP/socket or `System.IO.Pipelines` loops for the Artery MVP instead of materialized Akka.Streams TCP. + +Rationale: current Akka.Streams TCP still pays actor/materializer overhead and Akka.IO copies pipe reads before delivery. Artery's transport hot path should own framing, queueing, batching, and backpressure directly. + +### 3. Scala Artery TCP Framing As Baseline + +Use the Scala Akka 2.6 TCP framing idea: + +- connection header: magic bytes `AKKA` plus 1-byte stream ID, +- frame header: 4-byte little-endian frame length. + +Rationale: this is simple, proven, and separates control/ordinary/large streams cleanly. + +### 4. Artery Envelope Separate From SerializerV2 + +`SerializerV2` serializes payloads. The Artery envelope owns remoting metadata: protocol version, flags, origin UID, serializer ID, manifest, sender, recipient, control/system markers, and payload boundaries. + +Rationale: the serializer contract and remoting protocol evolve separately. + +### 5. Control Stream Before Lanes + +Implement control stream and reliable system-message delivery before ordinary message lanes. + +Rationale: correctness before throughput. System messages must not be starved by user traffic. + +### 6. UID-Scoped State + +Handshake, quarantine, compression tables, and reliable system-message state are scoped by remote address + UID/incarnation. + +Rationale: stale state after remote ActorSystem restart can corrupt actor refs, manifests, and system-message delivery. + +### 7. Bounded Queues + +Outbound queues must be bounded. User traffic overflow should be deterministic; control/system overflow should trigger quarantine semantics. + +Rationale: classic remoting's unbounded endpoint buffers are not acceptable for the new transport path. + +## Risks / Trade-offs + +**Ordering**: lanes can violate actor ordering if partitioning is wrong. Start with one ordinary stream, then add lane tests before enabling multiple lanes. + +**Remote lifecycle compatibility**: quarantine, DeathWatch, and remote deployment expectations are subtle. Keep tests close to classic behavior. + +**System-message reliability**: ACK/NACK and resend logic must be correct before performance tuning. + +**Protocol churn**: sourcegen must validate V2 before envelope format is locked down. + +**Backpressure behavior**: bounded queues may expose problems that classic remoting used to hide by buffering. diff --git a/openspec/changes/artery-tcp-remoting/proposal.md b/openspec/changes/artery-tcp-remoting/proposal.md new file mode 100644 index 00000000000..1afa4113c18 --- /dev/null +++ b/openspec/changes/artery-tcp-remoting/proposal.md @@ -0,0 +1,39 @@ +## Why + +The previous plan to replace DotNetty with an Akka.Streams TCP transport under classic `AkkaProtocolTransport` does not address the real high-throughput remoting problem. Classic remoting is built around `EndpointWriter`, `EndpointReader`, `AkkaPduCodec`, protobuf envelopes, `ByteString`, a single logical stream, and actor-mailbox buffering. Retrofitting that path still leaves the core architecture allocation-heavy and unable to model Artery features cleanly. + +Akka.NET 1.6 should add an Artery-style TCP remoting stack beside classic remoting. Classic remoting remains the compatibility path. Artery becomes the high-throughput path using the validated `SerializerV2` payload contract. + +## What Changes + +- Add `ArteryRemoting : RemoteTransport`, selected by configuration. +- Add Artery TCP connection framing: `AKKA` magic + stream id connection header, then 4-byte little-endian frame lengths. +- Add binary Artery envelope codec using `SerializerV2` payload serialization. +- Add association registry and UID-scoped association state. +- Add handshake with address and UID. +- Add control stream for handshake, liveness, quarantine, and system-message ACK/NACK. +- Add ordinary user message stream initially. +- Add reliable system-message delivery independent from user traffic. +- Keep classic remoting intact. + +### What Does Not Change + +- Classic remoting remains available and wire-compatible with its existing classic protocol. +- Artery does not need to be wire-compatible with classic remoting. +- QUIC is deferred to Akka.NET 1.7. +- TLS is deferred until plaintext Artery TCP validates the protocol, association, and performance model. +- Akka.Streams TCP remains a user-facing API but is not the Artery MVP hot path. + +## Capabilities + +### New Capabilities + +- `artery-tcp-remoting`: Artery-style TCP remoting path with V2 payload serialization, association state, control stream, reliable system messages, and eventual lane/compression support. + +## Impact + +- **Akka.Remote**: new Artery namespace/components beside classic remoting. +- **Configuration**: new config switch selects classic or Artery remoting. +- **Serialization**: uses the validated `SerializerV2` payload contract. +- **Benchmarks**: RemotePingPong and envelope microbenchmarks target Artery TCP. +- **Compatibility**: classic remoting remains the compatibility path; Artery is a new protocol path. diff --git a/openspec/changes/artery-tcp-remoting/specs/artery-tcp-remoting/spec.md b/openspec/changes/artery-tcp-remoting/specs/artery-tcp-remoting/spec.md new file mode 100644 index 00000000000..63955aa42d0 --- /dev/null +++ b/openspec/changes/artery-tcp-remoting/specs/artery-tcp-remoting/spec.md @@ -0,0 +1,109 @@ +## ADDED Requirements + +### Requirement: Artery remoting is separate from classic remoting + +The system SHALL provide an Artery-style remoting implementation beside the existing classic remoting implementation. + +#### Scenario: Artery selected by configuration +- **WHEN** Artery remoting is enabled in configuration +- **THEN** the remote actor ref provider SHALL use `ArteryRemoting` + +#### Scenario: Classic remoting preserved +- **WHEN** Artery remoting is disabled +- **THEN** classic remoting SHALL continue to operate independently + +### Requirement: Artery TCP framing + +Artery TCP SHALL use a connection header and length-prefixed frames. + +#### Scenario: Connection header +- **WHEN** a TCP stream is opened +- **THEN** the sender SHALL write magic bytes `AKKA` followed by a 1-byte stream ID + +#### Scenario: Frame header +- **WHEN** an envelope is written +- **THEN** it SHALL be preceded by a 4-byte little-endian frame length + +#### Scenario: Oversized frame rejected +- **WHEN** a frame length exceeds the configured maximum frame size +- **THEN** the connection SHALL be closed with an error + +### Requirement: Artery envelope uses SerializerV2 payloads + +Artery envelopes SHALL carry remoting metadata separately from the `SerializerV2` payload bytes. + +#### Scenario: Envelope encode +- **WHEN** a user message is sent +- **THEN** the envelope SHALL encode origin UID, serializer ID, manifest, recipient, sender, and payload bytes + +#### Scenario: Envelope decode before payload deserialization +- **WHEN** an envelope is received +- **THEN** remoting metadata SHALL be decoded before payload deserialization + +### Requirement: Handshake establishes UID-scoped association + +Artery remoting SHALL establish remote address and UID before accepting ordinary messages. + +#### Scenario: Handshake request +- **WHEN** an outbound association starts +- **THEN** it SHALL send a handshake request containing the local address and UID + +#### Scenario: Handshake response +- **WHEN** an inbound node accepts a handshake +- **THEN** it SHALL reply with its local address and UID + +#### Scenario: Wrong target rejected +- **WHEN** a handshake is addressed to the wrong local address +- **THEN** it SHALL be rejected or dropped with a warning + +#### Scenario: UID change resets association state +- **WHEN** the same remote address appears with a new UID +- **THEN** UID-scoped state SHALL be reset + +### Requirement: Control stream separated from ordinary traffic + +Control traffic SHALL use a control stream that cannot be starved by ordinary user messages. + +#### Scenario: Control messages routed separately +- **WHEN** handshake, liveness, quarantine, or system-message ACK/NACK messages are sent +- **THEN** they SHALL use the control stream + +#### Scenario: Ordinary messages use ordinary stream +- **WHEN** user actor messages are sent +- **THEN** they SHALL use the ordinary stream unless later large-message routing applies + +### Requirement: Reliable system-message delivery + +System messages SHALL be delivered with explicit ACK/NACK and resend semantics independent from ordinary user traffic. + +#### Scenario: System message sequenced +- **WHEN** a system message is sent +- **THEN** it SHALL be assigned a monotonically increasing sequence number for the association incarnation + +#### Scenario: System message acknowledged +- **WHEN** a receiver processes a system message in order +- **THEN** it SHALL acknowledge the sequence number over the control stream + +#### Scenario: Missing system message nacked +- **WHEN** a receiver observes a gap in system-message sequence numbers +- **THEN** it SHALL send a negative acknowledgement for the highest processed sequence number + +#### Scenario: System message resend +- **WHEN** a system message remains unacknowledged past the resend interval +- **THEN** it SHALL be resent until acknowledged or the give-up policy triggers + +#### Scenario: System message overflow quarantines +- **WHEN** the bounded system-message buffer overflows +- **THEN** the association SHALL be quarantined + +### Requirement: Outbound queues are bounded + +Artery remoting SHALL use bounded outbound queues for user and control traffic. + +#### Scenario: Ordinary queue overflow deterministic +- **WHEN** the ordinary outbound queue is full +- **THEN** the message SHALL be dropped or failed according to documented policy without unbounded memory growth + +#### Scenario: Control queue overflow severe +- **WHEN** the control/system outbound queue is full +- **THEN** the association SHALL fail or quarantine according to documented policy diff --git a/openspec/changes/artery-tcp-remoting/tasks.md b/openspec/changes/artery-tcp-remoting/tasks.md new file mode 100644 index 00000000000..9fb4a8aa0e9 --- /dev/null +++ b/openspec/changes/artery-tcp-remoting/tasks.md @@ -0,0 +1,95 @@ +## 1. Configuration And Entry Point + +- [ ] 1.1 Add Artery remoting configuration section +- [ ] 1.2 Add config switch for classic vs Artery remoting +- [ ] 1.3 Introduce `ArteryRemoting : RemoteTransport` +- [ ] 1.4 Wire `RemoteActorRefProvider` to select remoting implementation from config +- [ ] 1.5 Preserve classic remoting default until Artery is explicitly enabled + +## 2. TCP Framing Foundation + +- [ ] 2.1 Define stream IDs for control, ordinary, and future large-message streams +- [ ] 2.2 Implement connection header: `AKKA` magic + 1-byte stream ID +- [ ] 2.3 Implement 4-byte little-endian frame length encoder +- [ ] 2.4 Implement frame parser over `ReadOnlySequence` +- [ ] 2.5 Enforce maximum frame size +- [ ] 2.6 Add framing tests for complete, partial, multiple, and oversized frames + +## 3. Artery Envelope Codec + +- [ ] 3.1 Define MVP envelope header fields +- [ ] 3.2 Encode protocol version and flags +- [ ] 3.3 Encode origin UID +- [ ] 3.4 Encode serializer ID and manifest literal +- [ ] 3.5 Encode recipient actor ref literal or no-recipient marker +- [ ] 3.6 Encode sender actor ref literal or no-sender marker +- [ ] 3.7 Encode payload using `SerializerV2` +- [ ] 3.8 Decode envelope metadata without deserializing payload first +- [ ] 3.9 Add envelope codec tests, including multi-segment input + +## 4. Association State And Handshake + +- [ ] 4.1 Add association registry keyed by remote address +- [ ] 4.2 Track remote UID and association incarnation +- [ ] 4.3 Implement outbound handshake request +- [ ] 4.4 Implement inbound handshake response +- [ ] 4.5 Reject wrong target address during handshake +- [ ] 4.6 Reset UID-scoped state on remote UID change +- [ ] 4.7 Add handshake timeout and retry behavior +- [ ] 4.8 Add quarantine state keyed by UID + +## 5. Plaintext TCP Transport + +- [ ] 5.1 Implement TCP listener for inbound Artery connections +- [ ] 5.2 Implement outbound TCP connection creation +- [ ] 5.3 Attach inbound connection to stream by stream ID +- [ ] 5.4 Add ordinary stream send path +- [ ] 5.5 Add ordinary stream receive path +- [ ] 5.6 Dispatch decoded messages to recipients +- [ ] 5.7 Add basic two-ActorSystem remoting test over Artery TCP + +## 6. Control Stream + +- [ ] 6.1 Add outbound control queue +- [ ] 6.2 Add inbound control stream processing +- [ ] 6.3 Route handshake messages over control stream +- [ ] 6.4 Add liveness/heartbeat control messages +- [ ] 6.5 Add quarantine control message +- [ ] 6.6 Ensure control stream cannot be starved by ordinary messages + +## 7. Reliable System Messages + +- [ ] 7.1 Define system-message envelope +- [ ] 7.2 Add sequence numbers for outbound system messages +- [ ] 7.3 Add ACK/NACK control messages +- [ ] 7.4 Add resend timer +- [ ] 7.5 Add bounded system-message buffer +- [ ] 7.6 Quarantine on system-message buffer overflow +- [ ] 7.7 Quarantine or fail association after give-up timeout +- [ ] 7.8 Add duplicate and out-of-order system-message tests +- [ ] 7.9 Verify DeathWatch and remote deployment system-message behavior + +## 8. Bounded Queues And Backpressure + +- [ ] 8.1 Add bounded ordinary outbound queue +- [ ] 8.2 Add bounded control outbound queue +- [ ] 8.3 Define overflow behavior for user messages +- [ ] 8.4 Define overflow behavior for control/system messages +- [ ] 8.5 Add slow receiver tests proving queues do not grow unbounded + +## 9. Lifecycle And Compatibility Tests + +- [ ] 9.1 Verify classic remoting still works when Artery is disabled +- [ ] 9.2 Verify Artery remoting starts and stops cleanly +- [ ] 9.3 Verify remote association restart with new UID resets state +- [ ] 9.4 Verify quarantine events are published correctly +- [ ] 9.5 Verify cluster formation with Artery TCP if cluster scope is included in MVP + +## 10. Deferred Follow-Ups + +- [ ] 10.1 Add ordinary message lanes after ordering tests are designed +- [ ] 10.2 Add large-message stream +- [ ] 10.3 Add actor ref compression +- [ ] 10.4 Add manifest compression +- [ ] 10.5 Add TLS wrapping +- [ ] 10.6 Prepare QUIC transport adapter for 1.7 diff --git a/openspec/changes/messagepack-sourcegen-validation/design.md b/openspec/changes/messagepack-sourcegen-validation/design.md new file mode 100644 index 00000000000..45a1e12074e --- /dev/null +++ b/openspec/changes/messagepack-sourcegen-validation/design.md @@ -0,0 +1,70 @@ +## Context + +The POC at `Aaronontheweb/AkkaSerializationPoC` validated the preferred direction for generated serialization: + +- MessagePack is the default codec. +- `AkkaWriter` and `AkkaReader` are concrete sealed wrappers over MessagePack. +- There is no generalized codec abstraction layer. +- Source generation provides compile-time validation and avoids reflection. + +The source generator should not be developed against hypothetical serialization APIs. It should run after `serializer-v2` makes V2 canonical and after classic remoting and persistence are bridged. That lets generated serializers validate the exact API Artery will consume. + +## Goals / Non-Goals + +**Goals:** + +- Implement user-facing source-generated MessagePack serialization on top of `SerializerV2`. +- Validate generated serializers through `Serialization`, classic remoting, events, and snapshots. +- Confirm V2 API details before Artery envelopes are built. +- Support AOT-oriented, reflection-free serializer code. +- Preserve V1/V2 coexistence. + +**Non-Goals:** + +- Replacing all built-in protobuf serializers. +- Adding MessagePack dependency to core Akka. +- Implementing Artery envelopes. +- Removing V1 serializer support. + +## Decisions + +### 1. MessagePack Package Outside Core Akka + +`Akka.Serialization.V2` owns MessagePack dependencies, attributes, writer/reader helpers, and source generator integration. + +Core Akka owns only `SerializerV2` and compatibility infrastructure. + +### 2. Sealed Writer / Reader + +Use sealed `AkkaWriter` and `AkkaReader` classes rather than codec interfaces. + +Rationale: the POC showed this improves JIT devirtualization and keeps the API simpler. + +### 3. Sourcegen Validates V2 API Before Artery + +Generated serializers must prove: + +- bytes-written/result reporting works, +- unknown-size fallback works, +- manifests work, +- V1 adapter coexistence works, +- persistence can store and recover generated payloads, +- classic remoting can send generated payloads. + +### 4. Version-Tolerant Schema + +Fields are explicitly indexed using `[AkkaField(index)]`. Generated code should support unknown trailing fields where possible. + +### 5. Cross-Assembly Composition Is Required + +Real Akka.NET applications use multiple assemblies. The generator must support serializable messages and generated serializers across assembly boundaries. + +## Risks / Trade-offs + +**Generator complexity**: keep diagnostics focused and add incrementally. + +**MessagePack conventions**: document DateTime, Guid, decimal, nullable, collection, and nested object conventions. + +**API churn**: if sourcegen finds V2 API problems, fix V2 before Artery starts. + +**Persistence compatibility**: generated serializers must not compromise stored payload readability. diff --git a/openspec/changes/messagepack-sourcegen-validation/proposal.md b/openspec/changes/messagepack-sourcegen-validation/proposal.md new file mode 100644 index 00000000000..d0d77cf8b17 --- /dev/null +++ b/openspec/changes/messagepack-sourcegen-validation/proposal.md @@ -0,0 +1,34 @@ +## Why + +`SerializerV2` must be proven before Artery envelopes depend on it. The best proof is the source-generated MessagePack serializer path from the POC: it exercises the real API with non-trivial payloads, manifests, size hints, generated code, AOT-oriented patterns, and integration through remoting and persistence. + +This change runs after the atomic `serializer-v2` foundation. It is not a late optimization. It is the validation gate for the V2 API shape. + +## What Changes + +- Add `Akka.Serialization.V2` package for user-facing generated serializers. +- Add sealed `AkkaWriter` and `AkkaReader` wrappers over MessagePack-CSharp. +- Add attributes such as `[AkkaSerializable]`, `[AkkaField]`, and serializer configuration attributes. +- Add a Roslyn incremental source generator that emits V2 serializers. +- Validate generated serializers through `Serialization.cs`, classic remoting, persisted events, and persisted snapshots. +- Validate V2 API details needed by Artery: manifests, size hints, unknown-size fallback, bytes-written/result semantics, oversized payload handling, and V1 coexistence. + +### What Does Not Change + +- Core Akka does not take a MessagePack dependency. +- Artery envelopes are not implemented here. +- Classic remoting and persistence bridge work belongs to the preceding `serializer-v2` foundation. +- V1 serializers remain supported through `SerializerV1Adapter`. + +## Capabilities + +### New Capabilities + +- `messagepack-sourcegen-validation`: Source-generated MessagePack serializers for Akka.NET 1.6 and the integration tests proving `SerializerV2` is ready for Artery. + +## Impact + +- **New package**: `src/core/Akka.Serialization.V2/` +- **Tests**: generated serializer tests, remoting tests, persistence event tests, persistence snapshot tests, cross-assembly generator tests. +- **Build**: source generator package/project added to solution and pack targets. +- **Documentation**: update migration and usage docs for generated serializers. diff --git a/openspec/changes/messagepack-sourcegen-validation/specs/messagepack-sourcegen/spec.md b/openspec/changes/messagepack-sourcegen-validation/specs/messagepack-sourcegen/spec.md new file mode 100644 index 00000000000..f03c25bed44 --- /dev/null +++ b/openspec/changes/messagepack-sourcegen-validation/specs/messagepack-sourcegen/spec.md @@ -0,0 +1,49 @@ +## ADDED Requirements + +### Requirement: MessagePack writer and reader package + +The system SHALL provide an `Akka.Serialization.V2` package containing sealed `AkkaWriter` and `AkkaReader` types backed by MessagePack-CSharp. + +#### Scenario: Writer uses V2 buffer API +- **WHEN** a generated serializer writes a message +- **THEN** it SHALL write MessagePack bytes through `AkkaWriter` into the provided `IBufferWriter` + +#### Scenario: Reader uses V2 sequence API +- **WHEN** a generated serializer reads a message +- **THEN** it SHALL read MessagePack bytes through `AkkaReader` from the provided V2 input + +### Requirement: Source generator emits V2 serializers + +The system SHALL provide a Roslyn incremental source generator that emits `SerializerV2` implementations for annotated messages. + +#### Scenario: Serializable type annotated +- **WHEN** a type is annotated with `[AkkaSerializable]` and valid `[AkkaField]` members +- **THEN** the generator SHALL emit serializer code for that type + +#### Scenario: Invalid schema rejected +- **WHEN** field indexes are missing, duplicated, or unsupported +- **THEN** the generator SHALL produce compile-time diagnostics + +### Requirement: Generated serializers validate SerializerV2 API + +Generated serializers SHALL validate `SerializerV2` through real Akka.NET integration points before Artery envelopes are implemented. + +#### Scenario: Serialization round-trip +- **WHEN** a generated serializer is registered +- **THEN** the message SHALL round-trip through `Serialization.cs` + +#### Scenario: Classic remoting round-trip +- **WHEN** a generated-serializer message is sent over classic Akka.Remote +- **THEN** the receiver SHALL deserialize the original message + +#### Scenario: Persistence event round-trip +- **WHEN** a generated-serializer event is persisted +- **THEN** it SHALL recover as the original event + +#### Scenario: Persistence snapshot round-trip +- **WHEN** a generated-serializer snapshot is saved +- **THEN** it SHALL load as the original snapshot + +#### Scenario: V1 coexistence +- **WHEN** V1 and generated V2 serializers are registered in the same actor system +- **THEN** both SHALL work through the V2 serialization infrastructure diff --git a/openspec/changes/messagepack-sourcegen-validation/tasks.md b/openspec/changes/messagepack-sourcegen-validation/tasks.md new file mode 100644 index 00000000000..39a825da330 --- /dev/null +++ b/openspec/changes/messagepack-sourcegen-validation/tasks.md @@ -0,0 +1,69 @@ +## 1. Package Setup + +- [ ] 1.1 Create `src/core/Akka.Serialization.V2/` project +- [ ] 1.2 Add MessagePack dependency to the new package only +- [ ] 1.3 Add project to solution +- [ ] 1.4 Add test project for generated serialization +- [ ] 1.5 Configure pack/build metadata + +## 2. AkkaWriter / AkkaReader + +- [ ] 2.1 Port sealed `AkkaWriter` from POC direction +- [ ] 2.2 Port sealed `AkkaReader` from POC direction +- [ ] 2.3 Implement primitive read/write methods +- [ ] 2.4 Implement DateTime, DateTimeOffset, Guid, decimal conventions +- [ ] 2.5 Implement nullable handling +- [ ] 2.6 Implement object/field framing helpers +- [ ] 2.7 Implement unknown-field skip support +- [ ] 2.8 Add round-trip tests for supported built-in types + +## 3. MessagePack Serializer Base + +- [ ] 3.1 Add `MessagePackSerializer : SerializerV2` +- [ ] 3.2 Add generic protocol-scoped serializer base if needed by generator design +- [ ] 3.3 Bridge V2 buffer API to `AkkaWriter` / `AkkaReader` +- [ ] 3.4 Validate bytes-written/result behavior +- [ ] 3.5 Validate unknown-size fallback behavior +- [ ] 3.6 Validate manifest behavior + +## 4. Attributes And Diagnostics + +- [ ] 4.1 Add `[AkkaSerializable]` +- [ ] 4.2 Add `[AkkaField(index)]` +- [ ] 4.3 Add serializer marker/configuration attributes +- [ ] 4.4 Add diagnostics for missing field indexes +- [ ] 4.5 Add diagnostics for duplicate field indexes +- [ ] 4.6 Add diagnostics for unsupported member types +- [ ] 4.7 Add diagnostics for invalid constructors or inaccessible members + +## 5. Source Generator + +- [ ] 5.1 Implement Roslyn incremental source generator +- [ ] 5.2 Generate serializer class for annotated messages +- [ ] 5.3 Generate manifest dispatch +- [ ] 5.4 Generate write methods +- [ ] 5.5 Generate read methods +- [ ] 5.6 Support nested generated types +- [ ] 5.7 Support collection types selected for 1.6 MVP +- [ ] 5.8 Support cross-assembly generated serializers + +## 6. Integration Validation + +- [ ] 6.1 Register generated serializer through HOCON +- [ ] 6.2 Register generated serializer through programmatic setup if supported +- [ ] 6.3 Round-trip generated payload through `Serialization.cs` +- [ ] 6.4 Send generated payload over classic remoting +- [ ] 6.5 Persist and recover generated event payload +- [ ] 6.6 Save and load generated snapshot payload +- [ ] 6.7 Verify V1 and generated V2 serializers coexist +- [ ] 6.8 Verify oversized payload behavior is deterministic + +## 7. Documentation And Validation + +- [ ] 7.1 Document generated serializer usage +- [ ] 7.2 Document supported types and versioning rules +- [ ] 7.3 Document migration from V1 serializers +- [ ] 7.4 Run focused generated serialization tests +- [ ] 7.5 Run focused Akka.Remote tests using generated serializers +- [ ] 7.6 Run focused Akka.Persistence tests using generated serializers +- [ ] 7.7 Record any V2 API changes required before Artery starts diff --git a/openspec/changes/modernize-akka-io-tcp/design.md b/openspec/changes/modernize-akka-io-tcp/design.md index 21d9727b98f..0063ee0dfd9 100644 --- a/openspec/changes/modernize-akka-io-tcp/design.md +++ b/openspec/changes/modernize-akka-io-tcp/design.md @@ -22,10 +22,10 @@ Current target frameworks are `netstandard2.0` + `net6.0`. The `netstandard2.0` - Preserve the actor supervision hierarchy exactly **Non-Goals:** -- TLS implementation (Spec 2 — just needs `TlsStreamProvider` after this lands) -- Replacing the DotNetty transport (Spec 3 — depends on this) -- SerializerV2 integration (Spec 4 — parallel workstream) -- Performance benchmarking (Spec 5 — depends on Specs 3+4) +- TLS implementation (separate `TlsStreamProvider` work) +- Replacing or bypassing DotNetty remoting (future Artery TCP work depends on this direction) +- SerializerV2 integration (separate atomic foundation workstream) +- Performance benchmarking (depends on SerializerV2 and Artery TCP) - Changing Akka.IO.Udp (separate effort, similar pattern could apply later) - Source generator for serialization (deferred) - Modifying the Akka.Streams materializer architecture diff --git a/openspec/changes/modernize-akka-io-tcp/proposal.md b/openspec/changes/modernize-akka-io-tcp/proposal.md index e2be5ff27e1..18b34302f71 100644 --- a/openspec/changes/modernize-akka-io-tcp/proposal.md +++ b/openspec/changes/modernize-akka-io-tcp/proposal.md @@ -1,6 +1,6 @@ ## Why -Akka.NET's TCP I/O layer uses two legacy primitives that block the 1.6 transport and serialization goals: `ByteString` (incompatible with `System.Memory`) and `SocketAsyncEventArgs` (incompatible with `SslStream` for TLS). DotNetty's `ByteBuf` is also incompatible with `System.Memory`, making it a dead end. Modernizing the Akka.IO TCP internals to use `System.Memory` types and `Stream` + `System.IO.Pipelines` is the prerequisite for TLS support, replacing DotNetty, and integrating SerializerV2's `IBufferWriter` / `ReadOnlySequence` contract. The TurboMQTT project has already proven this pattern works for both TCP and TLS via the `IStreamProvider` abstraction. +Akka.NET's TCP I/O layer uses two legacy primitives that block the 1.6 transport and serialization goals: `ByteString` (incompatible with `System.Memory`) and `SocketAsyncEventArgs` (incompatible with `SslStream` for TLS). DotNetty's `ByteBuf` is also incompatible with `System.Memory`, making it a dead end for the future high-throughput path. Modernizing the Akka.IO TCP internals to use `System.Memory` types and `Stream` + `System.IO.Pipelines` is the prerequisite for TLS support, future Artery TCP work, and integrating SerializerV2's `IBufferWriter` / `ReadOnlySequence` contract. The TurboMQTT project has already proven this pattern works for both TCP and TLS via the `IStreamProvider` abstraction. ## What Changes @@ -40,4 +40,4 @@ Akka.NET's TCP I/O layer uses two legacy primitives that block the 1.6 transport - **Build system**: `Directory.Build.props` TFM changes (`NetStandardLibVersion`, `NetLibVersion`), possible removal of `Polyfill` package for netstandard2.0 - **NuGet dependencies**: Add `System.IO.Pipelines` for netstandard2.1 target - **Test suites**: All tests referencing `ByteString` need updating. All Akka.IO TCP tests need to pass with new internals. -- **Downstream enablement**: Spec 2 (TLS via `TlsStreamProvider`), Spec 3 (Akka.Streams transport replacing DotNetty), Spec 4 (SerializerV2), Spec 5 (Performance) +- **Downstream enablement**: TLS via `TlsStreamProvider`, the `SerializerV2` foundation, and the future Artery TCP remoting path diff --git a/openspec/changes/serializer-v2/design.md b/openspec/changes/serializer-v2/design.md index e0445bf0467..94db97a4fc4 100644 --- a/openspec/changes/serializer-v2/design.md +++ b/openspec/changes/serializer-v2/design.md @@ -1,95 +1,120 @@ ## Context -Akka.NET's serialization infrastructure centers on the `Serializer` base class (`src/core/Akka/Serialization/Serializer.cs`) with `ToBinary(object) → byte[]` and `FromBinary(byte[], Type) → object`. `SerializerWithStringManifest` extends it with manifest-based dispatch. The `Serialization` class (`src/core/Akka/Serialization/Serialization.cs`) manages registration via HOCON and `SerializationSetup`, storing serializers in `Dictionary` by ID and `ConcurrentDictionary` by type. +Akka.NET currently centers serialization around `Serializer` and `SerializerWithStringManifest` in `src/core/Akka/Serialization/Serializer.cs`. The primary API is `ToBinary(object) -> byte[]` and `FromBinary(byte[], Type/string) -> object`. This forces byte-array allocation even when the caller could write directly to a buffer or deserialize from a `ReadOnlySequence`. -All 15+ internal serializers (Cluster, Remote, Persistence, Sharding, etc.) use Google.Protobuf. They extend `SerializerWithStringManifest`, call `.ToByteArray()` for encoding, and `Parser.ParseFrom(byte[])` for decoding. Four of them (Sharding, PubSub, ReliableDelivery, Misc) wrap arbitrary user payloads via `WrappedPayloadSupport`, which calls `FindSerializerFor()` → `ToBinary()` on the inner message. +The immediate blast radius of changing this API is larger than core Akka: -A POC at github.com/Aaronontheweb/AkkaSerializationPoC (PR #42, spike/serializer-v2-redesign branch) validated the approach: sealed `AkkaWriter`/`AkkaReader` classes wrapping MessagePack, 22% faster deserialization than the interface-based dev branch, -322 net lines. The source generator is deferred — hand-written serializers validate the API first. +- Classic Akka.Remote uses `MessageSerializer` and `WrappedPayloadSupport` to place serializer ID, manifest, and payload bytes into protobuf envelopes. +- Akka.Persistence stores nested payloads in protobuf `PersistentPayload` records containing serializer ID, manifest, and payload bytes. +- Snapshot storage uses wrapper serializers directly. +- DistributedData and Akka.Delivery have direct serializer call sites. +- Tests frequently assert concrete serializer types returned from `FindSerializerForType()`. + +This means `SerializerV2` must be introduced with the classic remoting and persistence bridges in the same change. A V2-only core change would leave the repo in a partially broken state. ## Goals / Non-Goals **Goals:** -- `SerializerV2` base class in core Akka with `IBufferWriter` / `ReadOnlySequence` API -- `SerializerV1Adapter` wraps legacy serializers to V2 -- `Serialization.cs` uses V2 internally (auto-wraps V1 on registration) -- `MessagePackSerializer : SerializerV2` + sealed `AkkaWriter`/`AkkaReader` in separate package -- Mechanical port of simple internal Protobuf serializers to V2 base (same IDs, same wire format) -- Hand-written serializers validate the API before source generator investment -- Persistence data fully backward compatible (V1-serialized events remain readable) + +- Introduce `SerializerV2` as the canonical Akka.NET 1.6 serialization abstraction. +- Keep public `FindSerializerFor()` / `FindSerializerForType()` compatible while adding internal V2 lookup and buffer-first APIs. +- Keep V1 serializers working through `SerializerV1Adapter`. +- Require new native V2 serializers to emit non-empty, non-CLR manifests to avoid polymorphic deserialization, while preserving legacy manifests for existing serializer IDs. +- Preserve classic Akka.Remote wire compatibility. +- Preserve Akka.Persistence stored event and snapshot compatibility. +- Decide V2 API details needed by sourcegen and Artery before either depends on the API. +- Keep MessagePack and source generation out of core Akka. **Non-Goals:** -- Source generator (deferred until API validated) -- Rewriting internal Protobuf serializers to use MessagePack (they stay Protobuf, just change base class) -- Changing persistence envelope serializers (PersistenceMessageSerializer, PersistenceSnapshotSerializer) -- Changing the HOCON registration mechanism -- HOCON-less or attribute-only registration (future enhancement) + +- Implementing source-generated MessagePack serializers. +- Introducing Artery envelopes or Artery TCP. +- Replacing classic remoting wire format. +- Rewriting all internal protobuf serializers to MessagePack. +- Removing V1 serializer classes. ## Decisions -### 1. SerializerV2 is independent — does not extend Serializer +### 1. SerializerV2 Is Canonical With A Transitional Compatibility Surface + +`SerializerV2` is the canonical 1.6 serializer abstraction and exposes buffer-first serialization through `IBufferWriter` plus deserialization from `ReadOnlySequence`. + +For the foundation PR, `SerializerV2` remains usable through existing `Serializer` call sites so classic remoting, persistence, and public lookup APIs can stay compatible while internals move to V2. V1 compatibility is provided by `SerializerV1Adapter : SerializerV2`. + +Rationale: a hard public API break at this layer creates broad compatibility fallout before any native V2 serializers exist. The compatibility inheritance is transitional design debt; it should be revisited before native V2 serializers become widespread. + +### 2. V2 Still Provides Bridge Methods + +`SerializerV2` should expose compatibility bridge methods such as `ToBinary` and `FromBinary` so existing code paths can be migrated incrementally. + +These bridge methods should be implemented in terms of the V2 buffer API for native V2 serializers and delegated to the inner serializer for V1 adapters. + +Rationale: classic remoting and persistence still need byte arrays at protobuf boundaries. Bridge methods keep those compatibility paths clear while new internals can use buffer-first V2 APIs. -**Decision:** `SerializerV2` is a new base class with no inheritance relationship to `Serializer` or `SerializerWithStringManifest`. V1 serializers are wrapped in `SerializerV1Adapter : SerializerV2`. +### 3. Serialize Must Report Bytes Written -**Rationale:** Having V2 extend V1 permanently couples the new system to the `byte[]`-based API. The bridge methods (`ToBinary`/`FromBinary`) exist on V2 for transport compatibility but are implemented in terms of the buffer API (not inherited from V1). This allows the transport to eventually bypass the bridge entirely. +The V2 serialize API must make frame and payload length accounting explicit. -### 2. Two-layer design: core base class + MessagePack package +The exact shape can be `int`, `ValueTask`, or a small result type, but callers must not have to infer bytes written from unrelated state when building envelopes. -**Decision:** -- Layer 1: `SerializerV2` in core Akka — codec-agnostic, no MessagePack dependency -- Layer 2: `MessagePackSerializer : SerializerV2` in `Akka.Serialization.V2` — bridges to `AkkaWriter`/`AkkaReader` +Rationale: Artery envelopes and frame encoders need accurate payload length accounting. Classic remoting and persistence can ignore this in bridge paths, but the API must be suitable before sourcegen and Artery work begins. -``` -SerializerV2 (core Akka — IBufferWriter/ReadOnlySequence, no MessagePack) - ├── SerializerV1Adapter (wraps legacy Serializer) - ├── MessagePackSerializer (Akka.Serialization.V2 — AkkaWriter/AkkaReader) - │ └── MessagePackSerializer (protocol-scoped for future codegen) - └── [Internal Protobuf serializers extend SerializerV2 directly] -``` +### 4. SizeHint Needs Unknown Size -**Rationale:** Core Akka should not depend on MessagePack. Internal Protobuf serializers extend `SerializerV2` directly and use `proto.WriteTo(IBufferWriter)` — they don't need the MessagePack layer. User-facing serializers use `MessagePackSerializer` via the separate package. +`SizeHint` must support an unknown-size value. -### 3. Serialization.cs stores SerializerV2 internally +Rationale: V1 adapters and some serializers cannot cheaply know the serialized size. Forcing inaccurate guesses will cause poor buffer sizing and fragile frame accounting. -**Decision:** Change `_serializersById` to `Dictionary` and `_serializerMap` to `ConcurrentDictionary`. V1 serializers instantiated from HOCON are auto-wrapped in `SerializerV1Adapter`. `FindSerializerFor()` returns `SerializerV2`. +### 5. Manifest Is A V2 API -**Rationale:** V2 is the new foundation. All dispatch goes through V2. `SerializerV1Adapter.Inner` provides access to the underlying V1 serializer for callers that need backward compat. +Manifest production should be a direct V2 API, not repeated `is SerializerWithStringManifest` checks. -### 4. Mechanical port of internal Protobuf serializers +New native V2 serializers must return a non-empty serializer-owned manifest token. Manifests must not be CLR type names or assembly-qualified type names. This is required so V2 does not depend on polymorphic deserialization or CLR type guessing. Ports of existing serializer IDs may preserve their legacy manifest behavior, including empty or missing manifests, when changing the manifest would break existing wire or persisted data. `SerializerV1Adapter` may also preserve an empty or missing manifest when adapting legacy V1 serializers that historically omitted manifests. -**Decision:** Simple internal serializers (`ClusterMessageSerializer`, `SystemMessageSerializer`, `PrimitiveSerializers`, `ByteArraySerializer`) change base class to `SerializerV2` and use `proto.WriteTo(IBufferWriter)` / `Parser.ParseFrom(ReadOnlySequence)`. Same serializer IDs, same manifests, same wire format. Google.Protobuf natively supports both APIs. +Rationale: remoting and persistence both need serializer ID + manifest. V2 should make this uniform for V1 adapters, V2 hand-written serializers, and generated serializers. -**Rationale:** It's a find-and-replace level change. Wire format is byte-identical. No new IDs needed. Serializers with nested user payloads via `WrappedPayloadSupport` (Sharding, PubSub, ReliableDelivery, Misc) are deferred until the API is validated with simpler serializers first. +### 6. Serialization.cs Stores V2 Internally -### 5. AkkaWriter/AkkaReader are sealed (from POC PR #42) +`Serialization.cs` stores V2 serializers in its ID and type maps. V1 serializers instantiated through HOCON or setup are wrapped on registration. -**Decision:** Use the spike branch approach — sealed concrete classes wrapping MessagePack directly, no `ICodecWriter`/`ICodecReader` interface abstraction. +Public lookup APIs keep returning `Serializer` for compatibility and unwrap `SerializerV1Adapter` when necessary. Internal lookup APIs expose `SerializerV2` for buffer-first paths. -**Rationale:** 22% faster deserialization from JIT devirtualization. The interface layer was YAGNI — we're committed to MessagePack for user message codegen. `AkkaWriter.RawBuffer` escape hatch preserves extensibility for advanced scenarios. +Rationale: V2 is the new cheese for Akka.NET 1.6 internally, but public compatibility is required while classic remoting, persistence, and user serializers still rely on V1 APIs. -### 6. Bridge methods for transport compatibility +### 7. Classic Remoting Is Compatibility, Not Zero-Copy -**Decision:** `SerializerV2` has `ToBinary(object) → byte[]` and `FromBinary(byte[], string) → object` implemented in terms of the buffer API: +Classic remoting remains a byte-array compatibility path and preserves its existing protobuf wire format. Native V2 serializers can still participate through inherited bridge methods, but classic remoting does not need direct `IBufferWriter` integration. -```csharp -public virtual byte[] ToBinary(object obj) -{ - var buffer = new ArrayBufferWriter(SizeHint(obj)); - Serialize(buffer, obj); - return buffer.WrittenSpan.ToArray(); -} +Rationale: the purpose of the classic bridge is to keep existing classic remoting behavior working. Classic remoting will still allocate at protobuf / `ByteString` boundaries. The zero-copy remoting path belongs to Artery. -public virtual object FromBinary(byte[] bytes, string manifest) - => Deserialize(new ReadOnlySequence(bytes), manifest); -``` +### 8. Akka.Delivery Is The Initial V2 Buffer POC -**Rationale:** The current transport (and Spec 3's initial `FrameBufferWriter` path) can call `Serialize(IBufferWriter)` directly. The bridge is for backward compat with code that still expects `byte[]`. The bridge is virtual so it can be bypassed when direct buffer access is available. +Akka.Delivery chunking should use internal V2 serialization APIs to write payloads through `IBufferWriter` and deserialize assembled chunks through `ReadOnlySequence`. + +Rationale: delivery already owns chunked byte payloads and is not tied to classic remoting protobuf envelopes. It is the lowest-risk production path for proving native V2 buffer APIs before Artery. + +### 9. Persistence Compatibility Is Part Of The Foundation + +Persistence event and snapshot serializers must use V2 and preserve stored data compatibility in this same change. + +Rationale: persistence is the highest-risk compatibility surface. Old journal and snapshot data must remain readable, and V2 payloads must store serializer ID + manifest + bytes in the same conceptual model. + +### 10. Sourcegen Comes Next + +Source-generated MessagePack serializers should be implemented only after this foundation is green. + +Rationale: sourcegen validates the V2 API through real serialization, classic remoting, and persistence paths before Artery envelopes depend on it. ## Risks / Trade-offs -**[FindSerializerFor return type change]** → Breaking API change. All callers that type-check the result against `Serializer` or `SerializerWithStringManifest` must update. Mitigated: `SerializerV1Adapter.Inner` provides access to the original V1 serializer. +**Compatibility inheritance tension**: `SerializerV2` being usable as `Serializer` keeps this PR compatible, but permits awkward compositions such as wrapping V2 with `SerializerV1Adapter`. Guardrails should be added before native V2 serializers become common. + +**API break blast radius**: internal V2 APIs should be used for new code, but public lookup APIs remain compatible for this foundation PR. + +**Persistence compatibility**: old data must remain readable. Add explicit tests with V1-serialized event and snapshot bytes. -**[MessagePack as a new dependency]** → Only in the `Akka.Serialization.V2` package, not core Akka. Users who don't opt in are unaffected. MessagePack-CSharp has 215M+ downloads and is widely used in production. +**Classic remoting allocation remains**: acceptable for compatibility. Do not over-optimize classic remoting while Artery is the target high-throughput path. -**[Internal serializer port could introduce bugs]** → Mitigated by wire format being byte-identical (same Protobuf bytes, same IDs). Existing serialization round-trip tests catch regressions. +**Async API uncertainty**: persistence may need async serializer behavior. Decide before Artery, even if the initial API remains sync. -**[WrappedPayloadSupport serializers deferred]** → The 4 serializers with nested user payloads continue using V1 `FindSerializerFor()` → `ToBinary()` through the adapter. They work correctly but miss the zero-copy benefit for the inner payload. Addressed in a follow-up pass after API validation. +**V1 adapter behavior**: adapter must preserve identifiers, manifests, error semantics, and transport information handling. diff --git a/openspec/changes/serializer-v2/proposal.md b/openspec/changes/serializer-v2/proposal.md index 0704da6b5a9..fb17aa6fb4a 100644 --- a/openspec/changes/serializer-v2/proposal.md +++ b/openspec/changes/serializer-v2/proposal.md @@ -1,40 +1,48 @@ ## Why -Akka.NET's current serialization API (`Serializer` / `SerializerWithStringManifest`) is `byte[]`-based. Every serialization produces a `byte[]` allocation, and every deserialization requires a `byte[]` input. With the new Akka.Streams transport (Spec 3) using `IBufferWriter` for writes and `ReadOnlySequence` for reads, the serializer must speak the same language to achieve zero-copy end-to-end. The V2 API makes `SerializerV2` the new foundation — V1 serializers are adapted to it, not the reverse. A POC at github.com/Aaronontheweb/AkkaSerializationPoC (PR #42) has validated the MessagePack-based approach with 22% faster deserialization than the interface-based alternative. +Akka.NET 1.6 intentionally makes `SerializerV2` the canonical serialization abstraction. The current `Serializer` / `SerializerWithStringManifest` API is `byte[]`-based, which forces allocations on every serialization and deserialization path. That is incompatible with the larger 1.6 goals: source-generated serializers, AOT-friendly code, lower allocation remoting, and a future Artery-style transport. -## What Changes - -- **New `SerializerV2` base class in core Akka** — independent (does NOT extend `Serializer`), codec-agnostic, uses `IBufferWriter` / `ReadOnlySequence` as primary API, with `ToBinary()` / `FromBinary()` bridge methods for backward compat -- **New `SerializerV1Adapter : SerializerV2`** in core Akka — wraps legacy `Serializer` / `SerializerWithStringManifest` to participate in V2 infrastructure -- **New `Akka.Serialization.V2` NuGet package** — `MessagePackSerializer : SerializerV2` base class, sealed `AkkaWriter` / `AkkaReader` wrapping MessagePack, attributes (`[AkkaSerializable]`, `[AkkaField]`, `[AkkaSerializer]`) -- **Modify `Serialization.cs`** — use `SerializerV2` as internal storage type, auto-wrap V1 serializers in `SerializerV1Adapter`, `FindSerializerFor()` returns `SerializerV2` -- **Modify `MessageSerializer.cs`** — use V2 dispatch (call `Manifest()` directly, no `is SerializerWithStringManifest` type check) -- **Mechanical port of simple internal Protobuf serializers** — change base class to `SerializerV2`, use `proto.WriteTo(IBufferWriter)` / `Parser.ParseFrom(ReadOnlySequence)`. Same serializer IDs, same wire format. -- **Source generator deferred** — validate API with hand-written serializers first +This change is not only a core serialization API change. Once `Serialization.FindSerializerFor()` returns `SerializerV2`, classic Akka.Remote, Akka.Persistence, DistributedData, delivery, tests, and any caller that stores or type-checks serializers are affected immediately. Therefore this change is the atomic V2 foundation: core API plus required compatibility bridges. -### What does NOT change +## What Changes -- The `Serializer` and `SerializerWithStringManifest` classes remain in the codebase for backward compat (wrapped in adapter) -- Wire format for existing serializers (same Protobuf bytes, same serializer IDs, same manifests) -- HOCON serializer registration (`akka.actor.serializers`, `akka.actor.serialization-bindings`, `akka.actor.serialization-identifiers`) -- `SerializationSetup` programmatic registration API -- Persistence data compatibility (journals store serializerId + manifest + payload — V1 data readable forever) +- **New `SerializerV2` base class in core Akka** - codec-agnostic, based on `IBufferWriter` / `ReadOnlySequence`, and temporarily compatible with existing `Serializer` call sites. +- **New `SerializerV1Adapter : SerializerV2`** - wraps legacy `Serializer` / `SerializerWithStringManifest` implementations. +- **`Serialization.cs` becomes V2-first internally** - internal dictionaries store `SerializerV2`; HOCON and `SerializationSetup` V1 serializers are auto-wrapped; public `FindSerializerFor()` and `FindSerializerForType()` keep returning `Serializer` for compatibility while internal V2 lookup APIs are added. +- **Native V2 serializers require non-CLR manifests** - new native V2 serializers must return non-empty, serializer-owned manifest tokens to move Akka.NET away from polymorphic deserialization; existing serializer-id ports and V1 adapters may preserve empty manifests for backward compatibility. +- **Classic remoting compatibility bridge** - classic Akka.Remote can continue using byte-array `Serializer` APIs and existing protobuf wire format; native V2 serializers work there through V2 bridge methods. +- **Akka.Persistence V2 bridge** - persistence event and snapshot serializers use V2 while preserving existing stored event and snapshot compatibility. +- **Akka.Delivery V2 proof-of-concept** - delivery chunking writes through `IBufferWriter` and reassembles chunks as `ReadOnlySequence` to exercise the V2 path outside classic remoting. +- **Call-site fallout fixed in the same change** - delivery, DistributedData, tests, and other direct serializer call sites compile and preserve behavior. +- **API shape decisions are finalized before sourcegen / Artery** - bytes-written reporting, unknown `SizeHint`, manifest semantics, V1 fallback behavior, and sync/async serializer behavior are settled here. + +### What Does Not Change + +- Existing V1 serializer classes remain in the codebase. +- Existing V1 serializer registrations continue to work through `SerializerV1Adapter`. +- Classic Akka.Remote wire compatibility is preserved for the classic transport path. +- Existing Akka.Persistence journal events and snapshots remain readable. +- HOCON serializer registration remains supported. +- `SerializationSetup` remains supported, updated to participate in the V2-first model. +- MessagePack source generation is not implemented in this change; it is the next validation gate. +- Artery envelopes are not implemented in this change. ## Capabilities ### New Capabilities -- `serializer-v2-base`: The `SerializerV2` base class with `IBufferWriter` / `ReadOnlySequence` API, `SerializerV1Adapter`, and infrastructure changes to `Serialization.cs` and `MessageSerializer.cs`. Lives in core Akka. -- `messagepack-serializer`: The `MessagePackSerializer : SerializerV2` intermediate class, sealed `AkkaWriter`/`AkkaReader` wrapping MessagePack, and serializable type attributes. Lives in `Akka.Serialization.V2` package. +- `serializer-v2-foundation`: V2 base API, V1 adapter, `Serialization.cs` V2-first infrastructure, classic remoting bridge, persistence bridge, and required call-site updates. ### Modified Capabilities +- `classic-remoting-serialization`: classic remoting uses V2 payload serialization while retaining classic protobuf envelope compatibility. +- `persistence-payload-serialization`: persistence payloads use V2 serialization while retaining stored data compatibility. + ## Impact -- **Akka core** (`src/core/Akka/Serialization/`): New `SerializerV2.cs`, `SerializerV1Adapter.cs`. Modified `Serialization.cs` (internal storage type, auto-wrapping), `MessageSerializer.cs` (V2 dispatch). -- **Akka.Remote**: `MessageSerializer.cs` simplified — calls `Manifest()` directly. In Spec 3, `EndpointWriter` uses `serializer.Serialize(FrameBufferWriter)` directly. -- **New package**: `Akka.Serialization.V2` with `MessagePackSerializer`, `AkkaWriter`, `AkkaReader`, attributes. -- **NuGet dependencies**: `MessagePack` added to `Akka.Serialization.V2` (not core Akka). `System.IO.Pipelines` / `System.Memory` already in core from Spec 1. -- **Internal serializers**: Mechanical base class change for simple ones (same wire format). Complex ones with nested payloads via `WrappedPayloadSupport` deferred until V2 API is validated. -- **API surface**: `FindSerializerFor()` return type changes from `Serializer` to `SerializerV2`. -- **Test suites**: All existing serialization tests must pass (V1 serializers auto-wrapped transparently). New tests for V2 round-trip, adapter, and hand-written MessagePack serializers. +- **Akka core** (`src/core/Akka/Serialization/`): add `SerializerV2.cs`, `SerializerV1Adapter.cs`; modify `Serialization.cs`, `SerializationSetup.cs`, and tests. +- **Akka.Remote**: update `MessageSerializer.cs`, `Serialization/WrappedPayloadSupport.cs`, and affected tests. Classic `AkkaPduCodec` remains wire-compatible. +- **Akka.Persistence**: update `PersistenceMessageSerializer.cs`, `PersistenceSnapshotSerializer.cs`, `LocalSnapshotStore.cs`, and persistence serialization tests. +- **Other modules**: update delivery chunk serialization, DistributedData serializer usages, and tests that assert concrete serializer types. +- **API surface**: public lookup APIs remain compatible and return `Serializer`; internal V2 APIs expose `SerializerV2` where new buffer-first paths need it. +- **API approval tests**: update baselines for intentional additive API changes and native V2 serializer ports. diff --git a/openspec/changes/serializer-v2/specs/messagepack-serializer/spec.md b/openspec/changes/serializer-v2/specs/messagepack-serializer/spec.md deleted file mode 100644 index b768e862806..00000000000 --- a/openspec/changes/serializer-v2/specs/messagepack-serializer/spec.md +++ /dev/null @@ -1,69 +0,0 @@ -## ADDED Requirements - -### Requirement: MessagePackSerializer base class -The `Akka.Serialization.V2` package SHALL provide `MessagePackSerializer : SerializerV2` that bridges the `IBufferWriter` / `ReadOnlySequence` API to typed `AkkaWriter` / `AkkaReader` methods. - -#### Scenario: Serialize bridges to AkkaWriter -- **WHEN** `MessagePackSerializer.Serialize(buffer, obj)` is called -- **THEN** it SHALL create an `AkkaWriter(buffer)` and call the abstract `Write(AkkaWriter, obj)` method - -#### Scenario: Deserialize bridges to AkkaReader -- **WHEN** `MessagePackSerializer.Deserialize(buffer, manifest)` is called -- **THEN** it SHALL create an `AkkaReader(buffer)` and call the abstract `Read(AkkaReader, manifest)` method - -#### Scenario: Protocol-scoped generic variant -- **WHEN** a user defines `MySerializer : MessagePackSerializer` -- **THEN** the `TProtocol` type parameter SHALL scope which message types belong to this serializer (used by future source generator) - -### Requirement: Sealed AkkaWriter wraps MessagePack -The `AkkaWriter` class SHALL be sealed and wrap an `IBufferWriter` with typed write methods backed by MessagePack encoding. - -#### Scenario: Write primitive types -- **WHEN** `AkkaWriter.WriteInt32(value)`, `WriteInt64(value)`, `WriteString(value)`, `WriteBool(value)`, `WriteDouble(value)` are called -- **THEN** they SHALL encode the values in MessagePack format directly to the underlying `IBufferWriter` - -#### Scenario: Write complex types -- **WHEN** `AkkaWriter.WriteDateTime(value)`, `WriteDateTimeOffset(value)`, `WriteGuid(value)`, `WriteDecimal(value)` are called -- **THEN** they SHALL encode using the conventions defined in the POC (DateTime as [ticks, Kind], Guid as 16-byte binary, Decimal as [lo, mid, hi, flags]) - -#### Scenario: BeginObject for structured writes -- **WHEN** `AkkaWriter.BeginObject(fieldCount)` is called -- **THEN** it SHALL write a MessagePack array header enabling structured field-by-field serialization - -#### Scenario: RawBuffer escape hatch -- **WHEN** advanced MessagePack scenarios require direct buffer access -- **THEN** `AkkaWriter.RawBuffer` SHALL expose the underlying `IBufferWriter` - -### Requirement: Sealed AkkaReader wraps MessagePack -The `AkkaReader` class SHALL be sealed and wrap a `ReadOnlyMemory` (from the `ReadOnlySequence` input) with typed read methods backed by MessagePack decoding, tracking consumed byte offset. - -#### Scenario: Read primitive types -- **WHEN** `AkkaReader.ReadInt32()`, `ReadInt64()`, `ReadString()`, `ReadBool()`, `ReadDouble()` are called -- **THEN** they SHALL decode MessagePack values and advance the internal consumed offset - -#### Scenario: Read complex types -- **WHEN** `AkkaReader.ReadDateTime()`, `ReadDateTimeOffset()`, `ReadGuid()`, `ReadDecimal()` are called -- **THEN** they SHALL decode using the same conventions as AkkaWriter - -#### Scenario: Version tolerance via SkipField -- **WHEN** `AkkaReader.SkipField()` is called for unknown trailing fields (newer message version) -- **THEN** it SHALL skip the next MessagePack value without decoding, advancing the offset - -#### Scenario: TryReadNull for nullable fields -- **WHEN** `AkkaReader.TryReadNull()` is called and the next value is null -- **THEN** it SHALL consume the null and return true; otherwise leave position unchanged and return false - -### Requirement: Serialization attributes -The `Akka.Serialization.V2` package SHALL provide attributes for marking serializable types and their serializers. - -#### Scenario: AkkaSerializable attribute marks message types -- **WHEN** a class or record is annotated with `[AkkaSerializable]` -- **THEN** it SHALL be recognized as a type that can be serialized by a V2 MessagePack serializer - -#### Scenario: AkkaField attribute marks serializable properties -- **WHEN** a property is annotated with `[AkkaField(index)]` -- **THEN** it SHALL be included in serialization at the specified field index (used for version tolerance) - -#### Scenario: AkkaSerializer attribute marks serializer classes -- **WHEN** a partial class is annotated with `[AkkaSerializer]` -- **THEN** it SHALL be recognized as a target for future source generator code emission diff --git a/openspec/changes/serializer-v2/specs/serializer-v2-base/spec.md b/openspec/changes/serializer-v2/specs/serializer-v2-base/spec.md index ccd1084da9b..4f5a3e86088 100644 --- a/openspec/changes/serializer-v2/specs/serializer-v2-base/spec.md +++ b/openspec/changes/serializer-v2/specs/serializer-v2-base/spec.md @@ -1,80 +1,156 @@ ## ADDED Requirements -### Requirement: SerializerV2 base class with buffer API -The system SHALL provide an abstract `SerializerV2` class in core Akka with `Serialize(IBufferWriter, object)` and `Deserialize(ReadOnlySequence, string)` as the primary API. It SHALL NOT extend `Serializer` or `SerializerWithStringManifest`. +### Requirement: SerializerV2 base class with buffer-first API + +The system SHALL provide an abstract `SerializerV2` class in core Akka that is the canonical Akka.NET 1.6 serializer abstraction. + +#### Scenario: V2 remains compatible with existing serializer call sites +- **WHEN** `SerializerV2` is defined +- **THEN** it SHALL be usable anywhere existing compatibility code expects `Serializer` +- **AND** native V2 serializers SHALL still implement the buffer-first V2 API #### Scenario: Serialize to IBufferWriter -- **WHEN** `SerializerV2.Serialize(buffer, obj)` is called -- **THEN** the serializer SHALL write the serialized bytes directly into the provided `IBufferWriter` without allocating an intermediate `byte[]` +- **WHEN** `SerializerV2.Serialize(...)` is called with an `IBufferWriter` +- **THEN** the serializer SHALL write serialized bytes into the provided writer without requiring an intermediate `byte[]` + +#### Scenario: Serialize reports bytes written +- **WHEN** `SerializerV2.Serialize(...)` completes +- **THEN** the caller SHALL be able to determine how many payload bytes were written #### Scenario: Deserialize from ReadOnlySequence -- **WHEN** `SerializerV2.Deserialize(buffer, manifest)` is called with a `ReadOnlySequence` +- **WHEN** `SerializerV2.Deserialize(...)` is called with a `ReadOnlySequence` - **THEN** the serializer SHALL read from the sequence and return the deserialized object +#### Scenario: Manifest is direct V2 API +- **WHEN** a caller needs the manifest for an object +- **THEN** it SHALL call a direct V2 manifest API without type-checking for `SerializerWithStringManifest` + +#### Scenario: New native V2 manifest is required +- **WHEN** a new native `SerializerV2` is used to serialize an object +- **THEN** it SHALL return a non-empty manifest +- **AND** the manifest SHALL be a serializer-owned token rather than a CLR type name +- **AND** the serialization system SHALL reject empty native V2 manifests +- **AND** this requirement SHALL NOT prevent existing serializer-id ports or `SerializerV1Adapter` from accepting or preserving empty legacy manifests required for compatibility + +#### Scenario: Unknown size hint +- **WHEN** a serializer cannot cheaply predict serialized size +- **THEN** `SizeHint` SHALL return an explicit unknown-size value rather than a misleading estimate + #### Scenario: ToBinary bridge method -- **WHEN** `SerializerV2.ToBinary(obj)` is called -- **THEN** it SHALL create an `ArrayBufferWriter`, call `Serialize()`, and return the written bytes as `byte[]` +- **WHEN** compatibility code calls `SerializerV2.ToBinary(obj)` +- **THEN** it SHALL return a `byte[]` containing the serialized payload #### Scenario: FromBinary bridge method -- **WHEN** `SerializerV2.FromBinary(bytes, manifest)` is called -- **THEN** it SHALL wrap the `byte[]` in `new ReadOnlySequence(bytes)` and call `Deserialize()` +- **WHEN** compatibility code calls `SerializerV2.FromBinary(bytes, manifest)` +- **THEN** it SHALL deserialize the payload using the V2 deserialization path ### Requirement: SerializerV1Adapter wraps legacy serializers -The system SHALL provide `SerializerV1Adapter : SerializerV2` that wraps any `Serializer` or `SerializerWithStringManifest` instance to participate in the V2 infrastructure. -#### Scenario: V1 serializer wrapped for V2 dispatch -- **WHEN** a V1 `Serializer` is registered via HOCON or `SerializationSetup` -- **THEN** `Serialization.cs` SHALL auto-wrap it in `SerializerV1Adapter` for internal storage +The system SHALL provide `SerializerV1Adapter : SerializerV2` that wraps any legacy `Serializer` or `SerializerWithStringManifest` instance. -#### Scenario: Adapter delegates to V1 ToBinary/FromBinary -- **WHEN** `SerializerV1Adapter.Serialize(buffer, obj)` is called -- **THEN** it SHALL call the inner V1 serializer's `ToBinary(obj)` and write the resulting bytes to the buffer +#### Scenario: V1 serializer wrapped for V2 dispatch +- **WHEN** a V1 serializer is registered via HOCON or `SerializationSetup` +- **THEN** `Serialization.cs` SHALL wrap it in `SerializerV1Adapter` before storing it internally -#### Scenario: Adapter preserves serializer identity +#### Scenario: Adapter preserves serializer identifier - **WHEN** `SerializerV1Adapter.Identifier` is accessed -- **THEN** it SHALL return the inner V1 serializer's `Identifier` +- **THEN** it SHALL return the wrapped serializer's `Identifier` + +#### Scenario: Adapter preserves manifest behavior +- **WHEN** `SerializerV1Adapter` is asked for a manifest +- **THEN** it SHALL preserve the wrapped serializer's string-manifest or type-manifest behavior -#### Scenario: Access to inner V1 serializer -- **WHEN** code needs the original V1 `Serializer` instance -- **THEN** `SerializerV1Adapter.Inner` SHALL return the wrapped V1 serializer +#### Scenario: Adapter exposes inner serializer +- **WHEN** migration code or tests need the original V1 serializer +- **THEN** `SerializerV1Adapter.Inner` SHALL return it -### Requirement: Serialization.cs uses V2 internally -The `Serialization` class SHALL store `SerializerV2` instances in its internal dictionaries and return `SerializerV2` from `FindSerializerFor()`. +### Requirement: Serialization uses V2 internally -#### Scenario: FindSerializerFor returns SerializerV2 +The `Serialization` class SHALL store `SerializerV2` instances internally while preserving public lookup compatibility. + +#### Scenario: Public FindSerializerFor remains compatible - **WHEN** `Serialization.FindSerializerFor(msg)` is called -- **THEN** it SHALL return a `SerializerV2` instance (either a native V2 serializer or a `SerializerV1Adapter` wrapping a V1 serializer) +- **THEN** it SHALL return `Serializer` +- **AND** internal V2 lookup APIs SHALL be available for buffer-first call sites -#### Scenario: V1 serializers auto-wrapped on registration -- **WHEN** a V1 serializer is instantiated from HOCON configuration -- **THEN** it SHALL be wrapped in `SerializerV1Adapter` before storage in internal dictionaries +#### Scenario: Public FindSerializerForType remains compatible +- **WHEN** `Serialization.FindSerializerForType(type)` is called +- **THEN** it SHALL return `Serializer` +- **AND** internal V2 lookup APIs SHALL be available for buffer-first call sites -#### Scenario: V2 serializers registered directly -- **WHEN** a V2 serializer is instantiated from HOCON configuration (detected by `is SerializerV2`) -- **THEN** it SHALL be stored directly without wrapping +#### Scenario: Serialize dispatches through V2 writer API +- **WHEN** internal code calls the V2 serialization API with an `IBufferWriter` +- **THEN** the serialization system SHALL locate the configured `SerializerV2` +- **AND** write the payload into the provided writer +- **AND** return serializer ID, manifest, and bytes-written metadata #### Scenario: Deserialize dispatches through V2 - **WHEN** `Serialization.Deserialize(bytes, serializerId, manifest)` is called -- **THEN** it SHALL look up the `SerializerV2` by ID and call `FromBinary(bytes, manifest)` (bridge method) +- **THEN** it SHALL look up the V2 serializer by ID and deserialize through V2 + +#### Scenario: Missing serializer errors remain actionable +- **WHEN** a serializer ID cannot be found +- **THEN** the thrown error SHALL include the serializer ID and existing serializer error guidance + +### Requirement: Classic remoting uses V2 while preserving wire compatibility + +Classic Akka.Remote SHALL preserve the existing classic protobuf wire format and remain compatible with both V1 and V2 serializers through byte-array bridge APIs. + +#### Scenario: Classic remoting serializes V1 payload through adapter +- **WHEN** a V1-serialized message is sent over classic remoting +- **THEN** the message SHALL be serialized through `SerializerV1Adapter` and encoded in the existing classic wire envelope + +#### Scenario: Classic remoting serializes native V2 payload +- **WHEN** a native V2-serialized message is sent over classic remoting +- **THEN** the message SHALL be serialized through V2 compatibility bridge methods and encoded in the existing classic wire envelope + +#### Scenario: Classic remoting reads old payloads +- **WHEN** classic remoting receives an existing protobuf `Payload` +- **THEN** it SHALL deserialize the payload through V2 by serializer ID and manifest + +### Requirement: Akka.Persistence uses V2 while preserving stored data compatibility + +Akka.Persistence SHALL use `SerializerV2` for event and snapshot payloads while preserving the existing stored serializer ID + manifest + payload model. + +#### Scenario: Old journal event remains readable +- **WHEN** a journal event written with a V1 serializer is replayed +- **THEN** it SHALL deserialize successfully through V2 infrastructure + +#### Scenario: Old snapshot remains readable +- **WHEN** a snapshot written with a V1 serializer is loaded +- **THEN** it SHALL deserialize successfully through V2 infrastructure + +#### Scenario: New V2 event persists and recovers +- **WHEN** an event uses a native V2 serializer +- **THEN** persistence SHALL store serializer ID, manifest, and payload bytes and recover the original event + +#### Scenario: New V2 snapshot saves and loads +- **WHEN** a snapshot uses a native V2 serializer +- **THEN** persistence SHALL store serializer ID, manifest, and payload bytes and load the original snapshot + +### Requirement: Akka.Delivery exercises V2 buffer APIs + +Akka.Delivery SHALL provide the initial production proof-of-concept for V2 buffer-oriented serialization outside classic remoting. -### Requirement: MessageSerializer uses V2 dispatch -The `MessageSerializer` in Akka.Remote SHALL use V2 serializer dispatch, calling `Manifest()` directly on `SerializerV2` without type-checking for `SerializerWithStringManifest`. +#### Scenario: Delivery chunk creation uses V2 writer API +- **WHEN** Akka.Delivery creates chunks for a serializable message +- **THEN** it SHALL serialize the payload through `SerializerV2.Serialize(..., IBufferWriter)` +- **AND** each chunk SHALL carry serializer ID and manifest metadata from the V2 path -#### Scenario: Serialize with manifest -- **WHEN** `MessageSerializer.Serialize(system, transportInfo, message)` is called -- **THEN** it SHALL call `serializer.Manifest(message)` directly (all V2 serializers have manifests) and include it in the wire message +#### Scenario: Delivery chunk assembly uses ReadOnlySequence +- **WHEN** Akka.Delivery assembles received chunks +- **THEN** it SHALL pass a `ReadOnlySequence` to V2 deserialization +- **AND** it SHALL deserialize the original message by serializer ID and manifest -### Requirement: Internal Protobuf serializers ported to V2 -Simple internal Protobuf serializers SHALL extend `SerializerV2` directly, using `proto.WriteTo(IBufferWriter)` and `Parser.ParseFrom(ReadOnlySequence)`. Wire format and serializer IDs SHALL remain unchanged. +### Requirement: ByteArraySerializer is a native V2 serializer -#### Scenario: ClusterMessageSerializer uses V2 base -- **WHEN** `ClusterMessageSerializer` serializes a cluster message -- **THEN** it SHALL write Protobuf bytes directly to `IBufferWriter` via `proto.WriteTo()`, producing byte-identical output to the V1 implementation +The built-in byte-array serializer SHALL be ported to native `SerializerV2` while preserving legacy byte-array behavior. -#### Scenario: Deserialization uses ReadOnlySequence -- **WHEN** a Protobuf serializer deserializes a message -- **THEN** it SHALL parse from `ReadOnlySequence` via `Parser.ParseFrom()`, producing the same object as the V1 implementation +#### Scenario: ByteArraySerializer preserves legacy empty manifest +- **WHEN** `ByteArraySerializer` serializes a byte array through V2 +- **THEN** it SHALL emit the legacy empty manifest for serializer id `4` +- **AND** it SHALL keep new writes readable by older Akka.NET versions -#### Scenario: Serializer IDs preserved -- **WHEN** an internal serializer is ported to V2 -- **THEN** its `Identifier` property SHALL return the same integer ID as the V1 version +#### Scenario: ByteArraySerializer accepts legacy empty manifests +- **WHEN** `ByteArraySerializer` deserializes bytes with a null or empty manifest +- **THEN** it SHALL deserialize successfully for backward compatibility with V1 payloads diff --git a/openspec/changes/serializer-v2/tasks.md b/openspec/changes/serializer-v2/tasks.md index 3b9c8e9dea6..a8607cc5b3a 100644 --- a/openspec/changes/serializer-v2/tasks.md +++ b/openspec/changes/serializer-v2/tasks.md @@ -1,48 +1,91 @@ -## 1. SerializerV2 Base Class (Core Akka) - -- [ ] 1.1 Create `SerializerV2` abstract class in `src/core/Akka/Serialization/SerializerV2.cs` with `Serialize(IBufferWriter)`, `Deserialize(ReadOnlySequence)`, `Manifest()`, `Identifier`, `SizeHint()`, `ToBinary()` bridge, `FromBinary()` bridge -- [ ] 1.2 Create `SerializerV1Adapter : SerializerV2` in `src/core/Akka/Serialization/SerializerV1Adapter.cs` — wraps V1 `Serializer`/`SerializerWithStringManifest`, exposes `Inner` property -- [ ] 1.3 Modify `Serialization.cs`: change `_serializersById` to `Dictionary`, `_serializerMap` to `ConcurrentDictionary` -- [ ] 1.4 Modify `Serialization.cs` constructor: auto-wrap V1 serializers from HOCON in `SerializerV1Adapter`, detect and store V2 serializers directly -- [ ] 1.5 Update `FindSerializerFor()` return type to `SerializerV2` -- [ ] 1.6 Update `Deserialize(bytes, serializerId, manifest)` to use V2 dispatch -- [ ] 1.7 Fix all callers of `FindSerializerFor()` across the codebase for the return type change -- [ ] 1.8 Modify `MessageSerializer.cs`: call `serializer.Manifest(message)` directly, remove `is SerializerWithStringManifest` type check -- [ ] 1.9 Unit tests: `SerializerV1Adapter` round-trip with built-in serializers -- [ ] 1.10 Verify all existing serialization tests pass (V1 auto-wrapped transparently) - -## 2. Akka.Serialization.V2 Package - -- [ ] 2.1 Create `src/core/Akka.Serialization.V2/` project, target `netstandard2.1;net6.0`, reference `Akka` + `MessagePack` -- [ ] 2.2 Port `AkkaWriter` (sealed class) from POC PR #42: typed write methods, `BeginObject`, `RawBuffer` escape hatch -- [ ] 2.3 Port `AkkaReader` (sealed class) from POC PR #42: typed read methods, `BeginReadObject`, `SkipField`, `TryReadNull`, consumed offset tracking -- [ ] 2.4 Create `MessagePackSerializer : SerializerV2` base class: bridges `Serialize` → `Write(AkkaWriter)`, `Deserialize` → `Read(AkkaReader)` -- [ ] 2.5 Create `MessagePackSerializer` generic variant for protocol scoping -- [ ] 2.6 Port attributes from POC: `[AkkaSerializable]`, `[AkkaField(index)]`, `[AkkaSerializer]` -- [ ] 2.7 Add `MessagePack` version to `Directory.Build.props` -- [ ] 2.8 Add project to `Akka.slnx` -- [ ] 2.9 Unit tests: `AkkaWriter`/`AkkaReader` round-trip for all supported types (primitives, DateTime, Guid, Decimal, collections, nullable, nested objects) - -## 3. Hand-Written Validation Serializers - -- [ ] 3.1 Create a hand-written `MessagePackSerializer` for a sample user message type (demonstrate the Write/Read API without codegen) -- [ ] 3.2 Test: round-trip serialization through full `Serialization.cs` pipeline -- [ ] 3.3 Test: register V2 serializer via HOCON, send message to remote actor -- [ ] 3.4 Test: V1 and V2 serializers coexist (different message types, different serializer IDs) -- [ ] 3.5 Test: persistence write with V2 serializer, read back (verify journal stores serializerId + manifest correctly) - -## 4. Internal Protobuf Serializer Migration (Simple) - -- [ ] 4.1 Port `PrimitiveSerializers` (ID 17) to extend `SerializerV2` — use `proto.WriteTo(IBufferWriter)` / `Parser.ParseFrom(ReadOnlySequence)` -- [ ] 4.2 Port `ByteArraySerializer` (ID 4) to extend `SerializerV2` -- [ ] 4.3 Port `SystemMessageSerializer` (ID 22) to extend `SerializerV2` -- [ ] 4.4 Port `ClusterMessageSerializer` (ID 5) to extend `SerializerV2` -- [ ] 4.5 Verify wire format is byte-identical for each ported serializer (round-trip test with V1-serialized bytes) -- [ ] 4.6 Run full test suite: `dotnet test -c Release` - -## 5. Validation - -- [ ] 5.1 Verify `dotnet build -warnaserror` passes -- [ ] 5.2 Verify all Akka.Remote tests pass (V1 serializers auto-wrapped) -- [ ] 5.3 Verify all Akka.Persistence tests pass (V1 data readable) -- [ ] 5.4 Run `dotnet test -c Release src/core/Akka.API.Tests` — update API approval baselines for `FindSerializerFor` return type change +## 1. SerializerV2 API + +- [ ] 1.1 Create `SerializerV2` in `src/core/Akka/Serialization/SerializerV2.cs` +- [ ] 1.2 Define buffer-first serialization API using `IBufferWriter` +- [ ] 1.3 Define `ReadOnlySequence` deserialization API +- [x] 1.4 Define direct manifest API for all V2 serializers +- [ ] 1.5 Define `SizeHint` with explicit unknown-size support +- [ ] 1.6 Decide and implement bytes-written/result reporting for `Serialize` +- [ ] 1.7 Decide sync vs async V2 API shape before downstream sourcegen and Artery work +- [ ] 1.8 Add `ToBinary` / `FromBinary` bridge methods for compatibility paths +- [ ] 1.9 Add unit tests for native V2 serializer round-trip, manifest, size hint, and bridge methods +- [x] 1.10 Require new native V2 serializers to emit non-empty non-CLR manifests; allow legacy serializer-id ports and `SerializerV1Adapter` to preserve empty legacy manifests + +## 2. SerializerV1Adapter + +- [ ] 2.1 Create `SerializerV1Adapter : SerializerV2` +- [ ] 2.2 Preserve inner serializer `Identifier` +- [ ] 2.3 Preserve `SerializerWithStringManifest.Manifest` behavior +- [ ] 2.4 Preserve `IncludeManifest` type-manifest behavior for non-string-manifest V1 serializers +- [ ] 2.5 Delegate V1 serialization to `ToBinary` +- [ ] 2.6 Delegate V1 deserialization to `FromBinary` +- [ ] 2.7 Preserve `Serialization.CurrentTransportInformation` behavior +- [ ] 2.8 Expose `Inner` for migration and tests +- [ ] 2.9 Add adapter tests for built-in V1 serializers and custom V1 serializers + +## 3. Serialization.cs V2-First Infrastructure + +- [ ] 3.1 Change serializer ID map to store `SerializerV2` +- [ ] 3.2 Change serializer type map to store `SerializerV2` +- [ ] 3.3 Wrap HOCON-registered V1 serializers in `SerializerV1Adapter` +- [ ] 3.4 Store native V2 serializers directly +- [ ] 3.5 Update `SerializationSetup` handling for V1 and V2 serializers +- [x] 3.6 Preserve public `FindSerializerFor()` compatibility and add internal V2 lookup +- [x] 3.7 Preserve public `FindSerializerForType()` compatibility and add internal V2 type lookup +- [x] 3.8 Update `Serialization.ManifestFor` or replace it with V2-first manifest dispatch +- [x] 3.9 Add internal `Serialize(object, IBufferWriter)` V2 path +- [x] 3.10 Add internal `Deserialize(ReadOnlySequence, serializerId, manifest)` V2 path +- [ ] 3.11 Update error messages for missing serializer IDs to remain compatible and actionable +- [x] 3.12 Update API approval baselines for intentional 1.6 API break + +## 4. Classic Akka.Remote V2 Bridge + +- [ ] 4.1 Update `src/core/Akka.Remote/MessageSerializer.cs` to use V2 serializer lookup and manifest dispatch +- [ ] 4.2 Preserve existing classic protobuf `Payload` wire format +- [ ] 4.3 Preserve existing classic deserialization behavior and error handling +- [ ] 4.4 Update `src/core/Akka.Remote/Serialization/WrappedPayloadSupport.cs` to use V2 +- [ ] 4.5 Update nested payload serialization to preserve serializer ID + manifest + payload semantics +- [ ] 4.6 Keep `AkkaPduCodec` classic wire-compatible +- [ ] 4.7 Add classic remoting tests using V1 adapter payloads +- [ ] 4.8 Add classic remoting tests using native V2 payloads +- [ ] 4.9 Verify existing Akka.Remote tests pass + +## 5. Akka.Persistence V2 Bridge + +- [ ] 5.1 Update `PersistenceMessageSerializer.GetPersistentPayload` to use V2 +- [ ] 5.2 Update `PersistenceMessageSerializer.GetPayload` to deserialize through V2 +- [ ] 5.3 Update `PersistenceSnapshotSerializer` to use V2 +- [ ] 5.4 Update `LocalSnapshotStore` wrapper serializer type and calls for V2 +- [ ] 5.5 Preserve protobuf persistence envelope format for events and snapshots +- [ ] 5.6 Add tests proving old V1-serialized events remain readable +- [ ] 5.7 Add tests proving old V1-serialized snapshots remain readable +- [ ] 5.8 Add tests proving native V2 events persist and recover +- [ ] 5.9 Add tests proving native V2 snapshots save and load +- [ ] 5.10 Verify existing Akka.Persistence tests pass + +## 6. Compile Fallout Across Repo + +- [ ] 6.1 Fix DistributedData serializer call sites that store or call `Serializer` +- [x] 6.2 Fix Akka.Delivery chunk serialization call sites +- [ ] 6.3 Fix cluster and sharding serializer tests that assert concrete serializer types +- [ ] 6.4 Fix any code that type-checks against `SerializerWithStringManifest` +- [ ] 6.5 Fix any code that assumes `FindSerializerForType()` returns a V1 serializer +- [ ] 6.6 Keep V1 access possible through `SerializerV1Adapter.Inner` only where required + +## 7. Optional Low-Risk Internal Serializer Ports + +- [x] 7.1 Port `ByteArraySerializer` to native V2 if trivial +- [ ] 7.2 Port simple primitive serializers to native V2 if trivial +- [ ] 7.3 Port `SystemMessageSerializer` to native V2 only if compatibility tests are already in place +- [x] 7.4 Verify byte-identical output for any internal serializer ported in this milestone +- [x] 7.5 Preserve `ByteArraySerializer` legacy empty-manifest reads and writes for serializer id `4` +- [ ] 7.6 Revisit `SerializerV2 : Serializer` hierarchy before adding more native production V2 serializers + +## 8. Validation + +- [ ] 8.1 Run `dotnet build -warnaserror` +- [x] 8.2 Run focused Akka.Serialization tests +- [ ] 8.3 Run focused Akka.Remote tests +- [ ] 8.4 Run focused Akka.Persistence tests +- [x] 8.5 Run `dotnet test -c Release src/core/Akka.API.Tests` +- [ ] 8.6 Document any remaining V1-only compatibility boundaries for the sourcegen milestone diff --git a/openspec/changes/streams-tcp-transport/design.md b/openspec/changes/streams-tcp-transport/design.md deleted file mode 100644 index 75b6eb54f20..00000000000 --- a/openspec/changes/streams-tcp-transport/design.md +++ /dev/null @@ -1,129 +0,0 @@ -## Context - -Akka.Remote's transport layer is pluggable via the abstract `Transport` class (`src/core/Akka.Remote/Transport/Transport.cs`). The current concrete implementation is `DotNettyTransport` which uses DotNetty's channel pipeline for framing, TLS, and socket I/O. Above the transport sits `AkkaProtocolTransport` — an adapter that handles the Akka protocol handshake (Associate/Disassociate), heartbeats, and association state management. `AkkaProtocolTransport` doesn't change. - -The transport API: -- `Listen()` → binds a server socket, returns address + listener -- `Associate(remoteAddress)` → connects to remote, returns `AssociationHandle` -- `AssociationHandle.Write(ReadOnlyMemory)` → send framed data (after Spec 1, payload is `ReadOnlyMemory`) -- `AssociationHandle.ReadHandlerSource` → listener receives `InboundPayload` events - -With Specs 1+2, Akka.IO TCP now uses `Stream` + `Pipe` + `IStreamProvider` (with optional TLS). Akka.Streams TCP (`Tcp.Bind()`, `Tcp.OutgoingConnection()`) wraps Akka.IO TCP actors in a `Flow, ReadOnlyMemory>`. The new transport builds on this. - -## Goals / Non-Goals - -**Goals:** -- Implement `StreamsTcpTransport : Transport` using Akka.Streams TCP -- Integrated framing + serialization via `FrameBufferWriter : IBufferWriter` (single buffer, zero intermediate copies) -- Replace Protobuf PDU encoding with simple binary encoding directly to `IBufferWriter` -- All existing `akka.remote.dot-netty.tcp.*` HOCON configuration works unchanged -- All non-DotNetty-specific Akka.Remote specs pass -- Remove DotNetty dependency entirely - -**Non-Goals:** -- Changing the `AkkaProtocolTransport` adapter (handshake, heartbeat, association management) -- Changing the `Endpoint` / `EndpointWriter` / `EndpointReader` actor hierarchy -- UDP transport (separate, later) -- QUIC transport (future, different spec) -- Optimizing flush batching (Spec 5 — Performance) - -## Decisions - -### 1. FrameBufferWriter for integrated framing + serialization - -**Decision:** Create `FrameBufferWriter : IBufferWriter` that wraps a pooled `byte[]` with a start offset. The write path reserves 4 bytes for the length header, writes PDU metadata + serialized payload via `IBufferWriter`, then backfills the length. One buffer, one syscall. - -``` -Write path: - FrameBufferWriter(pooledArray, startOffset: 4) - → write PDU header (serializerId, manifest length, manifest bytes) - → serializer.Serialize(writer, msg) ← payload directly in same buffer - → backfill buffer[0..4] with total length - → ReadOnlyMemory(array, 0, 4 + writtenCount) - → stream.WriteAsync() -``` - -**Rationale:** The length header is always exactly 4 bytes. Reserve them upfront, write the rest, backfill. No need for a separate framing stage (which would require an extra copy). The serializer's `IBufferWriter` contract flows end-to-end from serialization through framing to the socket. - -**Alternative considered:** Separate Akka.Streams framing stage. Rejected for the transport — adds an unnecessary element copy between stages. The general-purpose `Framing.LengthField()` stage remains available for user-facing Streams TCP. - -### 2. Binary PDU encoding (replaces Protobuf AkkaPduCodec) - -**Decision:** Replace the Protobuf `SerializedMessage` / `AkkaProtocolMessage` envelope with simple binary encoding written directly to `IBufferWriter`. - -``` -PDU format (written to IBufferWriter): - [4 bytes] total frame length (little-endian int32) - [1 byte] PDU type (0x01 = payload, 0x02 = associate, 0x03 = disassociate, 0x04 = heartbeat) - --- for payload PDU type --- - [4 bytes] serializerId (little-endian int32) - [2 bytes] manifest length (little-endian uint16) - [N bytes] manifest (UTF8) - [M bytes] serialized payload (remaining bytes = frame length - header) -``` - -**Rationale:** The Protobuf PDU encoding allocates intermediate `byte[]` arrays and Protobuf `ByteString` wrappers. Direct binary encoding to `IBufferWriter` eliminates these allocations. The PDU format is simple enough that Protobuf's schema evolution isn't needed — it's a fixed internal protocol, not a public API. - -**Wire compatibility note:** This changes the PDU encoding format. A v1.6 node CANNOT talk to a v1.5 node. This is acceptable for a major version. The outer framing (4-byte length prefix) is preserved. - -### 3. StreamsTcpTransport implements Transport abstraction - -**Decision:** New `StreamsTcpTransport : Transport` that uses Akka.Streams TCP for both server-side listening and client-side association. - -Server path (`Listen`): -- `Tcp.Bind(listenAddress)` → `Source` → materialize -- Each `IncomingConnection` produces a `StreamsAssociationHandle` -- `InboundPayload` events delivered to registered listener - -Client path (`Associate`): -- `Tcp.OutgoingConnection(remoteAddress)` → materialize → `StreamsAssociationHandle` -- `Write(ReadOnlyMemory)` queues data for the materialized flow - -**Rationale:** Reuses the Akka.Streams TCP infrastructure (which, after Spec 1, uses `Stream` + `Pipe` + `IStreamProvider`). Backpressure propagates naturally through Streams demand signaling. - -### 4. Configuration key preservation - -**Decision:** Parse all existing `akka.remote.dot-netty.tcp.*` HOCON keys into `StreamsTcpTransportSettings`. The config section name stays the same. Only the transport class reference changes. - -**Rationale:** Zero user config changes for the most common case. Users who have no DotNetty-specific code or config changes see a transparent upgrade. The default transport class in `reference.conf` changes from DotNetty's `TcpTransport` to `StreamsTcpTransport`. - -Keys preserved: -- `hostname`, `port`, `public-hostname`, `public-port` -- `send-buffer-size`, `receive-buffer-size` -- `maximum-frame-size` -- `backlog` -- `tcp-nodelay`, `tcp-keepalive`, `tcp-reuse-addr` -- `enable-ssl` + all `ssl.*` sub-keys (→ Spec 2 TLS) -- `connection-timeout` -- `batching.*` (flush batching settings — Spec 5 optimization) - -### 5. Remove DotNetty dependency - -**Decision:** Delete `src/core/Akka.Remote/Transport/DotNetty/` entirely. Remove all DotNetty NuGet packages from `Akka.Remote.csproj`. - -**Rationale:** Clean break. No adapter layer or backward compat shim for DotNetty. The new transport is the only transport. DotNetty-specific programmatic APIs (`DotNettyTransportSettings`, `DotNettySslSetup`) are replaced by their equivalents. - -### 6. Read path: Pipe → length-delimited frame parsing → deserialize - -**Decision:** The read side uses the Pipe from Spec 1. `ReadOnlySequence` from `PipeReader` is parsed for length-delimited frames. Each complete frame's payload `ReadOnlySequence` is sliced and passed directly to `serializer.Deserialize()`. - -``` -Read path: - stream.ReadAsync() → Pipe.Writer - Pipe.Reader.ReadAsync() → ReadOnlySequence - → read 4-byte length → check if enough bytes for full frame - → if yes: slice payload → parse PDU header → serializer.Deserialize(payloadSlice) - → if no: AdvanceTo(consumed, examined) → wait for more data -``` - -**Rationale:** `ReadOnlySequence` is what `PipeReader` returns and what `serializer.Deserialize()` takes. The frame parser just reads the length, checks bounds, and slices. Zero copy from Pipe buffer to serializer. The only copy is the one in the Akka.IO TCP actor (required for unbounded message lifetime — decided in Spec 1). - -## Risks / Trade-offs - -**[Breaking wire compatibility with v1.5]** → The binary PDU encoding is incompatible with the Protobuf PDU used in v1.5. This is a major version break. Mixed-version clusters are not supported during upgrade. All nodes must be upgraded together. Document clearly in release notes. - -**[Akka.Remote now depends on Akka.Streams]** → Currently `Akka.Remote` depends on `Akka` core only (plus DotNetty). The new transport adds a dependency on `Akka.Streams`. This is a new transitive dependency for all remoting users. Acceptable since Streams is a core module, not an external package. - -**[Flush batching needs tuning]** → DotNetty's `FlushConsolidationHandler` batches flushes for throughput. The new transport needs equivalent batching (consolidate multiple writes before calling `stream.FlushAsync()`). Deferred to Spec 5 (Performance) for benchmarking-driven optimization. - -**[FrameBufferWriter growth on SizeHint underestimate]** → If `serializer.SizeHint()` underestimates, `FrameBufferWriter` rents a larger array from `ArrayPool` and copies. This is a fallback path — `SizeHint` should be accurate for most messages. Benchmark to ensure the growth path doesn't regress. diff --git a/openspec/changes/streams-tcp-transport/proposal.md b/openspec/changes/streams-tcp-transport/proposal.md deleted file mode 100644 index e4b95c2395e..00000000000 --- a/openspec/changes/streams-tcp-transport/proposal.md +++ /dev/null @@ -1,39 +0,0 @@ -## Why - -DotNetty's `ByteBuf` is incompatible with `System.Memory`, making it a dead end for zero-copy serialization and transport. With Spec 1 delivering `System.Memory` + `Stream` + `Pipe` in Akka.IO and Spec 2 adding TLS via `IStreamProvider`, we can replace the DotNetty transport with one built on Akka.Streams TCP. This eliminates the DotNetty dependency entirely and enables end-to-end zero-copy: `IBufferWriter` flows from the serializer through framing to the socket in a single buffer with no intermediate copies. - -## What Changes - -- New `StreamsTcpTransport : Transport` implementation in Akka.Remote using Akka.Streams TCP -- **Wire-compatible**: same `[4-byte little-endian length][payload]` framing as DotNetty -- **Config-compatible**: all `akka.remote.dot-netty.tcp.*` HOCON keys continue to work (mapped to new transport settings) -- **Behaviorally compatible**: `AkkaProtocolTransport` adapter layer (handshake, heartbeats, association management) sits on top unchanged -- New `FrameBufferWriter : IBufferWriter` — lightweight wrapper over pooled `byte[]` with offset, enabling integrated framing + serialization in a single buffer write (reserve 4 bytes for length header, write PDU + payload, backfill length) -- Replace `AkkaPduProtobuffCodec` (Protobuf-based PDU encoding) with simple binary PDU encoding directly into `IBufferWriter` (serializerId as int32, manifest as length-prefixed UTF8, payload bytes) -- Remove DotNetty NuGet dependency from Akka.Remote -- **BREAKING**: `DotNettySslSetup` replaced by `TlsSetup` (Spec 2) -- **BREAKING**: DotNetty-specific programmatic APIs removed - -### What does NOT change - -- `AkkaProtocolTransport` adapter layer (handshake, heartbeats, acking, association state machine) -- The abstract `Transport` base class and `AssociationHandle` contract -- Akka.Remote's `Endpoint` / `EndpointWriter` / `EndpointReader` actor hierarchy (though `EndpointWriter` internals change to use `IBufferWriter`) -- Wire format on the network (same 4-byte length framing, same SerializerId + Manifest + Payload structure) -- All HOCON configuration keys (names preserved, implementation remapped) - -## Capabilities - -### New Capabilities - -- `streams-tcp-transport`: Akka.Streams-based TCP transport replacing DotNetty. Covers the `Transport` implementation, `AssociationHandle`, integrated framing + serialization via `FrameBufferWriter`, binary PDU encoding, configuration mapping from DotNetty HOCON, and DotNetty dependency removal. - -### Modified Capabilities - -## Impact - -- **Akka.Remote** (`src/core/Akka.Remote/`): New transport implementation, `FrameBufferWriter`, binary PDU codec. Remove `Transport/DotNetty/` directory entirely. Update `MessageSerializer.cs` and `Endpoint.cs` for `IBufferWriter` write path. -- **Configuration**: All `akka.remote.dot-netty.tcp.*` keys remapped to `StreamsTcpTransportSettings`. Default transport class changes from `TcpTransport` (DotNetty) to `StreamsTcpTransport`. -- **NuGet dependencies**: Remove `DotNetty.Transport`, `DotNetty.Codecs`, `DotNetty.Handlers`, `DotNetty.Common`, `DotNetty.Buffers`. Add dependency on `Akka.Streams` from `Akka.Remote`. -- **Test suites**: All Akka.Remote specs that don't directly reference DotNetty APIs must pass. DotNetty-specific tests removed. -- **Downstream**: Spec 4 (SerializerV2) `Serialize(IBufferWriter)` writes directly into `FrameBufferWriter`. Spec 5 (Performance) benchmarks this transport. diff --git a/openspec/changes/streams-tcp-transport/specs/streams-tcp-transport/spec.md b/openspec/changes/streams-tcp-transport/specs/streams-tcp-transport/spec.md deleted file mode 100644 index d11618e3aa8..00000000000 --- a/openspec/changes/streams-tcp-transport/specs/streams-tcp-transport/spec.md +++ /dev/null @@ -1,106 +0,0 @@ -## ADDED Requirements - -### Requirement: StreamsTcpTransport implements Transport abstraction -The system SHALL provide `StreamsTcpTransport : Transport` that implements the pluggable transport API using Akka.Streams TCP. - -#### Scenario: Server-side listen -- **WHEN** `StreamsTcpTransport.Listen()` is called -- **THEN** it SHALL bind a TCP listener via `Tcp.Bind()`, return the bound address and a listener that emits `InboundPayload` events for each received frame - -#### Scenario: Client-side associate -- **WHEN** `StreamsTcpTransport.Associate(remoteAddress)` is called -- **THEN** it SHALL connect via `Tcp.OutgoingConnection()` (using `IStreamProvider` with optional TLS), materialize the connection, and return a `StreamsAssociationHandle` - -#### Scenario: AssociationHandle write -- **WHEN** `StreamsAssociationHandle.Write(ReadOnlyMemory)` is called -- **THEN** it SHALL enqueue the data for framing and transmission through the materialized Akka.Streams flow - -#### Scenario: Transport shutdown -- **WHEN** `StreamsTcpTransport.Shutdown()` is called -- **THEN** it SHALL close all active associations and the listener, completing all materialized streams - -### Requirement: FrameBufferWriter for integrated framing and serialization -The system SHALL provide `FrameBufferWriter : IBufferWriter` that writes into a pooled `byte[]` with offset support, enabling the length header to be reserved upfront and backfilled after the payload is written. - -#### Scenario: Single-buffer frame construction -- **WHEN** a message is serialized for transport -- **THEN** the write path SHALL reserve 4 bytes for the length header, write PDU metadata and serialized payload via `IBufferWriter`, backfill the 4-byte length, and produce a single contiguous `ReadOnlyMemory` frame - -#### Scenario: Buffer growth on SizeHint underestimate -- **WHEN** the serializer writes more bytes than the initial buffer capacity -- **THEN** `FrameBufferWriter` SHALL rent a larger array from `ArrayPool.Shared`, copy existing data, and continue writing - -#### Scenario: Buffer returned to pool after send -- **WHEN** the frame has been written to the socket via `stream.WriteAsync()` -- **THEN** the pooled `byte[]` SHALL be returned to `ArrayPool.Shared` - -### Requirement: Binary PDU encoding -The system SHALL use a simple binary PDU format written directly to `IBufferWriter`, replacing the Protobuf `AkkaPduProtobuffCodec`. - -#### Scenario: Payload PDU encoding -- **WHEN** a user message is sent to a remote actor -- **THEN** the PDU SHALL be encoded as: `[4-byte frame length][1-byte PDU type = 0x01][4-byte serializerId][2-byte manifest length][manifest UTF8 bytes][serialized payload bytes]` - -#### Scenario: Control PDU encoding (heartbeat) -- **WHEN** a heartbeat is sent -- **THEN** the PDU SHALL be encoded as: `[4-byte frame length][1-byte PDU type = 0x04]` (5 bytes total) - -#### Scenario: Control PDU encoding (associate/disassociate) -- **WHEN** an associate or disassociate control message is sent -- **THEN** the PDU SHALL be encoded with the appropriate PDU type byte and protocol-specific fields - -### Requirement: Length-delimited frame parsing on read path -The system SHALL parse incoming data from `PipeReader` as length-delimited frames, slicing complete frames as `ReadOnlySequence` for deserialization. - -#### Scenario: Complete frame available -- **WHEN** `PipeReader.ReadAsync()` returns a buffer containing at least `4 + frameLength` bytes -- **THEN** the parser SHALL read the 4-byte length, slice the payload as `ReadOnlySequence`, parse the PDU header, and pass the payload to `serializer.Deserialize()` - -#### Scenario: Partial frame buffering -- **WHEN** `PipeReader.ReadAsync()` returns a buffer with fewer bytes than the frame length indicates -- **THEN** the parser SHALL call `AdvanceTo(consumed, examined)` to signal that more data is needed, and wait for the next `ReadAsync()` call - -#### Scenario: Maximum frame size enforcement -- **WHEN** the 4-byte length header indicates a frame larger than `maximum-frame-size` -- **THEN** the parser SHALL close the connection with an error (prevents memory exhaustion from malformed/malicious data) - -### Requirement: DotNetty HOCON configuration compatibility -All existing `akka.remote.dot-netty.tcp.*` HOCON configuration keys SHALL continue to work without modification, mapped to the new `StreamsTcpTransportSettings`. - -#### Scenario: Standard connection settings preserved -- **WHEN** `hostname`, `port`, `public-hostname`, `public-port`, `send-buffer-size`, `receive-buffer-size`, `maximum-frame-size`, `backlog`, `tcp-nodelay`, `tcp-keepalive`, `tcp-reuse-addr`, `connection-timeout` are configured -- **THEN** the new transport SHALL apply these settings identically to the DotNetty transport - -#### Scenario: TLS settings preserved -- **WHEN** `enable-ssl = true` with `ssl.*` sub-keys are configured -- **THEN** the transport SHALL use `TlsStreamProvider` (Spec 2) with settings parsed from the same HOCON keys - -#### Scenario: Default transport class updated -- **WHEN** no explicit transport class is configured -- **THEN** the `reference.conf` default SHALL specify `StreamsTcpTransport` instead of `DotNettyTransport` - -### Requirement: DotNetty dependency removed -The `Akka.Remote` project SHALL NOT reference any DotNetty NuGet packages. The `Transport/DotNetty/` directory SHALL be deleted. - -#### Scenario: Clean build without DotNetty -- **WHEN** the solution is built -- **THEN** no DotNetty assemblies SHALL be referenced or loaded - -#### Scenario: Akka.Remote depends on Akka.Streams -- **WHEN** the `Akka.Remote` project is built -- **THEN** it SHALL reference `Akka.Streams` as a project/package dependency - -### Requirement: Behavioral compatibility with existing transport -All Akka.Remote specs that do not directly reference DotNetty library APIs SHALL pass with the new transport without modification. - -#### Scenario: AkkaProtocolTransport works unchanged -- **WHEN** the `AkkaProtocolTransport` adapter is layered on top of `StreamsTcpTransport` -- **THEN** association handshake, heartbeats, sequence numbers, ACKs, and disassociation SHALL behave identically to the DotNetty-based transport - -#### Scenario: EndpointWriter sends messages -- **WHEN** `EndpointWriter` serializes a message and writes it to the transport -- **THEN** it SHALL use the `FrameBufferWriter` path (`IBufferWriter` → single buffer → transport write) - -#### Scenario: Two ActorSystems communicate -- **WHEN** two ActorSystems are configured with `StreamsTcpTransport` and remoting enabled -- **THEN** they SHALL be able to send messages, resolve remote actor refs, and maintain cluster membership identically to the DotNetty transport diff --git a/openspec/changes/streams-tcp-transport/tasks.md b/openspec/changes/streams-tcp-transport/tasks.md deleted file mode 100644 index 8909a9d5925..00000000000 --- a/openspec/changes/streams-tcp-transport/tasks.md +++ /dev/null @@ -1,68 +0,0 @@ -## 1. FrameBufferWriter - -- [ ] 1.1 Create `FrameBufferWriter : IBufferWriter` in `src/core/Akka.Remote/` — wraps pooled `byte[]` with start offset, implements `GetMemory`/`GetSpan`/`Advance`, supports growth via `ArrayPool` -- [ ] 1.2 Add length-header backfill method: writes total frame length into reserved first 4 bytes -- [ ] 1.3 Add `GetFrameMemory()` method that returns `ReadOnlyMemory` spanning the complete frame (length header + payload) -- [ ] 1.4 Add `Return()` method that returns the pooled `byte[]` to `ArrayPool` -- [ ] 1.5 Unit tests for FrameBufferWriter: basic write, growth, backfill, pool return - -## 2. Binary PDU Codec - -- [ ] 2.1 Create `BinaryPduCodec` in `src/core/Akka.Remote/` with `WritePdu(IBufferWriter, ...)` methods for payload, associate, disassociate, heartbeat PDU types -- [ ] 2.2 Create `ReadPdu(ReadOnlySequence)` methods that parse PDU type and extract fields -- [ ] 2.3 Define PDU type constants (0x01 payload, 0x02 associate, 0x03 disassociate, 0x04 heartbeat) -- [ ] 2.4 Implement payload PDU: serializerId (int32) + manifest (length-prefixed UTF8) + payload bytes -- [ ] 2.5 Implement control PDUs: associate (with handshake info), disassociate (with reason), heartbeat (minimal) -- [ ] 2.6 Unit tests: round-trip encode/decode for all PDU types, edge cases (empty manifest, large payloads, max frame size) - -## 3. StreamsTcpTransport Implementation - -- [ ] 3.1 Create `StreamsTcpTransport : Transport` in `src/core/Akka.Remote/Transport/Streams/` -- [ ] 3.2 Implement `Listen()`: use `Tcp.Bind()` to create listener, materialize `Source`, return bound address + association event listener -- [ ] 3.3 Implement `Associate(remoteAddress)`: use `Tcp.OutgoingConnection()` to connect, materialize flow, return `StreamsAssociationHandle` -- [ ] 3.4 Implement `Shutdown()`: close listener, close all active associations, complete materialized streams -- [ ] 3.5 Implement `IsResponsibleFor(Address)`: protocol check for "tcp" / "ssl.tcp" -- [ ] 3.6 Create `StreamsAssociationHandle : AssociationHandle` with `Write(ReadOnlyMemory)` that queues framed data to the materialized flow - -## 4. Integrated Write Path - -- [ ] 4.1 Modify `EndpointWriter` (or create new equivalent) to use `FrameBufferWriter` for the write path -- [ ] 4.2 Write path: rent buffer → reserve 4 bytes → `BinaryPduCodec.WritePdu(writer, ...)` → `serializer.Serialize(writer, msg)` → backfill length → `AssociationHandle.Write(frame)` -- [ ] 4.3 Buffer lifecycle: return pooled array to `ArrayPool` after `stream.WriteAsync()` completes - -## 5. Frame Parser (Read Path) - -- [ ] 5.1 Create `FrameParser` that reads length-delimited frames from `ReadOnlySequence` (from `PipeReader`) -- [ ] 5.2 Handle partial frames: return consumed/examined positions for `PipeReader.AdvanceTo()` -- [ ] 5.3 Maximum frame size enforcement: reject frames exceeding `maximum-frame-size`, close connection -- [ ] 5.4 Parse complete frame: read 4-byte length → slice payload → `BinaryPduCodec.ReadPdu()` → dispatch -- [ ] 5.5 Unit tests: complete frames, partial frames, oversized frames, multiple frames in one read - -## 6. Configuration - -- [ ] 6.1 Create `StreamsTcpTransportSettings` that parses all `akka.remote.dot-netty.tcp.*` HOCON keys -- [ ] 6.2 Map: `hostname`, `port`, `public-hostname`, `public-port`, `send-buffer-size`, `receive-buffer-size`, `maximum-frame-size`, `backlog`, `tcp-nodelay`, `tcp-keepalive`, `tcp-reuse-addr`, `connection-timeout` -- [ ] 6.3 Map TLS settings: `enable-ssl` + `ssl.*` → `TlsSettings` (Spec 2) -- [ ] 6.4 Update `reference.conf`: change default transport class to `StreamsTcpTransport` -- [ ] 6.5 Preserve `batching.*` settings for Spec 5 (flush batching optimization) - -## 7. Remove DotNetty - -- [ ] 7.1 Delete `src/core/Akka.Remote/Transport/DotNetty/` directory -- [ ] 7.2 Remove DotNetty NuGet packages from `Akka.Remote.csproj` (`DotNetty.Transport`, `DotNetty.Codecs`, `DotNetty.Handlers`, `DotNetty.Common`, `DotNetty.Buffers`) -- [ ] 7.3 Add `Akka.Streams` project reference to `Akka.Remote.csproj` -- [ ] 7.4 Remove `DotNettyTransportSettings`, `DotNettySslSetup`, and all DotNetty-specific types -- [ ] 7.5 Fix all compilation errors from DotNetty removal - -## 8. Testing - -- [ ] 8.1 Verify all existing Akka.Remote specs pass (except DotNetty-specific ones) -- [ ] 8.2 Test: two ActorSystems communicate via `StreamsTcpTransport` (basic remoting) -- [ ] 8.3 Test: TLS remoting via `StreamsTcpTransport` + `TlsStreamProvider` -- [ ] 8.4 Test: association handshake, heartbeat, disassociation (AkkaProtocolTransport layer) -- [ ] 8.5 Test: large message handling (near maximum-frame-size) -- [ ] 8.6 Test: oversized frame rejection -- [ ] 8.7 Test: connection failure and recovery -- [ ] 8.8 Test: cluster formation with multiple nodes using new transport -- [ ] 8.9 Remove DotNetty-specific test files -- [ ] 8.10 Run full test suite: `dotnet test -c Release` diff --git a/openspec/changes/transport-performance/design.md b/openspec/changes/transport-performance/design.md index e6b37274c3b..979b09810b3 100644 --- a/openspec/changes/transport-performance/design.md +++ b/openspec/changes/transport-performance/design.md @@ -1,53 +1,69 @@ ## Context -The RemotePingPong benchmark is the standard throughput measurement for Akka.NET remoting. It measures round-trip messages/second between two ActorSystems. DotNetty's `FlushConsolidationHandler` provides write batching — consolidating multiple `flush()` calls during read loops for reduced syscalls. The new Akka.Streams transport must match or exceed this. The `FrameBufferWriter` + `Stream` + `Pipe` pipeline has fewer copy points than DotNetty, which should provide a baseline advantage, but flush batching and dispatch tuning are needed to maximize throughput. +The current DotNetty baseline on `dev` is approximately 680K messages/sec in RemotePingPong on the recorded reference machine. Artery TCP must exceed that before it becomes the preferred remoting path. + +The performance target changed from a classic-remoting transport swap to a new Artery stack. This means the critical hot paths are: + +- Artery binary envelope encode/decode, +- `SerializerV2` payload serialization, +- generated MessagePack serializers, +- TCP framing and batching, +- bounded outbound queues, +- control/system stream behavior under ordinary traffic load, +- actor dispatch after envelope decode. ## Goals / Non-Goals **Goals:** -- Establish DotNetty baseline on current `dev` branch (messages/sec, latency percentiles, allocations) -- New transport MUST exceed DotNetty throughput -- Implement flush batching (coalesce multiple writes before `stream.FlushAsync()`) -- Tune `Pipe` backpressure thresholds for throughput -- Tune `ArrayPool` / `MemoryPool` buffer sizing -- Profile and optimize hot paths (allocation-free serialization, dispatch overhead) -- Continuous benchmark tracking as optimizations land + +- Preserve DotNetty RemotePingPong baseline for comparison. +- Prove Artery TCP exceeds DotNetty throughput. +- Prove Artery TCP allocates less than classic remoting in the common path. +- Prove bounded queues prevent unbounded memory growth. +- Tune batching without hiding unacceptable tail latency. +- Validate generated MessagePack serializers outperform V1 adapter fallback. **Non-Goals:** -- QUIC transport benchmarking (future) -- Benchmarking serialization in isolation (already validated in POC) -- Micro-benchmarking individual components (focus on end-to-end throughput) + +- QUIC benchmarking. +- TLS benchmarking before plaintext Artery TCP is stable. +- Optimizing classic remoting beyond compatibility fixes. +- Treating microbenchmarks as sufficient without RemotePingPong. ## Decisions -### 1. RemotePingPong as primary benchmark +### 1. RemotePingPong Remains The Primary Gate + +RemotePingPong measures the integrated path and remains the main acceptance test. -**Decision:** Use the existing RemotePingPong benchmark as the single most important metric. Measure messages/second, P50/P99 latency, and allocation rate (bytes/op). +Gate: Artery TCP must exceed the current DotNetty peak baseline on the same hardware and runtime configuration. -**Rationale:** This is the benchmark the community knows and uses. It measures the full pipeline: serialization → framing → transport → network → transport → deframing → deserialization. End-to-end numbers are what matter. +### 2. Add Envelope Codec Microbenchmarks -### 2. Flush batching in write task +Artery envelope encode/decode must be benchmarked against the classic protobuf `AkkaPduCodec` path. -**Decision:** The write-to-stream background task SHALL coalesce pending writes before calling `stream.FlushAsync()`. Instead of flushing after every `WriteAsync`, batch writes within a configurable window (e.g., flush after N writes or after a micro-delay if no more writes are pending). +Suggested gate: materially lower allocations and at least 3x throughput improvement in codec-only benchmark before deeper tuning. -**Rationale:** DotNetty's `FlushConsolidationHandler` does this — it defers flushes during read loop execution. Without batching, each `Write` → `FlushAsync` is a syscall. Batching reduces syscalls proportionally to batch size. +### 3. Add SerializerV2 Payload Benchmarks -### 3. Pipe threshold tuning +Generated MessagePack V2 serializers should be benchmarked against V1 adapter fallback. -**Decision:** Make `Pipe` `pauseWriterThreshold` and `resumeWriterThreshold` configurable via HOCON (`batching.pause-writer-threshold`, `batching.resume-writer-threshold`) and benchmark different values. +Suggested gate: generated serializers allocate less and are faster than V1 fallback for representative user messages. -**Rationale:** These thresholds control how much data buffers in the Pipe before backpressure kicks in. Too low = excessive pausing. Too high = memory bloat. The right values depend on message size and throughput characteristics. +### 4. Backpressure Is A Correctness Gate -### 4. Profile-driven optimization +Bounded queues must be validated under slow receivers and bursty senders. Throughput wins are not acceptable if queue memory can grow unbounded. -**Decision:** Use dotnet-trace / JetBrains profiler to identify allocation hot spots and CPU bottlenecks after the initial integration. Optimize based on data, not speculation. +### 5. Batching Is Configurable -**Rationale:** The architecture is designed for performance (zero-copy buffers, pooled arrays, sealed classes for devirtualization). Actual bottlenecks will emerge from profiling the integrated system. +Artery TCP should batch writes by bytes, message count, and/or a short latency budget. Defaults must be benchmark-driven, and low-latency profiles should be possible. ## Risks / Trade-offs -**[Flush batching adds latency]** → Batching trades latency for throughput. For latency-sensitive use cases, batching can be disabled or configured with aggressive flush thresholds. Measure both throughput and latency percentiles. +**Throughput vs latency**: batching improves throughput but can hurt P99 latency. Record both. + +**Benchmark drift**: run comparisons on the same hardware, OS, .NET runtime, and process settings. -**[Benchmark results are hardware-dependent]** → Run all comparisons on the same machine in the same session. Document hardware specs. Focus on relative improvement (%) rather than absolute numbers. +**Queue policy changes user experience**: bounded queues may expose missing flow control. Document behavior clearly. -**[Regression after optimization]** → Each optimization is a separate commit with before/after numbers. Revert if an optimization regresses other metrics. +**False wins from incomplete correctness**: do not benchmark final throughput before control stream and reliable system-message behavior are in place. diff --git a/openspec/changes/transport-performance/proposal.md b/openspec/changes/transport-performance/proposal.md index 390d705a691..4327e0a334b 100644 --- a/openspec/changes/transport-performance/proposal.md +++ b/openspec/changes/transport-performance/proposal.md @@ -1,27 +1,28 @@ ## Why -The Akka.NET 1.6 transport and serialization overhaul (Specs 1-4) replaces DotNetty with Akka.Streams TCP and introduces SerializerV2 with `IBufferWriter` / `ReadOnlySequence`. These changes must not only maintain but exceed the performance of the DotNetty-based transport. Performance validation using the existing RemotePingPong benchmark establishes a before/after baseline. Beyond meeting the baseline, targeted optimizations (flush batching, dispatch improvements, buffer pooling) can push throughput significantly higher. +Artery TCP is the Akka.NET 1.6 high-throughput remoting path. It must beat the current DotNetty baseline while preserving remoting correctness under backpressure, system-message delivery, association restart, and quarantine scenarios. + +The earlier performance plan targeted an Akka.Streams TCP replacement under classic remoting. That is no longer the target. Performance work now validates and tunes the Artery stack: envelope codec, `SerializerV2` payload path, TCP batching, bounded queues, and dispatch behavior. ## What Changes -- Establish DotNetty baseline using RemotePingPong benchmark on current `dev` branch -- Run identical benchmark on the new Akka.Streams transport (after Specs 1-4) -- New transport MUST exceed DotNetty throughput (messages/second) -- Identify and implement optimizations: flush batching, write coalescing, Pipe tuning, buffer pool sizing, dispatch improvements -- Continuous benchmarking as optimizations land +- Preserve the existing DotNetty RemotePingPong baseline as the comparison point. +- Add Artery TCP RemotePingPong benchmark runs. +- Add envelope codec microbenchmarks comparing Artery binary envelope vs classic protobuf PDU path. +- Add serializer microbenchmarks comparing generated MessagePack V2 serializers vs V1 adapter fallback. +- Add queue/backpressure benchmarks and slow-receiver tests. +- Tune batching, socket/Pipe thresholds, and buffer pooling for Artery TCP. +- Document throughput, latency, and allocation results. ## Capabilities ### New Capabilities -- `transport-benchmarks`: Performance benchmarking infrastructure for comparing DotNetty vs Akka.Streams transport. Covers RemotePingPong benchmark setup, baseline capture, regression detection, and optimization validation. - -### Modified Capabilities +- `artery-transport-benchmarks`: Performance benchmarking and tuning for Artery TCP, including RemotePingPong, envelope codec microbenchmarks, serializer comparisons, and backpressure validation. ## Impact -- **Benchmarks** (`src/benchmark/`): RemotePingPong benchmark with configurable transport selection -- **Akka.Remote**: Flush batching, write coalescing, Pipe threshold tuning in `StreamsTcpTransport` -- **Akka.IO**: Buffer pool sizing, Pipe `pauseWriterThreshold` / `resumeWriterThreshold` tuning -- **FrameBufferWriter**: `ArrayPool` sizing, growth strategy optimization -- **Documentation**: Performance comparison results published in release notes +- **Benchmarks** (`src/benchmark/`): add or update RemotePingPong configuration for Artery TCP. +- **Akka.Remote Artery**: tune write batching, bounded queues, buffer pooling, envelope codec, and dispatch. +- **Akka.Serialization.V2**: validate generated MessagePack throughput and allocation rates. +- **Documentation**: publish DotNetty vs Artery TCP performance results and tuning decisions. diff --git a/openspec/changes/transport-performance/specs/transport-benchmarks/spec.md b/openspec/changes/transport-performance/specs/transport-benchmarks/spec.md index 487363e3a33..69182c7ff35 100644 --- a/openspec/changes/transport-performance/specs/transport-benchmarks/spec.md +++ b/openspec/changes/transport-performance/specs/transport-benchmarks/spec.md @@ -1,52 +1,65 @@ ## ADDED Requirements -### Requirement: DotNetty baseline established -A DotNetty performance baseline SHALL be captured using the RemotePingPong benchmark on the `dev` branch before any transport changes, documenting messages/second, P50/P99 latency, and allocation rate. +### Requirement: DotNetty baseline preserved -#### Scenario: Baseline capture -- **WHEN** the RemotePingPong benchmark is run against the current DotNetty transport on reference hardware -- **THEN** the results SHALL be recorded with hardware specs, .NET version, OS, and benchmark configuration +A DotNetty performance baseline SHALL be available for comparison before Artery TCP is evaluated. -### Requirement: New transport exceeds DotNetty throughput -The Akka.Streams-based transport with SerializerV2 SHALL achieve higher messages/second than the DotNetty baseline on the RemotePingPong benchmark. +#### Scenario: Baseline recorded +- **WHEN** the baseline is referenced +- **THEN** it SHALL include messages/sec, latency percentiles, allocation rate, hardware, OS, .NET version, and GC mode + +### Requirement: Artery TCP exceeds DotNetty throughput + +Artery TCP SHALL exceed the recorded DotNetty RemotePingPong throughput baseline on comparable hardware. #### Scenario: Throughput comparison -- **WHEN** the RemotePingPong benchmark is run against the new transport on the same hardware -- **THEN** messages/second SHALL exceed the DotNetty baseline +- **WHEN** RemotePingPong is run against Artery TCP +- **THEN** peak messages/sec SHALL exceed the DotNetty baseline + +#### Scenario: Allocation comparison +- **WHEN** allocation rate is measured for Artery TCP +- **THEN** it SHALL be lower than the classic DotNetty remoting path for representative workloads + +### Requirement: Envelope codec benchmarked + +The Artery binary envelope codec SHALL be benchmarked against the classic protobuf PDU codec. + +#### Scenario: Codec throughput comparison +- **WHEN** codec-only benchmarks are run +- **THEN** Artery envelope encode/decode throughput SHALL exceed classic protobuf PDU throughput + +#### Scenario: Codec allocation comparison +- **WHEN** codec-only allocations are measured +- **THEN** Artery envelope encode/decode SHALL allocate fewer bytes than the classic protobuf PDU path + +### Requirement: SerializerV2 sourcegen benchmarked -#### Scenario: Allocation rate comparison -- **WHEN** allocation rate (bytes/op) is measured for both transports -- **THEN** the new transport SHALL allocate fewer bytes per operation than DotNetty +Generated MessagePack V2 serializers SHALL be benchmarked against V1 adapter fallback. -### Requirement: Flush batching configurable -The write path SHALL support configurable flush batching to coalesce multiple writes before calling `stream.FlushAsync()`. +#### Scenario: Generated serializer faster than fallback +- **WHEN** representative messages are serialized and deserialized +- **THEN** generated MessagePack V2 serializers SHALL be faster and allocate less than V1 adapter fallback -#### Scenario: Batched flush reduces syscalls -- **WHEN** multiple `Write` commands are pending within a batch window -- **THEN** they SHALL be written to the stream and flushed together in a single `FlushAsync()` call +### Requirement: Bounded queue behavior validated -#### Scenario: Flush batching configurable via HOCON -- **WHEN** `akka.remote.dot-netty.tcp.batching.flush-count` or `batching.flush-interval` is configured -- **THEN** the transport SHALL batch writes according to the configured thresholds +Artery TCP outbound queues SHALL remain bounded under load. -#### Scenario: Flush batching disabled -- **WHEN** flush batching is disabled (configured to flush after every write) -- **THEN** each `WriteAsync` SHALL be followed immediately by `FlushAsync` +#### Scenario: Slow receiver +- **WHEN** a sender outpaces a receiver +- **THEN** outbound queues SHALL not grow without bound -### Requirement: Pipe backpressure thresholds tunable -The `Pipe` `pauseWriterThreshold` and `resumeWriterThreshold` SHALL be configurable and benchmarked at various settings. +#### Scenario: Control traffic not starved +- **WHEN** ordinary traffic is saturated +- **THEN** control and system-message traffic SHALL still make progress or fail according to documented policy -#### Scenario: Threshold configuration -- **WHEN** `batching.pause-writer-threshold` and `batching.resume-writer-threshold` are configured in HOCON -- **THEN** the `Pipe` SHALL use these values for backpressure control +### Requirement: Batching configurable and measured -### Requirement: Continuous benchmark tracking -Each optimization commit SHALL include before/after benchmark numbers to track incremental progress and detect regressions. +Artery TCP SHALL support benchmarked batching policies. -#### Scenario: Optimization validated -- **WHEN** an optimization is implemented (flush batching, pool tuning, dispatch improvement) -- **THEN** the RemotePingPong benchmark SHALL be run and results compared to the previous iteration +#### Scenario: Throughput profile +- **WHEN** throughput-oriented batching is enabled +- **THEN** throughput SHALL improve without breaking queue or control-stream correctness -#### Scenario: Regression detected -- **WHEN** a change causes messages/second to drop below the previous iteration -- **THEN** the change SHALL be investigated and either fixed or reverted +#### Scenario: Low-latency profile +- **WHEN** low-latency batching is configured +- **THEN** flush behavior SHALL prioritize lower latency over maximum throughput diff --git a/openspec/changes/transport-performance/tasks.md b/openspec/changes/transport-performance/tasks.md index 28c9b7d5eb8..e19ec6c46cd 100644 --- a/openspec/changes/transport-performance/tasks.md +++ b/openspec/changes/transport-performance/tasks.md @@ -1,44 +1,63 @@ -## 1. Baseline Capture +## 1. Baseline Preservation -- [ ] 1.1 Run RemotePingPong benchmark against current `dev` branch (DotNetty transport) -- [ ] 1.2 Record: messages/second, P50/P99 latency, allocation rate (bytes/op) -- [ ] 1.3 Document hardware specs, .NET version, OS, benchmark configuration -- [ ] 1.4 Commit baseline results to repo for future comparison +- [ ] 1.1 Verify existing DotNetty RemotePingPong baseline is documented +- [ ] 1.2 Re-run DotNetty baseline on current reference hardware if hardware/runtime changed +- [ ] 1.3 Record messages/sec, P50/P95/P99 latency, allocation rate, hardware, OS, .NET version, GC mode -## 2. Initial Comparison +## 2. Artery RemotePingPong -- [ ] 2.1 Run RemotePingPong benchmark against new transport (after Specs 1-4 integrated) -- [ ] 2.2 Compare messages/second, latency, and allocation rate vs DotNetty baseline -- [ ] 2.3 Profile with dotnet-trace or JetBrains profiler to identify hot spots -- [ ] 2.4 Document initial comparison results +- [ ] 2.1 Add benchmark configuration for Artery TCP +- [ ] 2.2 Run RemotePingPong with 1, 5, 10, 15, 20, 25, and 30 clients +- [ ] 2.3 Compare throughput against DotNetty baseline +- [ ] 2.4 Compare allocation rate against DotNetty baseline +- [ ] 2.5 Compare latency percentiles against DotNetty baseline -## 3. Flush Batching +## 3. Envelope Codec Benchmarks -- [ ] 3.1 Implement flush batching in write-to-stream task: coalesce pending writes before `FlushAsync()` -- [ ] 3.2 Add HOCON configuration: `batching.flush-count` (max writes before flush), `batching.flush-interval` (max time before flush) -- [ ] 3.3 Benchmark: compare unbatched vs batched flush at various thresholds -- [ ] 3.4 Select optimal default values based on benchmark results +- [ ] 3.1 Add microbenchmark for classic protobuf PDU encode/decode +- [ ] 3.2 Add microbenchmark for Artery envelope encode/decode +- [ ] 3.3 Benchmark empty, small, medium, and near-max-frame payloads +- [ ] 3.4 Benchmark multi-segment decode input +- [ ] 3.5 Record throughput and allocations -## 4. Pipe and Buffer Tuning +## 4. Serializer Benchmarks -- [ ] 4.1 Benchmark Pipe `pauseWriterThreshold` / `resumeWriterThreshold` at various settings (64KB, 256KB, 1MB) -- [ ] 4.2 Benchmark `FrameBufferWriter` initial buffer size vs `SizeHint` accuracy -- [ ] 4.3 Benchmark `MemoryPool` vs `ArrayPool` for read path buffer allocation -- [ ] 4.4 Make thresholds configurable via HOCON -- [ ] 4.5 Select optimal defaults based on results +- [ ] 4.1 Benchmark V1 adapter fallback serialization/deserialization +- [ ] 4.2 Benchmark generated MessagePack V2 serialization/deserialization +- [ ] 4.3 Benchmark representative actor messages +- [ ] 4.4 Benchmark persistence-style event payloads +- [ ] 4.5 Record size hint accuracy and buffer growth frequency -## 5. Dispatch and Hot Path Optimization +## 5. Batching And Flush Policy -- [ ] 5.1 Profile serialization hot path: ensure zero allocations for common message types -- [ ] 5.2 Profile framing hot path: ensure `FrameBufferWriter` doesn't grow for typical messages -- [ ] 5.3 Profile read path: ensure frame parsing from `ReadOnlySequence` is allocation-free -- [ ] 5.4 Investigate write coalescing: can multiple small messages share a single frame buffer? -- [ ] 5.5 Investigate dispatch overhead: is the actor mailbox the bottleneck for high-throughput scenarios? +- [ ] 5.1 Implement configurable Artery TCP write batching if not already present +- [ ] 5.2 Benchmark flush by message count +- [ ] 5.3 Benchmark flush by byte threshold +- [ ] 5.4 Benchmark flush by latency budget +- [ ] 5.5 Select default throughput profile +- [ ] 5.6 Validate low-latency profile -## 6. Final Validation +## 6. Queue And Backpressure Validation -- [ ] 6.1 Run RemotePingPong benchmark with all optimizations applied -- [ ] 6.2 Confirm new transport exceeds DotNetty throughput -- [ ] 6.3 Confirm allocation rate is lower than DotNetty -- [ ] 6.4 Document final comparison results and optimization decisions -- [ ] 6.5 Add benchmark configuration to CI for regression detection +- [ ] 6.1 Benchmark slow receiver with fast sender +- [ ] 6.2 Verify ordinary queue stays bounded +- [ ] 6.3 Verify control/system traffic is not starved by ordinary traffic +- [ ] 6.4 Verify ordinary queue overflow behavior is deterministic +- [ ] 6.5 Verify control queue overflow triggers severe failure/quarantine behavior + +## 7. Profiling And Optimization + +- [ ] 7.1 Profile Artery RemotePingPong with dotnet-trace or JetBrains profiler +- [ ] 7.2 Identify allocation hot spots +- [ ] 7.3 Identify CPU hot spots +- [ ] 7.4 Optimize envelope parsing/writing based on profiling +- [ ] 7.5 Optimize dispatch path based on profiling +- [ ] 7.6 Re-run benchmarks after each optimization + +## 8. Final Validation + +- [ ] 8.1 Confirm Artery TCP exceeds 680K msgs/sec baseline on reference hardware +- [ ] 8.2 Confirm allocation rate is lower than DotNetty/classic path +- [ ] 8.3 Confirm queue/backpressure tests pass +- [ ] 8.4 Confirm control/system traffic correctness under load +- [ ] 8.5 Document final results and recommended defaults diff --git a/src/benchmark/Akka.Benchmarks/DData/SerializerLwwDictionaryBenchmarks.cs b/src/benchmark/Akka.Benchmarks/DData/SerializerLwwDictionaryBenchmarks.cs index 8b5bd60d277..10628c76be5 100644 --- a/src/benchmark/Akka.Benchmarks/DData/SerializerLwwDictionaryBenchmarks.cs +++ b/src/benchmark/Akka.Benchmarks/DData/SerializerLwwDictionaryBenchmarks.cs @@ -15,6 +15,7 @@ using Akka.Configuration; using Akka.DistributedData; using Akka.DistributedData.Serialization; +using Akka.Serialization; using BenchmarkDotNet.Attributes; namespace Akka.Benchmarks.DData; @@ -37,7 +38,7 @@ public class SerializerLwwDictionaryBenchmarks private UniqueAddress[] _nodes; private object _c1; private ActorSystem sys; - private ReplicatedDataSerializer ser; + private Serializer ser; private byte[] _c1Ser; private string _c1Manifest; @@ -61,7 +62,7 @@ public void SetupSystem() } }"); sys = ActorSystem.Create("rddsb", conf); - ser = (ReplicatedDataSerializer)sys.Serialization.FindSerializerForType( + ser = sys.Serialization.FindSerializerForType( typeof(IReplicatedDataSerialization)); _c1Ser = ser.ToBinary(_c1); _c1Manifest = ser.Manifest(_c1); diff --git a/src/benchmark/Akka.Benchmarks/DData/SerializerORDictionaryBenchmarks.cs b/src/benchmark/Akka.Benchmarks/DData/SerializerORDictionaryBenchmarks.cs index 44a88cfa8c1..19e21174809 100644 --- a/src/benchmark/Akka.Benchmarks/DData/SerializerORDictionaryBenchmarks.cs +++ b/src/benchmark/Akka.Benchmarks/DData/SerializerORDictionaryBenchmarks.cs @@ -13,6 +13,7 @@ using Akka.Configuration; using Akka.DistributedData; using Akka.DistributedData.Serialization; +using Akka.Serialization; using BenchmarkDotNet.Attributes; namespace Akka.Benchmarks.DData; @@ -30,7 +31,7 @@ public class SerializerORDictionaryBenchmarks private ORDictionary> _c1; private ORSet _elements; private ActorSystem sys; - private ReplicatedDataSerializer ser; + private Serializer ser; private byte[] _c1Ser; private string _c1Manifest; @@ -70,7 +71,7 @@ public void SetupSystem() } }"); sys = ActorSystem.Create("rddsb", conf); - ser = (ReplicatedDataSerializer)sys.Serialization.FindSerializerForType( + ser = sys.Serialization.FindSerializerForType( typeof(IReplicatedDataSerialization)); _c1Ser = ser.ToBinary(_c1); _c1Manifest = ser.Manifest(_c1); diff --git a/src/benchmark/Akka.Benchmarks/DData/SerializerORSetBenchmarks.cs b/src/benchmark/Akka.Benchmarks/DData/SerializerORSetBenchmarks.cs index 83fe251acb6..f0e820af0f3 100644 --- a/src/benchmark/Akka.Benchmarks/DData/SerializerORSetBenchmarks.cs +++ b/src/benchmark/Akka.Benchmarks/DData/SerializerORSetBenchmarks.cs @@ -30,7 +30,7 @@ public class SerializerORSetBenchmarks public int NumNodes; private ActorSystem sys; - private ReplicatedDataSerializer ser; + private Serializer ser; private UniqueAddress[] _nodes; private RDDBenchTypes.TestVal[] _elements; private ORSet> _c1; @@ -70,7 +70,7 @@ public void SetupSystem() } }"); sys = ActorSystem.Create("rddsb", conf); - ser = (ReplicatedDataSerializer)sys.Serialization.FindSerializerForType( + ser = sys.Serialization.FindSerializerForType( typeof(IReplicatedDataSerialization)); _c1Ser = ser.ToBinary(_c1); _c1Manifest = ser.Manifest(_c1); diff --git a/src/contrib/cluster/Akka.Cluster.Sharding.Tests/ClusterShardingMessageSerializerSpec.cs b/src/contrib/cluster/Akka.Cluster.Sharding.Tests/ClusterShardingMessageSerializerSpec.cs index eb096c3b0b0..99a9860a2cf 100644 --- a/src/contrib/cluster/Akka.Cluster.Sharding.Tests/ClusterShardingMessageSerializerSpec.cs +++ b/src/contrib/cluster/Akka.Cluster.Sharding.Tests/ClusterShardingMessageSerializerSpec.cs @@ -47,7 +47,7 @@ public ClusterShardingMessageSerializerSpec() : base(SpecConfig) private void CheckSerialization(object obj) { - var serializer = (SerializerWithStringManifest)Sys.Serialization.FindSerializerFor(obj); + var serializer = Sys.Serialization.FindSerializerFor(obj); serializer.Should().BeOfType(); var blob = serializer.ToBinary(obj); var reference = serializer.FromBinary(blob, serializer.Manifest(obj)); @@ -133,7 +133,7 @@ public void ClusterShardingMessageSerializer_must_be_able_to_deserialize_old_ent message1.EntityId = "e1"; var blob = message1.ToByteArray(); - var serializer = (SerializerWithStringManifest)Sys.Serialization.FindSerializerForType(typeof(EventSourcedRememberEntitiesShardStore.EntitiesStarted)); + var serializer = Sys.Serialization.FindSerializerForType(typeof(EventSourcedRememberEntitiesShardStore.EntitiesStarted)); serializer.Should().BeOfType(); var reference = serializer.FromBinary(blob, "CB"); diff --git a/src/contrib/cluster/Akka.Cluster.Tools.Tests/ClusterClient/ClusterClientMessageSerializerSpec.cs b/src/contrib/cluster/Akka.Cluster.Tools.Tests/ClusterClient/ClusterClientMessageSerializerSpec.cs index 57f496e125f..3289e583824 100644 --- a/src/contrib/cluster/Akka.Cluster.Tools.Tests/ClusterClient/ClusterClientMessageSerializerSpec.cs +++ b/src/contrib/cluster/Akka.Cluster.Tools.Tests/ClusterClient/ClusterClientMessageSerializerSpec.cs @@ -12,6 +12,7 @@ using Akka.Configuration; using Akka.Serialization; using Akka.TestKit; +using FluentAssertions; using Xunit; namespace Akka.Cluster.Tools.Tests.ClusterClient @@ -69,7 +70,8 @@ public void Can_serialize_ReceptionistShutdown() private T AssertAndReturn(T message) { - var serializer = (ClusterClientMessageSerializer)Sys.Serialization.FindSerializerFor(message); + var serializer = Sys.Serialization.FindSerializerFor(message); + serializer.Should().BeOfType(); var serialized = serializer.ToBinary(message); return (T)serializer.FromBinary(serialized, serializer.Manifest(message)); } @@ -81,4 +83,3 @@ private void AssertEqual(T message) } } } - diff --git a/src/contrib/cluster/Akka.Cluster.Tools.Tests/ClusterClient/ClusterClientSerializerSpec.cs b/src/contrib/cluster/Akka.Cluster.Tools.Tests/ClusterClient/ClusterClientSerializerSpec.cs index 6736684cf3a..f83ae884d06 100644 --- a/src/contrib/cluster/Akka.Cluster.Tools.Tests/ClusterClient/ClusterClientSerializerSpec.cs +++ b/src/contrib/cluster/Akka.Cluster.Tools.Tests/ClusterClient/ClusterClientSerializerSpec.cs @@ -126,9 +126,8 @@ public void SerializeMessageTypeType(IClusterClientProtocolMessage message) } else { - var manifestSerializer = (SerializerWithStringManifest)serializer; - var manifest = manifestSerializer.Manifest(message); - deserialized = (IClusterClientProtocolMessage) manifestSerializer.FromBinary(serialized, manifest); + var manifest = serializer.Manifest(message); + deserialized = (IClusterClientProtocolMessage) serializer.FromBinary(serialized, manifest); } deserialized.Should().Be(message); diff --git a/src/contrib/cluster/Akka.Cluster.Tools.Tests/PublishSubscribe/DistributedPubSubMessageSerializerSpec.cs b/src/contrib/cluster/Akka.Cluster.Tools.Tests/PublishSubscribe/DistributedPubSubMessageSerializerSpec.cs index 0b04797872a..a0a2f86bc2c 100644 --- a/src/contrib/cluster/Akka.Cluster.Tools.Tests/PublishSubscribe/DistributedPubSubMessageSerializerSpec.cs +++ b/src/contrib/cluster/Akka.Cluster.Tools.Tests/PublishSubscribe/DistributedPubSubMessageSerializerSpec.cs @@ -108,7 +108,7 @@ public void Can_serialize_SendToOneSubscriber() private T AssertAndReturn(T message) { - var serializer = (SerializerWithStringManifest)Sys.Serialization.FindSerializerFor(message); + var serializer = Sys.Serialization.FindSerializerFor(message); serializer.Should().BeOfType(); var serialized = serializer.ToBinary(message); return (T)serializer.FromBinary(serialized, serializer.Manifest(message)); diff --git a/src/contrib/cluster/Akka.Cluster.Tools.Tests/Singleton/ClusterSingletonMessageSerializerSpec.cs b/src/contrib/cluster/Akka.Cluster.Tools.Tests/Singleton/ClusterSingletonMessageSerializerSpec.cs index 454002f8861..6faeef97028 100644 --- a/src/contrib/cluster/Akka.Cluster.Tools.Tests/Singleton/ClusterSingletonMessageSerializerSpec.cs +++ b/src/contrib/cluster/Akka.Cluster.Tools.Tests/Singleton/ClusterSingletonMessageSerializerSpec.cs @@ -6,6 +6,7 @@ //----------------------------------------------------------------------- using Akka.Cluster.Tools.Singleton; +using Akka.Cluster.Tools.Singleton.Serialization; using Akka.Configuration; using Akka.Serialization; using Akka.TestKit; @@ -51,7 +52,8 @@ public void Can_serialize_TakeOverFromMe() private T AssertAndReturn(T message) { - var serializer = (SerializerWithStringManifest)Sys.Serialization.FindSerializerFor(message); + var serializer = Sys.Serialization.FindSerializerFor(message); + Assert.IsType(serializer); var serialized = serializer.ToBinary(message); return (T)serializer.FromBinary(serialized, serializer.Manifest(message)); } diff --git a/src/contrib/cluster/Akka.DistributedData.LightningDB/LmdbDurableStore.cs b/src/contrib/cluster/Akka.DistributedData.LightningDB/LmdbDurableStore.cs index 8661538208b..14ae26d82bc 100644 --- a/src/contrib/cluster/Akka.DistributedData.LightningDB/LmdbDurableStore.cs +++ b/src/contrib/cluster/Akka.DistributedData.LightningDB/LmdbDurableStore.cs @@ -50,7 +50,7 @@ private WriteBehind() { } private readonly Config _config; private readonly Akka.Serialization.Serialization _serialization; - private readonly SerializerWithStringManifest _serializer; + private readonly Serializer _serializer; private readonly string _manifest; private readonly long _mapSize; @@ -70,8 +70,8 @@ public LmdbDurableStore(Config config) _log = Context.GetLogger(); _serialization = Context.System.Serialization; - _serializer = (SerializerWithStringManifest) _serialization.FindSerializerForType(typeof(DurableDataEnvelope)); - _manifest = _serializer.Manifest(new DurableDataEnvelope(GCounter.Empty)); + _serializer = _serialization.FindSerializerForType(typeof(DurableDataEnvelope)); + _manifest = Akka.Serialization.Serialization.ManifestFor(_serializer, new DurableDataEnvelope(GCounter.Empty)); var useWriteBehind = _config.GetString("write-behind-interval", "").ToLowerInvariant(); _writeBehindInterval = diff --git a/src/contrib/serializers/Akka.Serialization.Hyperion.Tests/HyperionConfigTests.cs b/src/contrib/serializers/Akka.Serialization.Hyperion.Tests/HyperionConfigTests.cs index d5abfc38c51..4f80071ce3f 100644 --- a/src/contrib/serializers/Akka.Serialization.Hyperion.Tests/HyperionConfigTests.cs +++ b/src/contrib/serializers/Akka.Serialization.Hyperion.Tests/HyperionConfigTests.cs @@ -13,6 +13,7 @@ using System.Threading.Tasks; using Akka.Actor; using Akka.Configuration; +using Akka.Serialization; using FluentAssertions; using Hyperion; using Hyperion.Internal; @@ -22,6 +23,11 @@ namespace Akka.Serialization.Hyperion.Tests { public class HyperionConfigTests { + private static HyperionSerializer GetHyperionSerializer(ActorSystem system) + { + return system.Serialization.FindSerializerForType(typeof(object)).Should().BeOfType().Subject; + } + [Fact] public void Hyperion_serializer_should_have_correct_defaults() { @@ -35,7 +41,7 @@ public void Hyperion_serializer_should_have_correct_defaults() "); using (var system = ActorSystem.Create(nameof(HyperionConfigTests), config)) { - var serializer = (HyperionSerializer)system.Serialization.FindSerializerForType(typeof(object)); + var serializer = GetHyperionSerializer(system); Assert.True(serializer.Settings.VersionTolerance); Assert.True(serializer.Settings.PreserveObjectReferences); Assert.Equal("NoKnownTypes", serializer.Settings.KnownTypesProvider.Name); @@ -63,7 +69,7 @@ public void Hyperion_serializer_should_allow_to_setup_custom_flags() "); using (var system = ActorSystem.Create(nameof(HyperionConfigTests), config)) { - var serializer = (HyperionSerializer)system.Serialization.FindSerializerForType(typeof(object)); + var serializer = GetHyperionSerializer(system); Assert.False(serializer.Settings.VersionTolerance); Assert.False(serializer.Settings.PreserveObjectReferences); Assert.Equal("NoKnownTypes", serializer.Settings.KnownTypesProvider.Name); @@ -95,7 +101,7 @@ public void TypeFilter_defined_in_config_should_filter_serializer_properly(objec "); using (var system = ActorSystem.Create(nameof(HyperionConfigTests), config)) { - var deserializer = (HyperionSerializer)system.Serialization.FindSerializerForType(typeof(object)); + var deserializer = GetHyperionSerializer(system); var serializer = new HyperionSerializer(null, deserializer.Settings.WithDisallowUnsafeType(false)); ((TypeFilter)deserializer.Settings.TypeFilter).FilteredTypes.Count.Should().Be(2); @@ -132,7 +138,7 @@ public void Hyperion_serializer_should_allow_to_setup_custom_types_provider_with "); using (var system = ActorSystem.Create(nameof(HyperionConfigTests), config)) { - var serializer = (HyperionSerializer)system.Serialization.FindSerializerForType(typeof(object)); + var serializer = GetHyperionSerializer(system); Assert.True(serializer.Settings.VersionTolerance); Assert.True(serializer.Settings.PreserveObjectReferences); Assert.Equal(typeof(DummyTypesProviderWithDefaultCtor), serializer.Settings.KnownTypesProvider); @@ -156,7 +162,7 @@ public void Hyperion_serializer_should_allow_to_setup_custom_types_provider_with "); using (var system = ActorSystem.Create(nameof(HyperionConfigTests), config)) { - var serializer = (HyperionSerializer)system.Serialization.FindSerializerForType(typeof(object)); + var serializer = GetHyperionSerializer(system); Assert.True(serializer.Settings.VersionTolerance); Assert.True(serializer.Settings.PreserveObjectReferences); Assert.Equal(typeof(DummyTypesProvider), serializer.Settings.KnownTypesProvider); @@ -199,7 +205,7 @@ public void Hyperion_serializer_should_read_cross_platform_package_name_override "); using (var system = ActorSystem.Create(nameof(HyperionConfigTests), config)) { - var serializer = (HyperionSerializer)system.Serialization.FindSerializerForType(typeof(object)); + var serializer = GetHyperionSerializer(system); var overrides = serializer.Settings.PackageNameOverrides.ToList(); Assert.NotEmpty(overrides); var @override = overrides[0]; @@ -237,7 +243,7 @@ public void Hyperion_serializer_should_allow_to_setup_surrogates() "); using (var system = ActorSystem.Create(nameof(HyperionConfigTests), config)) { - var serializer = (HyperionSerializer)system.Serialization.FindSerializerForType(typeof(object)); + var serializer = GetHyperionSerializer(system); FooHyperionSurrogate.Surrogated.Clear(); var expected = new Foo("bar"); diff --git a/src/contrib/serializers/Akka.Serialization.Hyperion.Tests/HyperionSerializerSetupSpec.cs b/src/contrib/serializers/Akka.Serialization.Hyperion.Tests/HyperionSerializerSetupSpec.cs index 7fccf03fc5e..aafd15580f1 100644 --- a/src/contrib/serializers/Akka.Serialization.Hyperion.Tests/HyperionSerializerSetupSpec.cs +++ b/src/contrib/serializers/Akka.Serialization.Hyperion.Tests/HyperionSerializerSetupSpec.cs @@ -19,6 +19,7 @@ using Akka.Actor; using Akka.Actor.Setup; using Akka.Configuration; +using Akka.Serialization; using Akka.TestKit; using Xunit; using FluentAssertions; @@ -107,7 +108,7 @@ public void SetupApplicationTest() .WithKnownTypeProvider()); var sys = ActorSystem.Create("test", actorSetup); - var serializer = (HyperionSerializer) sys.Serialization.FindSerializerForType(typeof(object)); + var serializer = sys.Serialization.FindSerializerForType(typeof(object)).Should().BeOfType().Subject; var settings = serializer.Settings; settings.Surrogates.Should().BeEquivalentTo(surrogate); diff --git a/src/core/Akka.API.Tests/verify/CoreAPISpec.ApproveCore.DotNet.verified.txt b/src/core/Akka.API.Tests/verify/CoreAPISpec.ApproveCore.DotNet.verified.txt index 7a4bc33d6c7..7ef5454a7ea 100644 --- a/src/core/Akka.API.Tests/verify/CoreAPISpec.ApproveCore.DotNet.verified.txt +++ b/src/core/Akka.API.Tests/verify/CoreAPISpec.ApproveCore.DotNet.verified.txt @@ -5150,11 +5150,15 @@ namespace Akka.Routing } namespace Akka.Serialization { - public class ByteArraySerializer : Akka.Serialization.Serializer + public class ByteArraySerializer : Akka.Serialization.SerializerV2 { public ByteArraySerializer(Akka.Actor.ExtendedActorSystem system) { } - public override bool IncludeManifest { get; } + public override object Deserialize(System.Buffers.ReadOnlySequence bytes, string manifest) { } public override object FromBinary(byte[] bytes, System.Type type) { } + public override object FromBinary(byte[] bytes, string manifest) { } + public override string Manifest(object obj) { } + public override int Serialize(object obj, System.Buffers.IBufferWriter writer) { } + public override int SizeHint(object obj) { } public override byte[] ToBinary(object obj) { } } [Akka.Annotations.InternalApi] @@ -5210,9 +5214,13 @@ namespace Akka.Serialization public Serialization(Akka.Actor.ExtendedActorSystem system) { } public Akka.Actor.ExtendedActorSystem System { get; } public void AddSerializationMap(System.Type type, Akka.Serialization.Serializer serializer) { } + public void AddSerializationMap(System.Type type, Akka.Serialization.SerializerV2 serializer) { } [System.Obsolete("No longer supported. Use the AddSerializer(name, serializer) overload instead.", true)] public void AddSerializer(Akka.Serialization.Serializer serializer) { } + [System.Obsolete("No longer supported. Use the AddSerializer(name, serializer) overload instead.", true)] + public void AddSerializer(Akka.Serialization.SerializerV2 serializer) { } public void AddSerializer(string name, Akka.Serialization.Serializer serializer) { } + public void AddSerializer(string name, Akka.Serialization.SerializerV2 serializer) { } public object Deserialize(byte[] bytes, int serializerId, System.Type type) { } public object Deserialize(byte[] bytes, int serializerId, string manifest) { } public Akka.Actor.IActorRef DeserializeActorRef(string path) { } @@ -5221,6 +5229,7 @@ namespace Akka.Serialization public byte[] Serialize(object o) { } public static Akka.Serialization.Information GetCurrentTransportInformation() { } public static string ManifestFor(Akka.Serialization.Serializer s, object msg) { } + public static string ManifestFor(Akka.Serialization.SerializerV2 s, object msg) { } public static string SerializedActorPath(Akka.Actor.IActorRef actorRef) { } public static T WithTransport(Akka.Actor.ExtendedActorSystem system, System.Func action) { } [System.Obsolete("Obsolete. Use the SerializeWithTransport(ExtendedActorSystem) method instead.")] @@ -5239,7 +5248,9 @@ namespace Akka.Serialization public virtual int Identifier { get; } public abstract bool IncludeManifest { get; } public abstract object FromBinary(byte[] bytes, System.Type type); + public virtual object FromBinary(byte[] bytes, string manifest) { } public T FromBinary(byte[] bytes) { } + public virtual string Manifest(object obj) { } public abstract byte[] ToBinary(object obj); public byte[] ToBinaryWithAddress(Akka.Actor.Address address, object obj) { } } @@ -5249,19 +5260,46 @@ namespace Akka.Serialization public Akka.Serialization.Serializer Serializer { get; } public System.Collections.Immutable.ImmutableHashSet UseFor { get; } public static Akka.Serialization.SerializerDetails Create(string alias, Akka.Serialization.Serializer serializer, System.Collections.Immutable.ImmutableHashSet useFor) { } + public static Akka.Serialization.SerializerDetails Create(string alias, Akka.Serialization.SerializerV2 serializer, System.Collections.Immutable.ImmutableHashSet useFor) { } } [Akka.Annotations.InternalApi] public static class SerializerIdentifierHelper { public static int GetSerializerIdentifierFromConfig(System.Type type, Akka.Actor.ExtendedActorSystem system) { } } + public sealed class SerializerV1Adapter : Akka.Serialization.SerializerV2 + { + public SerializerV1Adapter(Akka.Actor.ExtendedActorSystem system, Akka.Serialization.Serializer inner) { } + public override int Identifier { get; } + public Akka.Serialization.Serializer Inner { get; } + public override object Deserialize(System.Buffers.ReadOnlySequence bytes, string manifest) { } + public override object FromBinary(byte[] bytes, System.Type type) { } + public override object FromBinary(byte[] bytes, string manifest) { } + public override string Manifest(object obj) { } + public override int Serialize(object obj, System.Buffers.IBufferWriter writer) { } + public override int SizeHint(object obj) { } + public override byte[] ToBinary(object obj) { } + } + public abstract class SerializerV2 : Akka.Serialization.Serializer + { + public const int UnknownSize = -1; + protected SerializerV2(Akka.Actor.ExtendedActorSystem system) { } + public override sealed bool IncludeManifest { get; } + public abstract object Deserialize(System.Buffers.ReadOnlySequence bytes, string manifest); + public override object FromBinary(byte[] bytes, System.Type type) { } + public override object FromBinary(byte[] bytes, string manifest) { } + public abstract override string Manifest(object obj) { } + public abstract int Serialize(object obj, System.Buffers.IBufferWriter writer); + public virtual int SizeHint(object obj) { } + public override byte[] ToBinary(object obj) { } + } public abstract class SerializerWithStringManifest : Akka.Serialization.Serializer { protected SerializerWithStringManifest(Akka.Actor.ExtendedActorSystem system) { } public override sealed bool IncludeManifest { get; } public override sealed object FromBinary(byte[] bytes, System.Type type) { } - public abstract object FromBinary(byte[] bytes, string manifest); - public abstract string Manifest(object o); + public abstract override object FromBinary(byte[] bytes, string manifest) { } + public abstract override string Manifest(object o) { } } } namespace Akka.Util @@ -5658,4 +5696,4 @@ namespace Akka.Util.Reflection { public static System.Type GetType(string typeName) { } } -} \ No newline at end of file +} diff --git a/src/core/Akka.API.Tests/verify/CoreAPISpec.ApproveCore.Net.verified.txt b/src/core/Akka.API.Tests/verify/CoreAPISpec.ApproveCore.Net.verified.txt index 8e35c5962b8..7ef5454a7ea 100644 --- a/src/core/Akka.API.Tests/verify/CoreAPISpec.ApproveCore.Net.verified.txt +++ b/src/core/Akka.API.Tests/verify/CoreAPISpec.ApproveCore.Net.verified.txt @@ -3752,10 +3752,19 @@ namespace Akka.IO System.Net.Sockets.SocketAsyncEventArgs Acquire(Akka.Actor.IActorRef actor); void Release(System.Net.Sockets.SocketAsyncEventArgs e); } - public interface IStreamProvider - { - void Close(); - System.Threading.Tasks.Task ConnectAsync(string host, int port, System.Threading.CancellationToken ct); + public interface ITransportConnection : System.IAsyncDisposable + { + bool HasReadError { get; } + System.IO.Pipelines.PipeReader Input { get; } + System.Threading.Tasks.Task ReadCompleted { get; } + System.Exception? ReadError { get; } + System.Threading.Tasks.Task WriteCompleted { get; } + void Abort(); + System.Threading.Tasks.Task CloseAsync(); + System.Threading.Tasks.ValueTask FlushAsync(System.Threading.CancellationToken ct = default); + System.Threading.Tasks.Task ShutdownAsync(); + System.Threading.Tasks.ValueTask WriteAsync(System.Buffers.ReadOnlySequence data, System.Threading.CancellationToken ct = default); + System.Threading.Tasks.ValueTask WriteAsync(System.ReadOnlyMemory data, System.Threading.CancellationToken ct = default); } public class Inet { @@ -3983,8 +3992,8 @@ namespace Akka.IO } public sealed class Received : Akka.IO.Tcp.Event { - public Received(System.ReadOnlyMemory data) { } - public System.ReadOnlyMemory Data { get; } + public Received(System.Buffers.ReadOnlySequence data) { } + public System.Buffers.ReadOnlySequence Data { get; } public override string ToString() { } } public class Register : Akka.IO.Tcp.Command @@ -4051,10 +4060,12 @@ namespace Akka.IO public static readonly Akka.IO.Tcp.Write Empty; public override Akka.IO.Tcp.Event Ack { get; } public override long Bytes { get; } - public System.ReadOnlyMemory Data { get; } + public System.Buffers.ReadOnlySequence Data { get; } public override string ToString() { } + public static Akka.IO.Tcp.Write Create(System.Buffers.ReadOnlySequence data) { } public static Akka.IO.Tcp.Write Create(System.ReadOnlyMemory data) { } public static Akka.IO.Tcp.Write Create(byte[] data) { } + public static Akka.IO.Tcp.Write Create(System.Buffers.ReadOnlySequence data, Akka.IO.Tcp.Event ack) { } public static Akka.IO.Tcp.Write Create(System.ReadOnlyMemory data, Akka.IO.Tcp.Event ack) { } public static Akka.IO.Tcp.Write Create(byte[] data, Akka.IO.Tcp.Event ack) { } } @@ -4129,11 +4140,22 @@ namespace Akka.IO public static Akka.IO.TcpSettings Create(Akka.Actor.ActorSystem system) { } public static Akka.IO.TcpSettings Create(Akka.Configuration.Config config) { } } - public sealed class TcpStreamProvider : Akka.IO.IStreamProvider - { - public TcpStreamProvider() { } - public void Close() { } - public System.Threading.Tasks.Task ConnectAsync(string host, int port, System.Threading.CancellationToken ct) { } + public sealed class TcpTransportConnection : Akka.IO.ITransportConnection, System.IAsyncDisposable + { + public TcpTransportConnection(System.Net.Sockets.Socket socket, System.IO.Pipelines.PipeOptions? inputPipeOptions = null, System.IO.Pipelines.PipeOptions? outputPipeOptions = null) { } + public TcpTransportConnection(System.Net.Sockets.Socket socket, System.IO.Stream stream, System.IO.Pipelines.PipeOptions? inputPipeOptions = null, System.IO.Pipelines.PipeOptions? outputPipeOptions = null) { } + public bool HasReadError { get; } + public System.IO.Pipelines.PipeReader Input { get; } + public System.Threading.Tasks.Task ReadCompleted { get; } + public System.Exception? ReadError { get; } + public System.Threading.Tasks.Task WriteCompleted { get; } + public void Abort() { } + public System.Threading.Tasks.Task CloseAsync() { } + public System.Threading.Tasks.ValueTask DisposeAsync() { } + public System.Threading.Tasks.ValueTask FlushAsync(System.Threading.CancellationToken ct = default) { } + public System.Threading.Tasks.Task ShutdownAsync() { } + public System.Threading.Tasks.ValueTask WriteAsync(System.Buffers.ReadOnlySequence data, System.Threading.CancellationToken ct = default) { } + public System.Threading.Tasks.ValueTask WriteAsync(System.ReadOnlyMemory data, System.Threading.CancellationToken ct = default) { } } public class Udp : Akka.Actor.ExtensionIdProvider { @@ -5128,11 +5150,15 @@ namespace Akka.Routing } namespace Akka.Serialization { - public class ByteArraySerializer : Akka.Serialization.Serializer + public class ByteArraySerializer : Akka.Serialization.SerializerV2 { public ByteArraySerializer(Akka.Actor.ExtendedActorSystem system) { } - public override bool IncludeManifest { get; } + public override object Deserialize(System.Buffers.ReadOnlySequence bytes, string manifest) { } public override object FromBinary(byte[] bytes, System.Type type) { } + public override object FromBinary(byte[] bytes, string manifest) { } + public override string Manifest(object obj) { } + public override int Serialize(object obj, System.Buffers.IBufferWriter writer) { } + public override int SizeHint(object obj) { } public override byte[] ToBinary(object obj) { } } [Akka.Annotations.InternalApi] @@ -5188,9 +5214,13 @@ namespace Akka.Serialization public Serialization(Akka.Actor.ExtendedActorSystem system) { } public Akka.Actor.ExtendedActorSystem System { get; } public void AddSerializationMap(System.Type type, Akka.Serialization.Serializer serializer) { } + public void AddSerializationMap(System.Type type, Akka.Serialization.SerializerV2 serializer) { } [System.Obsolete("No longer supported. Use the AddSerializer(name, serializer) overload instead.", true)] public void AddSerializer(Akka.Serialization.Serializer serializer) { } + [System.Obsolete("No longer supported. Use the AddSerializer(name, serializer) overload instead.", true)] + public void AddSerializer(Akka.Serialization.SerializerV2 serializer) { } public void AddSerializer(string name, Akka.Serialization.Serializer serializer) { } + public void AddSerializer(string name, Akka.Serialization.SerializerV2 serializer) { } public object Deserialize(byte[] bytes, int serializerId, System.Type type) { } public object Deserialize(byte[] bytes, int serializerId, string manifest) { } public Akka.Actor.IActorRef DeserializeActorRef(string path) { } @@ -5199,6 +5229,7 @@ namespace Akka.Serialization public byte[] Serialize(object o) { } public static Akka.Serialization.Information GetCurrentTransportInformation() { } public static string ManifestFor(Akka.Serialization.Serializer s, object msg) { } + public static string ManifestFor(Akka.Serialization.SerializerV2 s, object msg) { } public static string SerializedActorPath(Akka.Actor.IActorRef actorRef) { } public static T WithTransport(Akka.Actor.ExtendedActorSystem system, System.Func action) { } [System.Obsolete("Obsolete. Use the SerializeWithTransport(ExtendedActorSystem) method instead.")] @@ -5217,7 +5248,9 @@ namespace Akka.Serialization public virtual int Identifier { get; } public abstract bool IncludeManifest { get; } public abstract object FromBinary(byte[] bytes, System.Type type); + public virtual object FromBinary(byte[] bytes, string manifest) { } public T FromBinary(byte[] bytes) { } + public virtual string Manifest(object obj) { } public abstract byte[] ToBinary(object obj); public byte[] ToBinaryWithAddress(Akka.Actor.Address address, object obj) { } } @@ -5227,19 +5260,46 @@ namespace Akka.Serialization public Akka.Serialization.Serializer Serializer { get; } public System.Collections.Immutable.ImmutableHashSet UseFor { get; } public static Akka.Serialization.SerializerDetails Create(string alias, Akka.Serialization.Serializer serializer, System.Collections.Immutable.ImmutableHashSet useFor) { } + public static Akka.Serialization.SerializerDetails Create(string alias, Akka.Serialization.SerializerV2 serializer, System.Collections.Immutable.ImmutableHashSet useFor) { } } [Akka.Annotations.InternalApi] public static class SerializerIdentifierHelper { public static int GetSerializerIdentifierFromConfig(System.Type type, Akka.Actor.ExtendedActorSystem system) { } } + public sealed class SerializerV1Adapter : Akka.Serialization.SerializerV2 + { + public SerializerV1Adapter(Akka.Actor.ExtendedActorSystem system, Akka.Serialization.Serializer inner) { } + public override int Identifier { get; } + public Akka.Serialization.Serializer Inner { get; } + public override object Deserialize(System.Buffers.ReadOnlySequence bytes, string manifest) { } + public override object FromBinary(byte[] bytes, System.Type type) { } + public override object FromBinary(byte[] bytes, string manifest) { } + public override string Manifest(object obj) { } + public override int Serialize(object obj, System.Buffers.IBufferWriter writer) { } + public override int SizeHint(object obj) { } + public override byte[] ToBinary(object obj) { } + } + public abstract class SerializerV2 : Akka.Serialization.Serializer + { + public const int UnknownSize = -1; + protected SerializerV2(Akka.Actor.ExtendedActorSystem system) { } + public override sealed bool IncludeManifest { get; } + public abstract object Deserialize(System.Buffers.ReadOnlySequence bytes, string manifest); + public override object FromBinary(byte[] bytes, System.Type type) { } + public override object FromBinary(byte[] bytes, string manifest) { } + public abstract override string Manifest(object obj) { } + public abstract int Serialize(object obj, System.Buffers.IBufferWriter writer); + public virtual int SizeHint(object obj) { } + public override byte[] ToBinary(object obj) { } + } public abstract class SerializerWithStringManifest : Akka.Serialization.Serializer { protected SerializerWithStringManifest(Akka.Actor.ExtendedActorSystem system) { } public override sealed bool IncludeManifest { get; } public override sealed object FromBinary(byte[] bytes, System.Type type) { } - public abstract object FromBinary(byte[] bytes, string manifest); - public abstract string Manifest(object o); + public abstract override object FromBinary(byte[] bytes, string manifest) { } + public abstract override string Manifest(object o) { } } } namespace Akka.Util @@ -5636,4 +5696,4 @@ namespace Akka.Util.Reflection { public static System.Type GetType(string typeName) { } } -} \ No newline at end of file +} diff --git a/src/core/Akka.Cluster.Tests/Serialization/ClusterMessageSerializerSpec.cs b/src/core/Akka.Cluster.Tests/Serialization/ClusterMessageSerializerSpec.cs index ddc06887b19..4c7f5bd061e 100644 --- a/src/core/Akka.Cluster.Tests/Serialization/ClusterMessageSerializerSpec.cs +++ b/src/core/Akka.Cluster.Tests/Serialization/ClusterMessageSerializerSpec.cs @@ -217,7 +217,7 @@ public void Can_serialize_ClusterRouterPoolWithEmptyRole() private T AssertAndReturn(T message) { - var serializer = (SerializerWithStringManifest) Sys.Serialization.FindSerializerFor(message); + var serializer = Sys.Serialization.FindSerializerFor(message); serializer.Should().BeOfType(); var serialized = serializer.ToBinary(message); diff --git a/src/core/Akka.Cluster/Serialization/ClusterMessageSerializer.cs b/src/core/Akka.Cluster/Serialization/ClusterMessageSerializer.cs index a00d21e1c78..f857e5b96da 100644 --- a/src/core/Akka.Cluster/Serialization/ClusterMessageSerializer.cs +++ b/src/core/Akka.Cluster/Serialization/ClusterMessageSerializer.cs @@ -591,12 +591,7 @@ internal static UniqueAddress UniqueAddressFrom(Proto.Msg.UniqueAddress uniqueAd [MethodImpl(MethodImplOptions.AggressiveInlining)] private static string GetObjectManifest(Serializer serializer, object obj) { - if (serializer is SerializerWithStringManifest manifestSerializer) - { - return manifestSerializer.Manifest(obj); - } - - return obj.GetType().TypeQualifiedName(); + return global::Akka.Serialization.Serialization.ManifestFor(serializer, obj); } } } diff --git a/src/core/Akka.FSharp/FsApi.fs b/src/core/Akka.FSharp/FsApi.fs index be2c80d4143..898a9dc66de 100644 --- a/src/core/Akka.FSharp/FsApi.fs +++ b/src/core/Akka.FSharp/FsApi.fs @@ -32,7 +32,7 @@ module Serialization = override _.Identifier = 99 override _.IncludeManifest = true override _.ToBinary(o) = serializeToBinary fsp o - override _.FromBinary(bytes, _) = deserializeFromBinary fsp bytes + override _.FromBinary(bytes: byte array, t: Type) : obj = deserializeFromBinary fsp bytes let exprSerializationSupport (system: ActorSystem) = @@ -55,7 +55,7 @@ module Actors = match o with | :? (byte[]) as bytes -> let serializer = context.System.Serialization.FindSerializerForType typeof<'Message> - serializer.FromBinary(bytes, typeof<'Message>) :?> 'Message + serializer.FromBinary(bytes, (typeof<'Message> : Type)) :?> 'Message | _ -> raise (InvalidCastException("Tried to cast object to " + typeof<'Message>.ToString())) /// @@ -701,4 +701,3 @@ module EventStreaming = /// Publishes an event on the provided event stream. Event channel is resolved from event's type. /// let publish (event: 'Event) (eventStream: Akka.Event.EventStream) : unit = eventStream.Publish event - diff --git a/src/core/Akka.Persistence.TCK/Journal/JournalSpec.cs b/src/core/Akka.Persistence.TCK/Journal/JournalSpec.cs index 7e646c53f65..08407856642 100644 --- a/src/core/Akka.Persistence.TCK/Journal/JournalSpec.cs +++ b/src/core/Akka.Persistence.TCK/Journal/JournalSpec.cs @@ -348,7 +348,7 @@ public virtual void Journal_optionally_may_reject_non_serializable_events() // Test that JSON actually fail var serializer = Sys.Serialization.FindSerializerForType(typeof(NotSerializableEvent)); - if (!(serializer is NewtonSoftJsonSerializer)) + if (serializer is not NewtonSoftJsonSerializer) { Output.WriteLine("[SKIP] This test only works with NewtonSoftJsonSerializer."); return; diff --git a/src/core/Akka.Persistence.Tests/Serialization/PersistenceMessageSerializerSpec.cs b/src/core/Akka.Persistence.Tests/Serialization/PersistenceMessageSerializerSpec.cs index aa5dd9b9afe..554dd2dee7c 100644 --- a/src/core/Akka.Persistence.Tests/Serialization/PersistenceMessageSerializerSpec.cs +++ b/src/core/Akka.Persistence.Tests/Serialization/PersistenceMessageSerializerSpec.cs @@ -7,13 +7,18 @@ using System; using System.Runtime.Serialization; +using System.Text; using Akka.Actor; using Akka.Configuration; using Akka.Persistence.Fsm; using Akka.Persistence.Serialization; using Akka.TestKit; +using Akka.Util; using FluentAssertions; +using Google.Protobuf; using Xunit; +using PersistentMessageProto = Akka.Persistence.Serialization.Proto.Msg.PersistentMessage; +using PersistentPayloadProto = Akka.Persistence.Serialization.Proto.Msg.PersistentPayload; namespace Akka.Persistence.Tests.Serialization { @@ -23,9 +28,11 @@ public class PersistenceMessageSerializerSpec : AkkaSpec akka.actor { serializers { my-payload = ""Akka.Persistence.Tests.Serialization.MyPayloadSerializer, Akka.Persistence.Tests"" + my-v2-payload = ""Akka.Persistence.Tests.Serialization.MyV2PayloadSerializer, Akka.Persistence.Tests"" } serialization-bindings { ""Akka.Persistence.Tests.Serialization.MyPayload, Akka.Persistence.Tests"" = my-payload + ""Akka.Persistence.Tests.Serialization.MyV2Payload, Akka.Persistence.Tests"" = my-v2-payload } }"; @@ -51,7 +58,13 @@ public void MessageSerializer_should_serialize_events_with_no_manifest_and_null_ { var p1 = new Persistent(new MyPayload("a"), sender: TestActor); var bytes = _serializer.ToBinary(p1); - var back = _serializer.FromBinary(bytes, null); + var serialized = PersistentMessageProto.Parser.ParseFrom(bytes); + + serialized.Payload.SerializerId.Should().Be(77123); + serialized.Payload.PayloadManifest.ToStringUtf8().Should().Be(typeof(MyPayload).TypeQualifiedName()); + serialized.Payload.Payload.ToByteArray().Should().Equal(Encoding.UTF8.GetBytes(".a")); + + var back = _serializer.FromBinary(bytes, (Type)null!); back.Should().BeOfType(); var persisted = (Persistent)back; @@ -63,6 +76,40 @@ public void MessageSerializer_should_serialize_events_with_no_manifest_and_null_ payload.Data.ShouldBe(".a."); } + [Fact] + public void MessageSerializer_should_serialize_native_v2_events() + { + var p1 = new Persistent(new MyV2Payload("a"), sender: TestActor); + var bytes = _serializer.ToBinary(p1); + var serialized = PersistentMessageProto.Parser.ParseFrom(bytes); + + serialized.Payload.SerializerId.Should().Be(77124); + serialized.Payload.PayloadManifest.ToStringUtf8().Should().Be(MyV2PayloadSerializer.PayloadManifest); + serialized.Payload.Payload.ToByteArray().Should().Equal(Encoding.UTF8.GetBytes("a")); + + var back = _serializer.FromBinary(bytes); + + back.Payload.Should().Be(new MyV2Payload("a")); + back.Sender.Should().BeEquivalentTo(TestActor); + } + + [Fact] + public void SnapshotSerializer_should_serialize_native_v2_snapshots() + { + var serializer = new PersistenceSnapshotSerializer(Sys.As()); + var snapshot = new Akka.Persistence.Serialization.Snapshot(new MyV2Payload("snap")); + var bytes = serializer.ToBinary(snapshot); + var serialized = PersistentPayloadProto.Parser.ParseFrom(bytes); + + serialized.SerializerId.Should().Be(77124); + serialized.PayloadManifest.ToStringUtf8().Should().Be(MyV2PayloadSerializer.PayloadManifest); + serialized.Payload.ToByteArray().Should().Equal(Encoding.UTF8.GetBytes("snap")); + + var back = serializer.FromBinary(bytes); + + back.Data.Should().Be(new MyV2Payload("snap")); + } + [Fact] public void MessageSerializer_should_serialize_state_change_event() { diff --git a/src/core/Akka.Persistence.Tests/Serialization/TestSerializers.cs b/src/core/Akka.Persistence.Tests/Serialization/TestSerializers.cs index d4a2fcadcb8..96983f9dbeb 100644 --- a/src/core/Akka.Persistence.Tests/Serialization/TestSerializers.cs +++ b/src/core/Akka.Persistence.Tests/Serialization/TestSerializers.cs @@ -6,6 +6,8 @@ //----------------------------------------------------------------------- using System; +using System.Buffers; +using System.Runtime.Serialization; using System.Text; using Akka.Actor; using Akka.Serialization; @@ -45,4 +47,68 @@ public override object FromBinary(byte[] bytes, Type type) throw new ArgumentException("unexpected manifest " + type); } } + + public sealed class MyV2Payload: IEquatable + { + public MyV2Payload(string data) + { + Data = data; + } + + public string Data { get; } + + public bool Equals(MyV2Payload other) + { + if (ReferenceEquals(null, other)) return false; + if (ReferenceEquals(this, other)) return true; + return Data == other.Data; + } + + public override bool Equals(object obj) + { + if (ReferenceEquals(null, obj)) return false; + if (ReferenceEquals(this, obj)) return true; + return obj.GetType() == GetType() && Equals((MyV2Payload)obj); + } + + public override int GetHashCode() + { + return Data.GetHashCode(); + } + } + + public sealed class MyV2PayloadSerializer : SerializerV2 + { + public const string PayloadManifest = "my-v2-payload"; + + public MyV2PayloadSerializer(ExtendedActorSystem system) : base(system) + { + } + + public override int Identifier => 77124; + + public override string Manifest(object obj) => PayloadManifest; + + public override int SizeHint(object obj) + { + return Encoding.UTF8.GetByteCount(((MyV2Payload)obj).Data); + } + + public override int Serialize(object obj, IBufferWriter writer) + { + var value = ((MyV2Payload)obj).Data; + var byteCount = Encoding.UTF8.GetByteCount(value); + var span = writer.GetSpan(byteCount); + var written = Encoding.UTF8.GetBytes(value, span); + writer.Advance(written); + return written; + } + + public override object Deserialize(ReadOnlySequence bytes, string manifest) + { + if (manifest != PayloadManifest) + throw new SerializationException($"Unknown manifest [{manifest}]"); + return new MyV2Payload(Encoding.UTF8.GetString(bytes.ToArray())); + } + } } diff --git a/src/core/Akka.Remote.Tests/Serialization/MessageSerializerV2Spec.cs b/src/core/Akka.Remote.Tests/Serialization/MessageSerializerV2Spec.cs new file mode 100644 index 00000000000..428d250bd51 --- /dev/null +++ b/src/core/Akka.Remote.Tests/Serialization/MessageSerializerV2Spec.cs @@ -0,0 +1,149 @@ +//----------------------------------------------------------------------- +// +// Copyright (C) 2009-2022 Lightbend Inc. +// Copyright (C) 2013-2025 .NET Foundation +// +//----------------------------------------------------------------------- + +using System; +using System.Buffers; +using System.Runtime.Serialization; +using System.Text; +using Akka.Actor; +using Akka.Configuration; +using Akka.Serialization; +using Akka.TestKit; +using FluentAssertions; +using Xunit; + +namespace Akka.Remote.Tests.Serialization +{ + public class MessageSerializerV2Spec: AkkaSpec + { + private static readonly Config Config = ConfigurationFactory.ParseString($@" + akka.actor {{ + serializers {{ + native-v2 = ""{typeof(NativeV2Serializer).AssemblyQualifiedName}"" + }} + serialization-bindings {{ + ""{typeof(NativeV2Message).AssemblyQualifiedName}"" = native-v2 + }} + }}"); + + public MessageSerializerV2Spec(ITestOutputHelper output) : base(output, Config) + { + } + + [Fact] + public void Classic_MessageSerializer_should_roundtrip_native_v2_payloads() + { + var message = new NativeV2Message("classic remoting v2"); + var address = new Address("akka.tcp", "Sys", "localhost", 2551); + var info = new Information(address, Sys); + + var serialized = MessageSerializer.Serialize((ExtendedActorSystem)Sys, info, message); + + serialized.SerializerId.Should().Be(NativeV2Serializer.SerializerId); + serialized.MessageManifest.ToStringUtf8().Should().Be(NativeV2Serializer.NativeManifest); + serialized.Message.ToByteArray().Should().Equal(Encoding.UTF8.GetBytes(message.Value)); + MessageSerializer.Deserialize((ExtendedActorSystem)Sys, serialized).Should().Be(message); + } + + [Fact] + public void Classic_MessageSerializer_should_preserve_legacy_byte_array_wire_metadata() + { + var message = new byte[] { 1, 2, 3, 4 }; + var address = new Address("akka.tcp", "Sys", "localhost", 2551); + var info = new Information(address, Sys); + + var serialized = MessageSerializer.Serialize((ExtendedActorSystem)Sys, info, message); + + serialized.SerializerId.Should().Be(4); + serialized.MessageManifest.IsEmpty.Should().BeTrue(); + serialized.Message.ToByteArray().Should().Equal(message); + + var deserialized = MessageSerializer.Deserialize((ExtendedActorSystem)Sys, serialized); + deserialized.Should().BeOfType(); + ((byte[])deserialized).Should().Equal(message); + } + + [Fact] + public void WrappedPayloadSupport_should_preserve_payload_serializer_metadata() + { + var message = new NativeV2Message("wrapped v2"); + var support = new Akka.Remote.Serialization.WrappedPayloadSupport((ExtendedActorSystem)Sys); + + var payload = support.PayloadToProto(message); + + payload.SerializerId.Should().Be(NativeV2Serializer.SerializerId); + payload.MessageManifest.ToStringUtf8().Should().Be(NativeV2Serializer.NativeManifest); + payload.Message.ToByteArray().Should().Equal(Encoding.UTF8.GetBytes(message.Value)); + support.PayloadFrom(payload).Should().Be(message); + } + + public sealed class NativeV2Message: IEquatable + { + public NativeV2Message(string value) + { + Value = value; + } + + public string Value { get; } + + public bool Equals(NativeV2Message other) + { + if (ReferenceEquals(null, other)) return false; + if (ReferenceEquals(this, other)) return true; + return Value == other.Value; + } + + public override bool Equals(object obj) + { + if (ReferenceEquals(null, obj)) return false; + if (ReferenceEquals(this, obj)) return true; + return obj.GetType() == GetType() && Equals((NativeV2Message)obj); + } + + public override int GetHashCode() + { + return Value.GetHashCode(); + } + } + + public sealed class NativeV2Serializer : SerializerV2 + { + public const int SerializerId = 77125; + public const string NativeManifest = "native-v2-message"; + + public NativeV2Serializer(ExtendedActorSystem system) : base(system) + { + } + + public override int Identifier => SerializerId; + + public override string Manifest(object obj) => NativeManifest; + + public override int SizeHint(object obj) + { + return Encoding.UTF8.GetByteCount(((NativeV2Message)obj).Value); + } + + public override int Serialize(object obj, IBufferWriter writer) + { + var value = ((NativeV2Message)obj).Value; + var byteCount = Encoding.UTF8.GetByteCount(value); + var span = writer.GetSpan(byteCount); + var written = Encoding.UTF8.GetBytes(value, span); + writer.Advance(written); + return written; + } + + public override object Deserialize(ReadOnlySequence bytes, string manifest) + { + if (manifest != NativeManifest) + throw new SerializationException($"Unknown manifest [{manifest}]"); + return new NativeV2Message(Encoding.UTF8.GetString(bytes.ToArray())); + } + } + } +} diff --git a/src/core/Akka.Remote.Tests/Serialization/MiscMessageSerializerSpec.cs b/src/core/Akka.Remote.Tests/Serialization/MiscMessageSerializerSpec.cs index 39a12a46ece..ae39aee28a9 100644 --- a/src/core/Akka.Remote.Tests/Serialization/MiscMessageSerializerSpec.cs +++ b/src/core/Akka.Remote.Tests/Serialization/MiscMessageSerializerSpec.cs @@ -392,12 +392,8 @@ private T AssertAndReturn(T message) var serializer = Sys.Serialization.FindSerializerFor(message); serializer.Should().BeOfType(); var serializedBytes = serializer.ToBinary(message); - - if (serializer is SerializerWithStringManifest serializerManifest) - { - return (T)serializerManifest.FromBinary(serializedBytes, serializerManifest.Manifest(message)); - } - return (T)serializer.FromBinary(serializedBytes, typeof(T)); + var manifest = Akka.Serialization.Serialization.ManifestFor(serializer, message); + return (T)serializer.FromBinary(serializedBytes, manifest); } private void AssertEqual(T message) diff --git a/src/core/Akka.Remote.Tests/Serialization/PrimitiveSerializersSpec.cs b/src/core/Akka.Remote.Tests/Serialization/PrimitiveSerializersSpec.cs index 984c11e28c3..617ba1e3f8c 100644 --- a/src/core/Akka.Remote.Tests/Serialization/PrimitiveSerializersSpec.cs +++ b/src/core/Akka.Remote.Tests/Serialization/PrimitiveSerializersSpec.cs @@ -58,7 +58,7 @@ public void Can_serialize_String(string value) private T AssertAndReturn(T message) { - var serializer = (SerializerWithStringManifest)Sys.Serialization.FindSerializerFor(message); + var serializer = Sys.Serialization.FindSerializerFor(message); serializer.Should().BeOfType(); var serializedBytes = serializer.ToBinary(message); var manifest = serializer.Manifest(message); @@ -67,7 +67,7 @@ private T AssertAndReturn(T message) private T AssertCrossPlatformAndReturn(T message) { - var serializer = (SerializerWithStringManifest)Sys.Serialization.FindSerializerFor(message); + var serializer = Sys.Serialization.FindSerializerFor(message); serializer.Should().BeOfType(); var serializedBytes = serializer.ToBinary(message); // GetType() will make sure that each namespace is compatible with the serializer diff --git a/src/core/Akka.Remote/MessageSerializer.cs b/src/core/Akka.Remote/MessageSerializer.cs index 8b99ed08e91..46833e1b2c0 100644 --- a/src/core/Akka.Remote/MessageSerializer.cs +++ b/src/core/Akka.Remote/MessageSerializer.cs @@ -59,18 +59,10 @@ public static SerializedMessage Serialize(ExtendedActorSystem system, Informatio SerializerId = serializer.Identifier }; - if (serializer is SerializerWithStringManifest serializer2) + var manifest = Akka.Serialization.Serialization.ManifestFor(serializer, message); + if (!string.IsNullOrEmpty(manifest)) { - var manifest = serializer2.Manifest(message); - if (!string.IsNullOrEmpty(manifest)) - { - serializedMsg.MessageManifest = ByteString.CopyFromUtf8(manifest); - } - } - else - { - if (serializer.IncludeManifest) - serializedMsg.MessageManifest = ByteString.CopyFromUtf8(message.GetType().TypeQualifiedName()); + serializedMsg.MessageManifest = ByteString.CopyFromUtf8(manifest); } return serializedMsg; diff --git a/src/core/Akka.Streams.Tests/IO/OutputStreamSourceSpec.cs b/src/core/Akka.Streams.Tests/IO/OutputStreamSourceSpec.cs index 20336679673..056b5373400 100644 --- a/src/core/Akka.Streams.Tests/IO/OutputStreamSourceSpec.cs +++ b/src/core/Akka.Streams.Tests/IO/OutputStreamSourceSpec.cs @@ -56,6 +56,12 @@ public OutputStreamSourceSpec(ITestOutputHelper helper) : base( private static async Task ExpectTimeout(Task f, TimeSpan duration) => (await f.AwaitWithTimeout(duration)).Should().BeFalse(); + private static async Task ExpectNextBytesAsync(TestSubscriber.ManualProbe> probe, byte[] expected) + { + var actual = await probe.ExpectNextAsync(); + actual.ToArray().Should().Equal(expected); + } + [Fact] public async Task OutputStreamSource_must_read_bytes_from_OutputStream() { @@ -69,7 +75,33 @@ await this.AssertAllStagesStoppedAsync(async () => await outputStream.WriteAsync(_bytesArray, 0, _bytesArray.Length) .WaitAsync(Timeout); s.Request(1); - await probe.ExpectNextAsync(_byteString); + await ExpectNextBytesAsync(probe, _bytesArray); + outputStream.Dispose(); + await probe.ExpectCompleteAsync(); + }, _materializer); + } + + [Fact] + public async Task OutputStreamSource_must_copy_bytes_from_OutputStream_before_emitting() + { + await this.AssertAllStagesStoppedAsync(async () => + { + var (outputStream, probe) = StreamConverters.AsOutputStream() + .ToMaterialized(this.SinkProbe>(), Keep.Both) + .Run(_materializer); + var s = await probe.ExpectSubscriptionAsync(); + + var buffer = new byte[] { 1, 2, 3 }; + var expected = buffer.ToArray(); + await outputStream.WriteAsync(buffer, 0, buffer.Length) + .WaitAsync(Timeout); + + buffer[0] = 9; + buffer[1] = 9; + buffer[2] = 9; + + s.Request(1); + await ExpectNextBytesAsync(probe, expected); outputStream.Dispose(); await probe.ExpectCompleteAsync(); }, _materializer); @@ -97,7 +129,7 @@ await outputStream.WriteAsync(_bytesArray, 0, _bytesArray.Length) s.Request(1); await f.WaitAsync(Timeout); - await probe.AsyncBuilder().ExpectNext(_byteString).ExecuteAsync(); + await ExpectNextBytesAsync(probe, _bytesArray); } await probe.AsyncBuilder().ExpectComplete().ExecuteAsync(); @@ -122,7 +154,7 @@ await outputStream.WriteAsync(_bytesArray, 0, (int)_byteString.Length) var f = outputStream.FlushAsync(); s.Request(1); await f.WaitAsync(Timeout); - await probe.ExpectNextAsync(_byteString); + await ExpectNextBytesAsync(probe, _bytesArray); var f2 = outputStream.FlushAsync(); await f2.WaitAsync(Timeout); @@ -158,7 +190,8 @@ await outputStream.WriteAsync(_bytesArray, 0, (int)_byteString.Length) s.Request(17); await f.WaitAsync(Timeout); - await probe.ExpectNextNAsync(Enumerable.Repeat(_byteString, 17).ToList()); + for (var i = 0; i < 17; i++) + await ExpectNextBytesAsync(probe, _bytesArray); } await probe.ExpectCompleteAsync(); @@ -223,7 +256,7 @@ await outputStream.WriteAsync(_bytesArray, 0, _bytesArray.Length) s.Request(1); await sourceProbe.ExpectMsgAsync(); - await probe.ExpectNextAsync(_byteString); + await ExpectNextBytesAsync(probe, _bytesArray); s.Cancel(); await sourceProbe.ExpectMsgAsync(); diff --git a/src/core/Akka.Streams.Tests/Serialization/StreamRefSerializer.cs b/src/core/Akka.Streams.Tests/Serialization/StreamRefSerializer.cs index 40568ed1f20..5da0597a615 100644 --- a/src/core/Akka.Streams.Tests/Serialization/StreamRefSerializer.cs +++ b/src/core/Akka.Streams.Tests/Serialization/StreamRefSerializer.cs @@ -25,7 +25,8 @@ public StreamRefSerializer(ITestOutputHelper output) public void StreamsConfigBugTest() { var message = new SequencedOnNext(10, "test"); - var serializer = (SerializerWithStringManifest)Sys.Serialization.FindSerializerFor(message); + var serializer = Sys.Serialization.FindSerializerFor(message); + serializer.Should().BeOfType(); var manifest = serializer.Manifest(message); byte[] bytes = null; diff --git a/src/core/Akka.Streams/Implementation/Fusing/Ops.cs b/src/core/Akka.Streams/Implementation/Fusing/Ops.cs index e6bc190579b..67c812c71aa 100644 --- a/src/core/Akka.Streams/Implementation/Fusing/Ops.cs +++ b/src/core/Akka.Streams/Implementation/Fusing/Ops.cs @@ -4285,14 +4285,22 @@ public override void OnPush() } catch (Exception ex) { - _promise.TrySetException(ex); + if (_promise.TrySetException(ex)) + { + // The materialized task may be discarded; observe the fault to avoid a later UnobservedTaskException. + _ = _promise.Task.Exception; + } FailStage(ex); } } } else { - _promise.TrySetException(result.Exception); + if (_promise.TrySetException(result.Exception)) + { + // The materialized task may be discarded; observe the fault to avoid a later UnobservedTaskException. + _ = _promise.Task.Exception; + } FailStage(result.Exception); } }); @@ -4304,7 +4312,11 @@ public override void OnPush() } catch (Exception ex) { - _promise.TrySetException(ex); + if (_promise.TrySetException(ex)) + { + // The materialized task may be discarded; observe the fault to avoid a later UnobservedTaskException. + _ = _promise.Task.Exception; + } FailStage(ex); } } @@ -4325,7 +4337,11 @@ public override void OnUpstreamFinish() public override void OnUpstreamFailure(Exception ex) { - _promise.TrySetException(ex); + if (_promise.TrySetException(ex)) + { + // The materialized task may be discarded; observe the fault to avoid a later UnobservedTaskException. + _ = _promise.Task.Exception; + } base.OnUpstreamFailure(ex); } diff --git a/src/core/Akka.Streams/Implementation/IO/OutputStreamSourceStage.cs b/src/core/Akka.Streams/Implementation/IO/OutputStreamSourceStage.cs index 4dfec299d03..af18b2a74de 100644 --- a/src/core/Akka.Streams/Implementation/IO/OutputStreamSourceStage.cs +++ b/src/core/Akka.Streams/Implementation/IO/OutputStreamSourceStage.cs @@ -442,7 +442,12 @@ private void SendMessage(IAdapterToStageMessage msg, bool handleCancelled = true /// TBD /// TBD public override void Write(byte[] buffer, int offset, int count) - => SendData(new ReadOnlySequence(buffer, offset, count)); + { + // Stream.Write returns before downstream demand may consume this chunk, so + // copy out of the caller-owned buffer before publishing it to the stream. + var source = new ReadOnlyMemory(buffer, offset, count); + SendData(new ReadOnlySequence(source.ToArray())); + } /// /// TBD diff --git a/src/core/Akka.Tests/Delivery/ConsumerControllerSpecs.cs b/src/core/Akka.Tests/Delivery/ConsumerControllerSpecs.cs index 6696fce1ed0..326b63b18c7 100644 --- a/src/core/Akka.Tests/Delivery/ConsumerControllerSpecs.cs +++ b/src/core/Akka.Tests/Delivery/ConsumerControllerSpecs.cs @@ -12,6 +12,7 @@ using Akka.Configuration; using Akka.Delivery; using Akka.Delivery.Internal; +using Akka.Serialization; using Akka.Util; using FluentAssertions; using Xunit; @@ -576,6 +577,36 @@ public async Task ConsumedController_with_ChunkedMessages_must_collect_and_assem await producerControllerProbe.ExpectMsgAsync(new ProducerController.Ack(5)); } + [Fact] + public async Task ConsumerController_with_ChunkedMessages_must_collect_and_assemble_native_v2_byte_array_chunks() + { + NextId(); + var consumerController = Sys.ActorOf(ConsumerController.Create(Sys, Option.None), + $"consumerController-{_idCount}"); + var producerControllerProbe = CreateTestProbe(); + + var consumerProbe = CreateTestProbe(); + consumerController.Tell(new ConsumerController.Start(consumerProbe)); + + var payload = new byte[] { 1, 2, 3, 4 }; + var serializer = Sys.Serialization.FindSerializerV2For(payload); + var chunks = ProducerController.CreateChunks(payload, chunkSize: 1, Sys.Serialization).ToList(); + + chunks.Should().HaveCount(payload.Length); + chunks.Should().OnlyContain(c => c.SerializerId == serializer.Identifier); + chunks.Should().OnlyContain(c => c.Manifest == string.Empty); + + var seqMessages = chunks.Select((c, i) => + ConsumerController.SequencedMessage.FromChunkedMessage(ProducerId, 1 + i, c, i == 0, false, + producerControllerProbe)).ToList(); + + foreach (var message in seqMessages) + consumerController.Tell(message); + + var delivery = await consumerProbe.ExpectMsgAsync>(); + delivery.Message.Should().Equal(payload); + } + [Fact] public async Task ConsumerController_with_ChunkedMessages_must_send_Request_after_half_window_size_when_many_chunks() diff --git a/src/core/Akka.Tests/Pattern/CircuitBreakerSpec.cs b/src/core/Akka.Tests/Pattern/CircuitBreakerSpec.cs index 61649c50811..ec5c1d8a37f 100644 --- a/src/core/Akka.Tests/Pattern/CircuitBreakerSpec.cs +++ b/src/core/Akka.Tests/Pattern/CircuitBreakerSpec.cs @@ -241,7 +241,7 @@ public async Task Must_increment_failure_count_on_callTimeout() var future = breaker.Instance.WithCircuitBreaker(async ct => { - await Task.Delay(150, ct); + await Task.Delay(Dilated(TimeSpan.FromSeconds(1)), ct); ThrowException(); }); diff --git a/src/core/Akka.Tests/Serialization/CustomSerializerSpec.cs b/src/core/Akka.Tests/Serialization/CustomSerializerSpec.cs index 79074dd25a6..b52c783cc7d 100644 --- a/src/core/Akka.Tests/Serialization/CustomSerializerSpec.cs +++ b/src/core/Akka.Tests/Serialization/CustomSerializerSpec.cs @@ -32,6 +32,11 @@ public CustomSerializerSpec(ITestOutputHelper output) _output = output; } + private static T Inner(Serializer serializer) where T: Serializer + { + return serializer.Should().BeOfType().Subject; + } + /// /// Here we basically verify that a serializer decides where its Serializer Identifier is coming /// from. When using the default Serializer base class, it read from hocon config. But this should not be @@ -53,7 +58,7 @@ public void Custom_serializer_must_be_owner_of_its_serializerId() //The above config explictly does not configures the serialization-identifiers section using (var system = ActorSystem.Create(nameof(CustomSerializerSpec), config)) { - var serializer = (CustomSerializer)system.Serialization.FindSerializerForType(typeof(object)); + var serializer = Inner(system.Serialization.FindSerializerForType(typeof(object))); Assert.Equal(666, serializer.Identifier); } } @@ -79,7 +84,7 @@ public void Custom_SerializerWithStringManifest_should_work_with_base_class_bind { var firstMessage = new FirstMessage("First message"); var serialization = system.Serialization; - var serializer = (CustomManifestSerializer)serialization.FindSerializerFor(firstMessage); + var serializer = Inner(serialization.FindSerializerFor(firstMessage)); var serialized = serializer.ToBinary(firstMessage); var manifest = serializer.Manifest(firstMessage); @@ -156,7 +161,7 @@ public void Hardwired_serialization_identifiers_should_override_HOCON_configurat { var firstMessage = new FirstMessage("First message"); var serialization = system.Serialization; - var serializer = (CustomSerializer)serialization.FindSerializerFor(firstMessage); + var serializer = Inner(serialization.FindSerializerFor(firstMessage)); var serializerById = serialization.GetSerializerById(1); serializer.Identifier.Should().Be(666); // This is because identifier is hardwired, so it could not be @@ -257,7 +262,7 @@ public void Custom_programmatic_SerializerWithStringManifest_should_work_with_ba { var firstMessage = new FirstMessage("First message"); var serialization = system.Serialization; - var serializer = (CustomManifestSerializer)serialization.FindSerializerFor(firstMessage); + var serializer = Inner(serialization.FindSerializerFor(firstMessage)); var serialized = serializer.ToBinary(firstMessage); var manifest = serializer.Manifest(firstMessage); diff --git a/src/core/Akka.Tests/Serialization/NewtonSoftJsonSerializerSetupSpec.cs b/src/core/Akka.Tests/Serialization/NewtonSoftJsonSerializerSetupSpec.cs index 991a28d12fe..6a24779dab7 100644 --- a/src/core/Akka.Tests/Serialization/NewtonSoftJsonSerializerSetupSpec.cs +++ b/src/core/Akka.Tests/Serialization/NewtonSoftJsonSerializerSetupSpec.cs @@ -51,7 +51,7 @@ public NewtonSoftJsonSerializerSetupSpec(ITestOutputHelper output) [Fact] public void Setup_should_be_used_inside_Json_serializer() { - var serializer = (NewtonSoftJsonSerializer) Sys.Serialization.FindSerializerForType(typeof(object)); + var serializer = Sys.Serialization.FindSerializerForType(typeof(object)).Should().BeOfType().Subject; var settings = serializer.Settings; settings.ReferenceLoopHandling.Should().Be(ReferenceLoopHandling.Error); settings.MissingMemberHandling.Should().Be(MissingMemberHandling.Error); @@ -62,7 +62,7 @@ public void Setup_should_be_used_inside_Json_serializer() [Fact] public void Setup_should_not_change_mandatory_settings() { - var serializer = (NewtonSoftJsonSerializer) Sys.Serialization.FindSerializerForType(typeof(object)); + var serializer = Sys.Serialization.FindSerializerForType(typeof(object)).Should().BeOfType().Subject; var settings = serializer.Settings; settings.ContractResolver.Should().BeOfType(); settings.ObjectCreationHandling.Should().Be(ObjectCreationHandling.Replace); diff --git a/src/core/Akka.Tests/Serialization/NewtonsoftJsonConfigSpec.cs b/src/core/Akka.Tests/Serialization/NewtonsoftJsonConfigSpec.cs index 3f43a569019..2102ed66f3b 100644 --- a/src/core/Akka.Tests/Serialization/NewtonsoftJsonConfigSpec.cs +++ b/src/core/Akka.Tests/Serialization/NewtonsoftJsonConfigSpec.cs @@ -10,6 +10,7 @@ using Akka.Actor; using Akka.Configuration; using Akka.Serialization; +using FluentAssertions; using Newtonsoft.Json; using Newtonsoft.Json.Converters; using Xunit; @@ -23,7 +24,7 @@ public void Json_serializer_should_have_correct_defaults() { using (var system = ActorSystem.Create(nameof(NewtonsoftJsonConfigSpec))) { - var serializer = (NewtonSoftJsonSerializer)system.Serialization.FindSerializerForType(typeof(object)); + var serializer = system.Serialization.FindSerializerForType(typeof(object)).Should().BeOfType().Subject; Assert.Equal(TypeNameHandling.All, serializer.Settings.TypeNameHandling); Assert.Equal(PreserveReferencesHandling.Objects, serializer.Settings.PreserveReferencesHandling); Assert.Equal(2, serializer.Settings.Converters.Count); @@ -45,7 +46,7 @@ public void Json_serializer_should_allow_to_setup_custom_flags() "); using (var system = ActorSystem.Create(nameof(NewtonsoftJsonConfigSpec), config)) { - var serializer = (NewtonSoftJsonSerializer)system.Serialization.FindSerializerForType(typeof(object)); + var serializer = system.Serialization.FindSerializerForType(typeof(object)).Should().BeOfType().Subject; Assert.Equal(TypeNameHandling.None, serializer.Settings.TypeNameHandling); Assert.Equal(PreserveReferencesHandling.None, serializer.Settings.PreserveReferencesHandling); Assert.Equal(2, serializer.Settings.Converters.Count); @@ -69,7 +70,7 @@ public void Json_serializer_should_allow_to_setup_custom_converters() "); using (var system = ActorSystem.Create(nameof(NewtonsoftJsonConfigSpec), config)) { - var serializer = (NewtonSoftJsonSerializer)system.Serialization.FindSerializerForType(typeof(object)); + var serializer = system.Serialization.FindSerializerForType(typeof(object)).Should().BeOfType().Subject; Assert.Equal(TypeNameHandling.All, serializer.Settings.TypeNameHandling); Assert.Equal(PreserveReferencesHandling.Objects, serializer.Settings.PreserveReferencesHandling); Assert.Equal(4, serializer.Settings.Converters.Count); diff --git a/src/core/Akka.Tests/Serialization/SerializationSpec.cs b/src/core/Akka.Tests/Serialization/SerializationSpec.cs index 46bc4789c09..e4d36aa33be 100644 --- a/src/core/Akka.Tests/Serialization/SerializationSpec.cs +++ b/src/core/Akka.Tests/Serialization/SerializationSpec.cs @@ -6,11 +6,13 @@ //----------------------------------------------------------------------- using System; +using System.Buffers; using System.Collections.Concurrent; using System.Collections.Generic; using System.IO; using System.Reflection; using System.Runtime.Serialization; +using System.Text; using System.Text.RegularExpressions; using Akka.Actor; using Akka.Configuration; @@ -542,23 +544,80 @@ public void Can_serialize_FutureActorRef() [Fact] public void Can_get_serializer_by_binding() { - Sys.Serialization.FindSerializerFor(null).GetType().ShouldBe(typeof(NullSerializer)); - Sys.Serialization.FindSerializerFor(new byte[]{1,2,3}).GetType().ShouldBe(typeof(ByteArraySerializer)); - Sys.Serialization.FindSerializerFor("dummy").GetType().ShouldBe(typeof(DummySerializer)); - Sys.Serialization.FindSerializerFor(123).GetType().ShouldBe(typeof(NewtonSoftJsonSerializer)); + Sys.Serialization.FindSerializerFor(null).Should().BeOfType(); + Sys.Serialization.FindSerializerFor(new byte[]{1,2,3}).Should().BeOfType(); + Sys.Serialization.FindSerializerFor("dummy").Should().BeOfType(); + Sys.Serialization.FindSerializerFor(123).Should().BeOfType(); } [Fact] public void Can_apply_a_config_based_serializer_by_the_binding() { - var dummy = (DummySerializer)Sys.Serialization.FindSerializerFor("dummy"); + var dummy = Sys.Serialization.FindSerializerFor("dummy").Should().BeOfType().Subject; dummy.Config.ShouldBe(null); - var dummy2 = (DummyConfigurableSerializer) Sys.Serialization.GetSerializerById(-7); + var dummy2 = Sys.Serialization.GetSerializerById(-7).Should().BeOfType().Subject; dummy2.Config.ShouldNotBe(null); dummy2.Config.GetString("test-key", null).ShouldBe("test value"); } + [Fact] + public void Can_roundtrip_native_v2_serializer() + { + var message = new NativeV2Message("v2 payload"); + var serializer = Sys.Serialization.FindSerializerFor(message); + + serializer.Should().BeOfType(); + var manifest = Akka.Serialization.Serialization.ManifestFor(serializer, message); + manifest.Should().Be(NativeV2Serializer.NativeManifest); + + var bytes = Sys.Serialization.Serialize(message); + var deserialized = Sys.Serialization.Deserialize(bytes, serializer.Identifier, manifest); + + deserialized.Should().Be(message); + } + + [Fact] + public void Can_roundtrip_byte_array_serializer_through_v2_buffer_api() + { + var message = new byte[] { 1, 2, 3, 4 }; + var serializer = Sys.Serialization.FindSerializerV2For(message); + + serializer.Should().BeOfType(); + serializer.Identifier.Should().Be(4); + serializer.Manifest(message).Should().BeEmpty(); + serializer.SizeHint(message).Should().Be(message.Length); + + var writer = new ArrayBufferWriter(); + var metadata = Sys.Serialization.Serialize(message, writer); + + metadata.SerializerId.Should().Be(serializer.Identifier); + metadata.Manifest.Should().BeEmpty(); + metadata.BytesWritten.Should().Be(message.Length); + writer.WrittenSpan.ToArray().Should().Equal(message); + + var deserialized = Sys.Serialization.Deserialize( + new ReadOnlySequence(writer.WrittenMemory), + metadata.SerializerId, + metadata.Manifest); + + deserialized.Should().BeOfType(); + ((byte[])deserialized).Should().Equal(message); + + var legacyDeserialized = serializer.FromBinary(message, string.Empty); + legacyDeserialized.Should().BeSameAs(message); + } + + [Fact] + public void Native_v2_serializer_must_return_non_empty_manifest() + { + var serializer = new EmptyManifestV2Serializer((ExtendedActorSystem)Sys); + Action action = () => Akka.Serialization.Serialization.ManifestFor(serializer, new NativeV2Message("no manifest")); + + action.Should().Throw() + .WithMessage("*must return a non-empty manifest*"); + } + private static string LegacyTypeQualifiedName(Type type) { @@ -613,7 +672,7 @@ public void Missing_custom_serializer_id_should_append_help_message() [Fact(DisplayName = "Should be able to serialize object property with JObject value")] public void ObjectPropertyJObjectTest() { - var serializer = (NewtonSoftJsonSerializer) Sys.Serialization.FindSerializerForType(typeof(object)); + var serializer = Sys.Serialization.FindSerializerForType(typeof(object)).Should().BeOfType().Subject; var obj = JObject.FromObject(new { FormattedMessage = "We are apple 20 points above value 10.01 ms", @@ -648,7 +707,7 @@ public void ObjectPropertyJObjectTest() [Fact(DisplayName = "Should be able to serialize object property with anonymous type value")] public void ObjectPropertyObjectTest() { - var serializer = (NewtonSoftJsonSerializer) Sys.Serialization.FindSerializerForType(typeof(object)); + var serializer = Sys.Serialization.FindSerializerForType(typeof(object)).Should().BeOfType().Subject; var obj = new { FormattedMessage = "We are apple 20 points above value 10.01 ms", @@ -673,10 +732,12 @@ private static string GetConfig() => @" serializers { dummy = """ + typeof(DummySerializer).AssemblyQualifiedName + @""" dummy2 = """ + typeof(DummyConfigurableSerializer).AssemblyQualifiedName + @""" + native-v2 = """ + typeof(NativeV2Serializer).AssemblyQualifiedName + @""" } serialization-bindings { ""System.String"" = dummy + """ + typeof(NativeV2Message).AssemblyQualifiedName + @""" = native-v2 } serialization-settings { dummy2 { @@ -751,6 +812,91 @@ public override object FromBinary(byte[] bytes, Type type) } } + public sealed class NativeV2Message : IEquatable + { + public NativeV2Message(string value) + { + Value = value; + } + + public string Value { get; } + + public bool Equals(NativeV2Message other) + { + if (ReferenceEquals(null, other)) return false; + if (ReferenceEquals(this, other)) return true; + return Value == other.Value; + } + + public override bool Equals(object obj) + { + if (ReferenceEquals(null, obj)) return false; + if (ReferenceEquals(this, obj)) return true; + return obj.GetType() == GetType() && Equals((NativeV2Message)obj); + } + + public override int GetHashCode() + { + return Value.GetHashCode(); + } + } + + public sealed class NativeV2Serializer : SerializerV2 + { + public const string NativeManifest = "native-v2"; + + public NativeV2Serializer(ExtendedActorSystem system) : base(system) + { + } + + public override int Identifier => -106; + + public override string Manifest(object obj) => NativeManifest; + + public override int SizeHint(object obj) + { + return Encoding.UTF8.GetByteCount(((NativeV2Message)obj).Value); + } + + public override int Serialize(object obj, IBufferWriter writer) + { + var value = ((NativeV2Message)obj).Value; + var byteCount = Encoding.UTF8.GetByteCount(value); + var span = writer.GetSpan(byteCount); + var written = Encoding.UTF8.GetBytes(value, span); + writer.Advance(written); + return written; + } + + public override object Deserialize(ReadOnlySequence bytes, string manifest) + { + if (manifest != NativeManifest) + throw new SerializationException($"Unknown manifest [{manifest}]"); + return new NativeV2Message(Encoding.UTF8.GetString(bytes.ToArray())); + } + } + + public sealed class EmptyManifestV2Serializer : SerializerV2 + { + public EmptyManifestV2Serializer(ExtendedActorSystem system) : base(system) + { + } + + public override int Identifier => -107; + + public override string Manifest(object obj) => string.Empty; + + public override int Serialize(object obj, IBufferWriter writer) + { + return 0; + } + + public override object Deserialize(ReadOnlySequence bytes, string manifest) + { + return new object(); + } + } + public sealed class OtherClassA { } public sealed class OtherClassB { } @@ -832,4 +978,3 @@ public void Should_use_object_fallback_by_default() private sealed class SomeRandomType { } } } - diff --git a/src/core/Akka/Actor/Internal/ActorSystemImpl.cs b/src/core/Akka/Actor/Internal/ActorSystemImpl.cs index b1003dd9c54..217d9fc7bd1 100644 --- a/src/core/Akka/Actor/Internal/ActorSystemImpl.cs +++ b/src/core/Akka/Actor/Internal/ActorSystemImpl.cs @@ -699,4 +699,3 @@ public void Add(Action code) public Task TerminationTask { get { return _atomicRef.Value ?? _terminationTask; } } } } - diff --git a/src/core/Akka/Delivery/Internal/ConsumerControllerImpl.cs b/src/core/Akka/Delivery/Internal/ConsumerControllerImpl.cs index 3e68e50472d..e676bf39e4d 100644 --- a/src/core/Akka/Delivery/Internal/ConsumerControllerImpl.cs +++ b/src/core/Akka/Delivery/Internal/ConsumerControllerImpl.cs @@ -599,34 +599,55 @@ long GetNewRequestedSeqNr() private SequencedMessage AssembleChunks(ImmutableList> collectedChunks) { - var bufferSize = collectedChunks.Sum(chunk => chunk.Message.Chunk!.Value.SerializedMessage.Length); - byte[] bytes; - using (var mem = MemoryPool.Shared.Rent(bufferSize)) - { - var curIndex = 0; - var memory = mem.Memory; - foreach (var b in collectedChunks.Select(c => c.Message.Chunk!.Value.SerializedMessage)) - { - b.CopyTo(memory.Slice(curIndex)); - curIndex += b.Length; - } - - // have to slice the buffer here since the memory pool may have allocated more than we needed - bytes = memory.Slice(0, curIndex).ToArray(); - } - var headMessage = collectedChunks.Last(); // this is the last chunk var headChunk = headMessage.Message.Chunk!.Value; // serialization exceptions are thrown, because it will anyway be stuck with same error if retried and // we can't just ignore the message - var message = (T)_serialization.Deserialize(bytes, headChunk.SerializerId, headChunk.Manifest); + var message = (T)_serialization.Deserialize(ToReadOnlySequence(collectedChunks), headChunk.SerializerId, headChunk.Manifest); return new SequencedMessage(headMessage.ProducerId, headMessage.SeqNr, message, collectedChunks.First().First, headMessage.Ack, headMessage.ProducerController); } + private static ReadOnlySequence ToReadOnlySequence(ImmutableList> collectedChunks) + { + if (collectedChunks.Count == 0) + return ReadOnlySequence.Empty; + + var firstMemory = collectedChunks[0].Message.Chunk!.Value.SerializedMessage; + if (collectedChunks.Count == 1) + return new ReadOnlySequence(firstMemory); + + var first = new ReadOnlyMemorySegment(firstMemory); + var last = first; + foreach (var chunk in collectedChunks.Skip(1)) + { + last = last.Append(chunk.Message.Chunk!.Value.SerializedMessage); + } + + return new ReadOnlySequence(first, 0, last, last.Memory.Length); + } + + private sealed class ReadOnlyMemorySegment : ReadOnlySequenceSegment + { + public ReadOnlyMemorySegment(ReadOnlyMemory memory) + { + Memory = memory; + } + + public ReadOnlyMemorySegment Append(ReadOnlyMemory memory) + { + var segment = new ReadOnlyMemorySegment(memory) + { + RunningIndex = RunningIndex + Memory.Length + }; + Next = segment; + return segment; + } + } + private static State InitialState(ConsumerController.Start start, Option registering, bool stopping) { return new State(Context.System.DeadLetters, "n/a", start.DeliverTo, 0, 0, 0, diff --git a/src/core/Akka/Delivery/Internal/ProducerControllerImpl.cs b/src/core/Akka/Delivery/Internal/ProducerControllerImpl.cs index cad1f02ac9f..2c4f1b8c7fc 100644 --- a/src/core/Akka/Delivery/Internal/ProducerControllerImpl.cs +++ b/src/core/Akka/Delivery/Internal/ProducerControllerImpl.cs @@ -7,6 +7,7 @@ #nullable enable using System; +using System.Buffers; using System.Collections.Generic; using System.Collections.Immutable; using System.Linq; @@ -741,13 +742,12 @@ private ImmutableList> Chunk(T msg, bool ack, Serialization. internal static IEnumerable CreateChunks(T msg, int chunkSize, Serialization.Serialization serialization) { - var serializer = serialization.FindSerializerFor(msg); - var manifest = Serialization.Serialization.ManifestFor(serializer, msg); - var serializerId = serializer.Identifier; - var bytes = serialization.Serialize(msg); + var writer = new ArrayBufferWriter(); + var metadata = serialization.Serialize(msg, writer); + var bytes = writer.WrittenMemory; if (bytes.Length <= chunkSize) { - var chunkedMessage = new ChunkedMessage(bytes.AsMemory(), true, true, serializerId, manifest); + var chunkedMessage = new ChunkedMessage(bytes, true, true, metadata.SerializerId, metadata.Manifest); yield return chunkedMessage; } else @@ -758,8 +758,8 @@ internal static IEnumerable CreateChunks(T msg, int chunkSize, { var isLast = i == chunkCount - 1; var nextChunk = Math.Min(chunkSize, bytes.Length - i * chunkSize); // needs to be the next chunkSize or remaining bytes, whichever is smaller. - var chunkedMessage = new ChunkedMessage(bytes.AsMemory(i * chunkSize, nextChunk), first, - isLast, serializerId, manifest); + var chunkedMessage = new ChunkedMessage(bytes.Slice(i * chunkSize, nextChunk), first, + isLast, metadata.SerializerId, metadata.Manifest); first = false; yield return chunkedMessage; diff --git a/src/core/Akka/Serialization/ByteArraySerializer.cs b/src/core/Akka/Serialization/ByteArraySerializer.cs index ee1d217ee6b..577e3bd2137 100644 --- a/src/core/Akka/Serialization/ByteArraySerializer.cs +++ b/src/core/Akka/Serialization/ByteArraySerializer.cs @@ -6,15 +6,16 @@ //----------------------------------------------------------------------- using System; +using System.Buffers; using Akka.Actor; namespace Akka.Serialization { /// - /// This is a special that serializes and deserializes byte arrays only - /// (just returns the byte array unchanged/uncopied). + /// This is a special that serializes and deserializes byte arrays only. + /// Legacy byte-array APIs return the supplied array unchanged. /// - public class ByteArraySerializer : Serializer + public class ByteArraySerializer : SerializerV2 { /// /// Initializes a new instance of the class. @@ -26,9 +27,52 @@ public ByteArraySerializer(ExtendedActorSystem system) } /// - /// Returns whether this serializer needs a manifest in the fromBinary method + /// Byte arrays preserve their legacy empty manifest for wire compatibility. /// - public override bool IncludeManifest => false; + public override string Manifest(object obj) + { + return string.Empty; + } + + /// + /// Returns the exact byte array length when is a byte array. + /// + public override int SizeHint(object obj) + { + if (obj == null) + return 0; + if (obj is byte[] bytes) + return bytes.Length; + throw new NotSupportedException("The object to convert is not a byte array."); + } + + /// + /// Serializes the given byte array directly into . + /// + public override int Serialize(object obj, IBufferWriter writer) + { + if (obj == null) + return 0; + + if (obj is not byte[] bytes) + throw new NotSupportedException("The object to convert is not a byte array."); + + if (bytes.Length == 0) + return 0; + + var span = writer.GetSpan(bytes.Length); + bytes.CopyTo(span); + writer.Advance(bytes.Length); + return bytes.Length; + } + + /// + /// Deserializes bytes into a byte array. + /// + public override object Deserialize(ReadOnlySequence bytes, string manifest) + { + return bytes.ToArray(); + } /// /// Serializes the given object into a byte array @@ -57,5 +101,13 @@ public override object FromBinary(byte[] bytes, Type type) { return bytes; } + + /// + /// Deserializes a byte array without copying for legacy compatibility callers. + /// + public override object FromBinary(byte[] bytes, string manifest) + { + return bytes; + } } } diff --git a/src/core/Akka/Serialization/Serialization.cs b/src/core/Akka/Serialization/Serialization.cs index f1fc654e99d..f1abf2f08ae 100644 --- a/src/core/Akka/Serialization/Serialization.cs +++ b/src/core/Akka/Serialization/Serialization.cs @@ -6,6 +6,7 @@ //----------------------------------------------------------------------- using System; +using System.Buffers; using System.Collections.Concurrent; using System.Collections.Generic; using System.Collections.Immutable; @@ -75,6 +76,26 @@ public override int GetHashCode() } } + /// + /// INTERNAL API. + /// + [InternalApi] + internal readonly struct SerializedPayloadMetadata + { + public SerializedPayloadMetadata(int serializerId, string manifest, int bytesWritten) + { + SerializerId = serializerId; + Manifest = manifest; + BytesWritten = bytesWritten; + } + + public int SerializerId { get; } + + public string Manifest { get; } + + public int BytesWritten { get; } + } + /// /// The serialization system used by Akka.NET to serialize and deserialize objects /// per the 's serialization configuration. @@ -90,17 +111,26 @@ public class Serialization /// /// WARNING: if you change this method it's likely that the DaemonMsgCreateSerializer and other calls will need changes too. /// + public static string ManifestFor(SerializerV2 s, object msg) + { + var manifest = s.Manifest(msg); + if (s is not SerializerV1Adapter and not ByteArraySerializer && string.IsNullOrEmpty(manifest)) + throw new SerializationException($"Native SerializerV2 [{s.GetType()}] must return a non-empty manifest for message type [{msg?.GetType()}] unless preserving a legacy serializer ID manifest contract."); + return manifest; + } + + /// + /// Used to determine the manifest for a legacy serializer, if applicable. + /// + /// The serializer we want to use on the message. + /// The message payload. + /// A populated string is applicable; otherwise. public static string ManifestFor(Serializer s, object msg) { - switch (s) - { - case SerializerWithStringManifest s2: - return s2.Manifest(msg); - case Serializer { IncludeManifest: true }: - return msg.GetType().TypeQualifiedName(); - default: - return string.Empty; - } + if (s is SerializerV2 v2) + return ManifestFor(v2, msg); + + return s.Manifest(msg); } /// @@ -141,11 +171,11 @@ public static T WithTransport(ActorSystem system, Address address, Func ac return res; } - private readonly Serializer _nullSerializer; + private readonly SerializerV2 _nullSerializer; - private readonly ConcurrentDictionary _serializerMap = new(); - private readonly Dictionary _serializersById = new(); - private readonly Dictionary _serializersByName = new(); + private readonly ConcurrentDictionary _serializerMap = new(); + private readonly Dictionary _serializersById = new(); + private readonly Dictionary _serializersByName = new(); private readonly ImmutableHashSet _serializerDetails; private readonly MinimalLogger _initializationLogger; @@ -165,7 +195,7 @@ public Serialization(ExtendedActorSystem system) _initializationLogger = system.Settings.StdoutLogger; System = system; - _nullSerializer = new NullSerializer(system); + _nullSerializer = AdaptSerializer(new NullSerializer(system)); AddSerializer("null", _nullSerializer); @@ -192,17 +222,17 @@ public Serialization(ExtendedActorSystem system) var serializerConfig = serializerSettingsConfig.GetConfig(kvp.Key); var serializer = !serializerConfig.IsNullOrEmpty() - ? (Serializer)Activator.CreateInstance(serializerType, system, serializerConfig) - : (Serializer)Activator.CreateInstance(serializerType, system); + ? Activator.CreateInstance(serializerType, system, serializerConfig) + : Activator.CreateInstance(serializerType, system); - AddSerializer(kvp.Key, serializer); + AddSerializer(kvp.Key, AdaptSerializer(serializer)); } // Add any serializers that are registered via the SerializationSetup // This has to be done here because SerializationSetup ALWAYS win. foreach (var details in _serializerDetails) { - AddSerializer(details.Alias, details.Serializer); + AddSerializer(details.Alias, details.SerializerV2); } foreach (var kvp in serializerBindingConfig) @@ -233,7 +263,7 @@ public Serialization(ExtendedActorSystem system) // populate the serialization map foreach (var t in details.UseFor) { - AddSerializationMap(t, details.Serializer); + AddSerializationMap(t, details.SerializerV2); } } } @@ -317,12 +347,12 @@ private T WithTransport(Func action) } } - private Serializer GetSerializerByName(string name) + private SerializerV2 GetSerializerByName(string name) { if (name == null) return null; - _serializersByName.TryGetValue(name, out Serializer serializer); + _serializersByName.TryGetValue(name, out SerializerV2 serializer); return serializer; } @@ -331,6 +361,22 @@ private Serializer GetSerializerByName(string name) /// public ExtendedActorSystem System { get; } + internal static SerializerV2 AdaptSerializer(object serializer) + { + return serializer switch + { + SerializerV2 v2 => v2, + Serializer v1 => new SerializerV1Adapter(v1.ExtendedSystem, v1), + null => throw new ArgumentNullException(nameof(serializer)), + _ => throw new ArgumentException($"Serializer [{serializer.GetType()}] must inherit from [{typeof(Serializer)}] or [{typeof(SerializerV2)}].", nameof(serializer)) + }; + } + + private static Serializer AsPublicSerializer(SerializerV2 serializer) + { + return serializer is SerializerV1Adapter adapter ? adapter.Inner : serializer; + } + /// /// Adds the serializer to the internal state of the serialization subsystem /// @@ -338,6 +384,17 @@ private Serializer GetSerializerByName(string name) [Obsolete("No longer supported. Use the AddSerializer(name, serializer) overload instead.", true)] [MethodImpl(MethodImplOptions.AggressiveInlining)] public void AddSerializer(Serializer serializer) + { + AddSerializer(AdaptSerializer(serializer)); + } + + /// + /// Adds the serializer to the internal state of the serialization subsystem + /// + /// Serializer instance + [Obsolete("No longer supported. Use the AddSerializer(name, serializer) overload instead.", true)] + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public void AddSerializer(SerializerV2 serializer) { var id = serializer.Identifier; if(_logSerializerOverrideOnStart && _serializersById.ContainsKey(id) && _serializersById[id].GetType() != serializer.GetType()) @@ -357,6 +414,16 @@ public void AddSerializer(Serializer serializer) /// Configuration name of the serializer /// Serializer instance public void AddSerializer(string name, Serializer serializer) + { + AddSerializer(name, AdaptSerializer(serializer)); + } + + /// + /// Adds the serializer to the internal state of the serialization subsystem + /// + /// Configuration name of the serializer + /// Serializer instance + public void AddSerializer(string name, SerializerV2 serializer) { var id = serializer.Identifier; if(_logSerializerOverrideOnStart && _serializersById.ContainsKey(id) && _serializersById[id].GetType() != serializer.GetType()) @@ -385,6 +452,18 @@ public void AddSerializer(string name, Serializer serializer) /// TBD [MethodImpl(MethodImplOptions.AggressiveInlining)] public void AddSerializationMap(Type type, Serializer serializer) + { + AddSerializationMap(type, AdaptSerializer(serializer)); + } + + /// + /// TBD + /// + /// TBD + /// TBD + /// TBD + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public void AddSerializationMap(Type type, SerializerV2 serializer) { if(_logSerializerOverrideOnStart && _serializerMap.ContainsKey(type) && _serializerMap[type].GetType() != serializer.GetType()) LogWarning( @@ -443,6 +522,23 @@ public byte[] Serialize(object o) return WithTransport(() => FindSerializerFor(o).ToBinary(o)); } + /// + /// Serializes the given message into using the V2 serializer contract. + /// + internal SerializedPayloadMetadata Serialize(object o, IBufferWriter writer) + { + if (writer == null) + throw new ArgumentNullException(nameof(writer)); + + return WithTransport(() => + { + var serializer = FindSerializerV2For(o); + var manifest = ManifestFor(serializer, o); + var bytesWritten = serializer.Serialize(o, writer); + return new SerializedPayloadMetadata(serializer.Identifier, manifest, bytesWritten); + }); + } + /// /// Deserializes the given array of bytes using the specified serializer id, using the optional type hint to the Serializer. /// @@ -494,22 +590,34 @@ public object Deserialize(byte[] bytes, int serializerId, string manifest) if (oldInfo == null) Serialization.CurrentTransportInformation = SerializationInfo; - if (serializer is SerializerWithStringManifest stringManifest) - return stringManifest.FromBinary(bytes, manifest); if (string.IsNullOrEmpty(manifest)) - return serializer.FromBinary(bytes, null); - Type type; - try - { - type = TypeCache.GetType(manifest); - } - catch (Exception ex) - { - throw new SerializationException( - $"Cannot find manifest class [{manifest}] for serializer with id [{serializerId}].", ex); - } + return serializer.FromBinary(bytes, string.Empty); + return serializer.FromBinary(bytes, manifest); + } + finally + { + Serialization.CurrentTransportInformation = oldInfo; + } + } - return serializer.FromBinary(bytes, type); + /// + /// Deserializes the given bytes using the V2 serializer contract. + /// + internal object Deserialize(ReadOnlySequence bytes, int serializerId, string manifest) + { + if (!_serializersById.TryGetValue(serializerId, out var serializer)) + throw new SerializationException( + $"Cannot find serializer with id [{serializerId}] (manifest [{manifest}]). The most probable reason" + + " is that the configuration entry 'akka.actor.serializers' is not in sync between the two systems." + + $" {Serializer.GetErrorForSerializerId(serializerId)}"); + + var oldInfo = Serialization.CurrentTransportInformation; + try + { + if (oldInfo == null) + Serialization.CurrentTransportInformation = SerializationInfo; + + return serializer.Deserialize(bytes, manifest ?? string.Empty); } finally { @@ -525,7 +633,12 @@ public object Deserialize(byte[] bytes, int serializerId, string manifest) /// The serializer configured for the given object type public Serializer FindSerializerFor(object obj, string defaultSerializerName = null) { - return obj == null ? _nullSerializer : FindSerializerForType(obj.GetType(), defaultSerializerName); + return AsPublicSerializer(FindSerializerV2For(obj, defaultSerializerName)); + } + + internal SerializerV2 FindSerializerV2For(object obj, string defaultSerializerName = null) + { + return obj == null ? _nullSerializer : FindSerializerV2ForType(obj.GetType(), defaultSerializerName); } //cache to eliminate lots of typeof operator calls @@ -545,11 +658,16 @@ public Serializer FindSerializerFor(object obj, string defaultSerializerName = n /// /// The serializer configured for the given object type public Serializer FindSerializerForType(Type objectType, string defaultSerializerName = null) + { + return AsPublicSerializer(FindSerializerV2ForType(objectType, defaultSerializerName)); + } + + internal SerializerV2 FindSerializerV2ForType(Type objectType, string defaultSerializerName = null) { if (_serializerMap.TryGetValue(objectType, out var fullMatchSerializer)) return fullMatchSerializer; - Serializer serializer = null; + SerializerV2 serializer = null; Type type = objectType; // TODO: see if we can do a better job with proper type sorting here - most specific to least specific (object serializer goes last) @@ -657,7 +775,7 @@ public static string SerializedActorPath(IActorRef actorRef) internal Serializer GetSerializerById(int serializerId) { - return _serializersById[serializerId]; + return AsPublicSerializer(_serializersById[serializerId]); } } } diff --git a/src/core/Akka/Serialization/SerializationSetup.cs b/src/core/Akka/Serialization/SerializationSetup.cs index e1ad224bc15..63d814475d3 100644 --- a/src/core/Akka/Serialization/SerializationSetup.cs +++ b/src/core/Akka/Serialization/SerializationSetup.cs @@ -41,7 +41,7 @@ public static SerializationSetup Create( /// /// Constructor is internal API. /// - /// Use the factory method instead + /// Use one of the Create factory methods instead /// public sealed class SerializerDetails { @@ -49,9 +49,15 @@ internal SerializerDetails(string @alias, Serializer serializer, ImmutableHashSe { Alias = alias; Serializer = serializer; + SerializerV2 = Serialization.AdaptSerializer(serializer); UseFor = useFor; } + internal SerializerDetails(string @alias, SerializerV2 serializer, ImmutableHashSet useFor) + : this(alias, (Serializer)serializer, useFor) + { + } + /// /// The name of the serializer - must be unique. /// @@ -66,6 +72,8 @@ internal SerializerDetails(string @alias, Serializer serializer, ImmutableHashSe /// public Serializer Serializer { get; } + internal SerializerV2 SerializerV2 { get; } + /// /// The types of messages that this /// @@ -82,6 +90,18 @@ public static SerializerDetails Create(string alias, Serializer serializer, Immu { return new SerializerDetails(alias, serializer, useFor); } + + /// + /// Factory method for creating programmatic setups for Serializers. + /// + /// Register the serializer under this alias (this allows it to be used by bindings in the config) + /// The serializer implementation. + /// A set of types (classes, base classes, or interfaces) that will be bound to this serializer. + /// This is the programmatic equivalent of the `akka.actor.serialization.serialization-bindings` HOCON section. + public static SerializerDetails Create(string alias, SerializerV2 serializer, ImmutableHashSet useFor) + { + return new SerializerDetails(alias, serializer, useFor); + } } } diff --git a/src/core/Akka/Serialization/Serializer.cs b/src/core/Akka/Serialization/Serializer.cs index 183f9e2d020..b7e9a187233 100644 --- a/src/core/Akka/Serialization/Serializer.cs +++ b/src/core/Akka/Serialization/Serializer.cs @@ -14,6 +14,7 @@ using Akka.Annotations; using Akka.Util; using Akka.Configuration; +using Akka.Util.Reflection; namespace Akka.Serialization { @@ -45,6 +46,8 @@ public abstract class Serializer /// protected readonly ExtendedActorSystem system; + internal ExtendedActorSystem ExtendedSystem => system; + private readonly FastLazy _value; /// @@ -75,6 +78,14 @@ protected Serializer(ExtendedActorSystem system) /// A byte array containing the serialized object public abstract byte[] ToBinary(object obj); + /// + /// Returns the manifest that should be stored with . + /// + public virtual string Manifest(object obj) + { + return IncludeManifest ? obj.GetType().TypeQualifiedName() : string.Empty; + } + /// /// Serializes the given object into a byte array and uses the given address to decorate serialized ActorRef's /// @@ -96,6 +107,27 @@ public byte[] ToBinaryWithAddress(Address address, object obj) /// The object contained in the array public abstract object FromBinary(byte[] bytes, Type type); + /// + /// Deserializes a byte array into an object using a string manifest. + /// + public virtual object FromBinary(byte[] bytes, string manifest) + { + if (string.IsNullOrEmpty(manifest)) + return FromBinary(bytes, (Type)null); + + Type type; + try + { + type = TypeCache.GetType(manifest); + } + catch (Exception ex) + { + throw new SerializationException($"Cannot find manifest class [{manifest}] for serializer with id [{Identifier}].", ex); + } + + return FromBinary(bytes, type); + } + /// /// Deserializes a byte array into an object. /// @@ -158,7 +190,7 @@ public sealed override object FromBinary(byte[] bytes, Type type) /// The array containing the serialized object /// The type hint used to deserialize the object contained in the array. /// The object contained in the array - public abstract object FromBinary(byte[] bytes, string manifest); + public abstract override object FromBinary(byte[] bytes, string manifest); /// /// Returns the manifest (type hint) that will be provided in the method. @@ -169,7 +201,7 @@ public sealed override object FromBinary(byte[] bytes, Type type) /// /// The object for which the manifest is needed. /// The manifest needed for the deserialization of the specified . - public abstract string Manifest(object o); + public abstract override string Manifest(object o); } /// @@ -206,4 +238,3 @@ public static int GetSerializerIdentifierFromConfig(Type type, ExtendedActorSyst } } } - diff --git a/src/core/Akka/Serialization/SerializerV1Adapter.cs b/src/core/Akka/Serialization/SerializerV1Adapter.cs new file mode 100644 index 00000000000..d591bf970d8 --- /dev/null +++ b/src/core/Akka/Serialization/SerializerV1Adapter.cs @@ -0,0 +1,106 @@ +//----------------------------------------------------------------------- +// +// Copyright (C) 2009-2022 Lightbend Inc. +// Copyright (C) 2013-2025 .NET Foundation +// +//----------------------------------------------------------------------- + +#nullable enable +using System; +using System.Buffers; +using System.Runtime.Serialization; +using Akka.Actor; +using Akka.Util; +using Akka.Util.Reflection; + +namespace Akka.Serialization +{ + /// + /// Adapts legacy implementations to the contract. + /// + public sealed class SerializerV1Adapter : SerializerV2 + { + /// + /// Initializes a new instance of the class. + /// + public SerializerV1Adapter(ExtendedActorSystem system, Serializer inner) : base(system) + { + Inner = inner ?? throw new ArgumentNullException(nameof(inner)); + } + + /// + /// The wrapped V1 serializer. + /// + public Serializer Inner { get; } + + /// + public override int Identifier => Inner.Identifier; + + /// + public override string Manifest(object obj) + { + return Inner switch + { + SerializerWithStringManifest stringManifest => stringManifest.Manifest(obj), + { IncludeManifest: true } => obj.GetType().TypeQualifiedName(), + _ => string.Empty + }; + } + + /// + public override int SizeHint(object obj) => UnknownSize; + + /// + public override int Serialize(object obj, IBufferWriter writer) + { + var bytes = Inner.ToBinary(obj) ?? Array.Empty(); + if (bytes.Length == 0) + return 0; + + var span = writer.GetSpan(bytes.Length); + bytes.CopyTo(span); + writer.Advance(bytes.Length); + return bytes.Length; + } + + /// + public override object Deserialize(ReadOnlySequence bytes, string manifest) + { + return FromBinary(bytes.ToArray(), manifest); + } + + /// + public override byte[] ToBinary(object obj) + { + return Inner.ToBinary(obj); + } + + /// + public override object FromBinary(byte[] bytes, string manifest) + { + if (Inner is SerializerWithStringManifest stringManifest) + return stringManifest.FromBinary(bytes, manifest); + + if (string.IsNullOrEmpty(manifest)) + return Inner.FromBinary(bytes, (Type)null!); + + Type type; + try + { + type = TypeCache.GetType(manifest); + } + catch (Exception ex) + { + throw new SerializationException($"Cannot find manifest class [{manifest}] for serializer with id [{Identifier}].", ex); + } + + return Inner.FromBinary(bytes, type); + } + + /// + public override object FromBinary(byte[] bytes, Type type) + { + return Inner.FromBinary(bytes, type); + } + } +} diff --git a/src/core/Akka/Serialization/SerializerV2.cs b/src/core/Akka/Serialization/SerializerV2.cs new file mode 100644 index 00000000000..cbc65d225e5 --- /dev/null +++ b/src/core/Akka/Serialization/SerializerV2.cs @@ -0,0 +1,88 @@ +//----------------------------------------------------------------------- +// +// Copyright (C) 2009-2022 Lightbend Inc. +// Copyright (C) 2013-2025 .NET Foundation +// +//----------------------------------------------------------------------- + +#nullable enable +using System; +using System.Buffers; +using Akka.Actor; +using Akka.Util; + +namespace Akka.Serialization +{ + /// + /// A serializer that writes directly into caller-owned buffers and reads from sequence-backed input. + /// + public abstract class SerializerV2 : Serializer + { + /// + /// Returned by when the serialized size cannot be cheaply predicted. + /// + public const int UnknownSize = -1; + + /// + /// Initializes a new instance of the class. + /// + /// The actor system to associate with this serializer. + protected SerializerV2(ExtendedActorSystem system) : base(system) + { + } + + /// + /// V2 serializers use manifest-aware dispatch. New V2 serializers should emit + /// non-empty manifests; legacy serializer-id ports may preserve empty manifests. + /// + public sealed override bool IncludeManifest => true; + + /// + /// Returns the manifest used by this serializer for . + /// + public abstract override string Manifest(object obj); + + /// + /// Returns a best-effort serialized size hint, or when unknown. + /// + public virtual int SizeHint(object obj) => UnknownSize; + + /// + /// Serializes into . + /// + /// The number of payload bytes written. + public abstract int Serialize(object obj, IBufferWriter writer); + + /// + /// Deserializes using a serializer manifest. + /// + public abstract object Deserialize(ReadOnlySequence bytes, string manifest); + + /// + /// Serializes the given object into a byte array for compatibility boundaries. + /// + public override byte[] ToBinary(object obj) + { + var sizeHint = SizeHint(obj); + var writer = sizeHint > 0 ? new ArrayBufferWriter(sizeHint) : new ArrayBufferWriter(); + Serialize(obj, writer); + return writer.WrittenMemory.ToArray(); + } + + /// + /// Deserializes a byte array using a serializer manifest. + /// + public override object FromBinary(byte[] bytes, string manifest) + { + return Deserialize(new ReadOnlySequence(bytes), manifest); + } + + /// + /// Deserializes a byte array using a type manifest. + /// + public override object FromBinary(byte[] bytes, Type type) + { + return FromBinary(bytes, type?.TypeQualifiedName() ?? string.Empty); + } + } +} diff --git a/src/examples/Akka.Persistence.Custom/Journal/SqliteJournal.cs b/src/examples/Akka.Persistence.Custom/Journal/SqliteJournal.cs index a4f9ee50ae2..9c252f05f93 100644 --- a/src/examples/Akka.Persistence.Custom/Journal/SqliteJournal.cs +++ b/src/examples/Akka.Persistence.Custom/Journal/SqliteJournal.cs @@ -339,20 +339,8 @@ protected sealed override async Task> WriteMessagesAsy action: state => { var (thePayload, theSerializer) = state; - var thisManifest = ""; - - // There are two kinds of serializer when it comes to manifest - // support, we have to support both of them for proper payload - // serialization - if (theSerializer is SerializerWithStringManifest stringManifest) - { - thisManifest = stringManifest.Manifest(thePayload); - } - else if (theSerializer.IncludeManifest) - { - thisManifest = thePayload.GetType().TypeQualifiedName(); - } - + var thisManifest = Akka.Serialization.Serialization.ManifestFor(theSerializer, thePayload); + // Return the serialized byte array and the manifest for the // serialized data return (theSerializer.ToBinary(thePayload), thisManifest); diff --git a/src/examples/Akka.Persistence.Custom/Snapshot/SqliteSnapshotStore.cs b/src/examples/Akka.Persistence.Custom/Snapshot/SqliteSnapshotStore.cs index ab143f7e5a3..df48be7e577 100644 --- a/src/examples/Akka.Persistence.Custom/Snapshot/SqliteSnapshotStore.cs +++ b/src/examples/Akka.Persistence.Custom/Snapshot/SqliteSnapshotStore.cs @@ -266,20 +266,8 @@ protected sealed override async Task SaveAsync( action: state => { var (thePayload, theSerializer) = state; - var thisManifest = ""; - - // There are two kinds of serializer when it comes to manifest - // support, we have to support both of them for proper payload - // serialization - if (theSerializer is SerializerWithStringManifest stringManifest) - { - thisManifest = stringManifest.Manifest(thePayload); - } - else if (theSerializer.IncludeManifest) - { - thisManifest = thePayload.GetType().TypeQualifiedName(); - } - + var thisManifest = Akka.Serialization.Serialization.ManifestFor(theSerializer, thePayload); + // Return the serialized byte array and the manifest for the // serialized data return (theSerializer.ToBinary(thePayload), thisManifest);