akka.cluster.sharding - storage api update by zbynek001 · Pull Request #4629 · akkadotnet/akka.net

zbynek001 · 2020-11-12T15:42:27Z

Another big one ^^
This MR is updating sharding to current scala version, including unit tests and multinode tests.

What's still missing compared to scala:

~~ExternalShardAllocation~~
RemoveInternalClusterShardingData app

There were quite many changes related to the storage api change, so basically some parts were completely rewritten. Especially Shard and ShardCoordinator.

Also includes some pre-requirements to make some part easier, especially MessageBuffer and Dropped deadletter. Not sure if you'll like it.

~~Tests are now using sqlite persistence provider~~
Tests are now using shared inmem persistence providers for both journal & snapshots to avoid issues with files

In theory, everything "should" be backwards compatible

Additionally included in this pr:

Race condition in DeprecatedSupervisionSpec fixed (migrated from Race condition in DeprecatedSupervisionSpec fixed akka/akka-core#29914)
Small clarification of recovery strategy in config (migrated from Small clarification of recovery strategy in config akka/akka-core#30167)
Resolve snapshot check skipped for some events (migrated from Resolve snapshot check skipped for some events (#30225) akka/akka-core#30226)
Defer coordinator stop until region graceful stop has completed (migrated from fix: Defer coordinator stop until region graceful stop has completed … akka/akka-core#30338)
sharding: actively signal 'region stopped' to the coordinator (migrated from sharding: actively signal 'region stopped' to the coordinator akka/akka-core#30402)

zbynek001 · 2020-11-18T11:08:21Z

This should be ready to review.
Lots of changes, but it brings sharding up-to date with latest scala version.
Persistent mode was split into two parts:

CoordinatorStore
ShardStore

which can use different persistent mode configured via state-store-mode & remember-entities-store.
Possible combinations:

state-store-mode	remember-entities-store	CoordinatorStore mode	ShardStore mode
persistence (default)	- (ignored)	persistence	persistence
ddata	ddata	ddata	ddata
ddata	eventsourced (new)	ddata	persistence

There should be no breaking changes from user perspective. Only some internal messages/objects were moved.
There should be no change in the PersistentId behavior and default persistent configuration (akka.cluster.sharding.state-store-mode)

Log Dropped from DeadLetterListener

* make use of the existing logging of dead letter also for UnhandledMessage Supress ActorSelectionMessage with DeadLetterSuppression (migrated from akka/akka-core#28341) * for example the Cluster InitJoin message is marked with DeadLetterSuppression but was anyway logged because sent with actorSelection * for other WrappedMessage than ActorSelectionMessage we shouldn't unwrap and publish the inner in SuppressedDeadLetter because that might loose some information * therefore those are silenced in the DeadLetterListener instead Better deadLetter logging of wrapped messages (migrated from akka/akka-core#28253)

Aaronontheweb · 2020-12-15T18:11:50Z

I'm going to push a 1.4.13 release without this change for now, but I have it assigned to myself to review it as part of our current sprint. This will make it into a subsequent v1.4.14 or later release.

Aaronontheweb

Made it through 38/104 files - going to work on the rest tomorrow. Liking what I'm seeing so far but it's only been a review of the tests.

Aaronontheweb · 2022-02-10T02:47:13Z

+
+namespace Akka.Cluster.Sharding.Tests
+{
+    public class ClusterShardCoordinatorDowningSpecConfig : MultiNodeClusterShardingConfig


Liking the common base class for testing sharding. Makes a lot of sense that we add that.

Aaronontheweb · 2022-02-10T03:01:00Z

+            }
+            # don't leak ddata state across runs
+            akka.cluster.sharding.distributed-data.durable.keys = []
+            akka.persistence.journal.leveldb-shared.store.native = off


Is this a JVM setting? Pretty sure we don't have any LevelDb support.

Aaronontheweb · 2022-02-10T03:03:41Z

+            EnterBarrier("received failed stats from timed out shards vs empty");
+        }
+
+        private void Querying_cluster_sharding_must_return_shard_state_of_sharding_regions_if_one_or_more_shards_timeout_versus_all_as_empty()


Nice - I've run into this issue before with pbm cluster-sharding region stats -r {regionName} before; glad this has been fixed.

Aaronontheweb · 2022-02-10T03:16:39Z

-                """);
+                }
+
+                akka.persistence.snapshot-store.plugin = ""akka.persistence.memory-snapshot-store-shared""


Aaronontheweb · 2022-02-10T03:21:19Z

+        }
+
+        [Fact]
+        public void ClusterShardingMessageSerializer_must_serialize_ShardRegionQuery()


LGTM - all of the new tests in this class look fine, but I haven't checked the underlying serializer impl itself which is where I know @Arkatufus raised some points a while back.

Aaronontheweb · 2022-02-10T03:36:17Z

Akka.Cluster.Sharding.Tests.RememberEntitiesShardIdExtractorChangeSpec.Sharding_with_remember_entities_enabled_should_allow_a_change_to_the_shard_id_extractor

some of these specs are failing - we'll investigate.

…stence

Aaronontheweb

65/104

Aaronontheweb

70/104

Aaronontheweb · 2022-02-15T14:34:11Z

-        private readonly bool _rememberEntities;
+        private sealed class RememberEntitiesStoreStopped
+        {
+            public static RememberEntitiesStoreStopped Instance = new RememberEntitiesStoreStopped();


need to make this readonly (nitpick)

Aaronontheweb · 2022-02-15T14:35:04Z

+                replicator,
+                majorityMinCap,
+                rememberEntitiesStoreProvider))
+                .WithDeploy(Deploy.Local);


That'll make sure that we don't end up trying to serialize this actor when serialize-creators=on - good catch

Aaronontheweb · 2022-02-15T14:36:16Z

+        private bool _terminating = false;
+        private readonly UniqueAddress _selfUniqueAddress;
+        private readonly LWWRegisterKey<ShardCoordinator.CoordinatorState> _coordinatorStateKey;
+        private ImmutableHashSet<(IActorRef, GetShardHome)> _getShardHomeRequests = ImmutableHashSet<(IActorRef, GetShardHome)>.Empty;


Does GetShardHome contain a "by value" GetHashCode implementation? Otherwise this might be problematic

Looks like we do

akka.net/src/contrib/cluster/Akka.Cluster.Sharding/PersistentShardCoordinator.cs

Lines 445 to 451 in 21762b9

public override int GetHashCode()

{

unchecked

{

return Shard?.GetHashCode() ?? 0;

}

}

Aaronontheweb · 2022-02-15T14:40:22Z

+                    case GetFailure m when m.Key.Equals(_coordinatorStateKey):
+                        _initialStateRetries++;
+                        var template =
+                          "{0}: The ShardCoordinator was unable to get an initial state within 'waiting-for-state-timeout': {1} millis (retrying). Has ClusterSharding been started on all nodes?";


Compiler might do this on the fly anyway but might as well declare a const here

Aaronontheweb · 2022-02-15T14:46:23Z

+
+namespace Akka.Cluster.Sharding.External
+{
+    public class ClientTimeoutException : Exception


Need to add some XML-DOC comments here to explain how / why this error is raised

Aaronontheweb · 2022-02-15T14:52:49Z

+    /// <summary>
+    /// API May Change
+    /// </summary>
+    public sealed class ExternalShardAllocation : IExtension


Suggested change

public sealed class ExternalShardAllocation : IExtension

[ApiMayChange]

public sealed class ExternalShardAllocation : IExtension

Aaronontheweb · 2022-02-15T14:53:27Z

+using System;
+using System.Collections.Concurrent;
+using Akka.Actor;
+


Suggested change

using Akka.Annotations;

Aaronontheweb · 2022-02-15T14:54:23Z

+        }
+
+        /// <summary>
+        /// uses a string primitive types are optimized in ddata to not serialize every entity separately


Aaronontheweb · 2022-02-15T15:01:18Z

+            return new LWWDictionaryKey<ShardId, string>($"external-sharding-{typeName}");
+        }
+
+        private class DDataStateActor : ActorBase, IWithUnboundedStash


I'd appreciate an internal comment here explaining this actor's role in the design, as that may not be obvious to the next person who has to work on this

Aaronontheweb · 2022-02-15T15:49:11Z

+{
+    using ShardId = String;
+
+    public class ExternalShardAllocationStrategy : IStartableAllocationStrategy


Does this need to be sealed so we don't end up with multiple GetShardLocation implementations?

Aaronontheweb · 2022-02-15T15:56:49Z

Still going through this file by file, but I've left another review

Aaronontheweb

Fixed the Akka.Annotations usage here

Aaronontheweb · 2022-03-04T12:35:09Z

+{
+    using ShardId = String;
+
+    public sealed class ShardLocations


Aaronontheweb

I need to double check the protobuf compatibility, but otherwise this looks fine - I'm on the fence about doing this in v1.4. Might need to pull it in as part of v1.5.

Aaronontheweb · 2022-03-04T13:11:53Z

+        {
+            _system = system;
+            _typeName = typeName;
+            _log = Logging.GetLogger(system, GetType());


Might not be a bad idea to enrich this logger name with the _typeName

Aaronontheweb · 2022-03-04T13:13:46Z

+    using ShardId = String;
+
+    /// <summary>
+    /// Only intended for testing, not an extension point.


Got it, thank you

Aaronontheweb · 2022-03-04T13:16:38Z

+            _replicator.Tell(Dsl.Get(_allShardsKey, _readMajority));
+        }
+
+        protected override bool Receive(object message)


Aaronontheweb · 2022-03-04T13:17:35Z

+        /// By splitting the elements over 5 keys we can support 10000 entities per shard.
+        /// The Gossip message size of 5 ORSet with 2000 ids is around 200 KiB.
+        /// This is by intention not configurable because it's important to have the same
+        /// configuration on each node.


We need to benchmark our DData payload size to make sure that this is true in Akka.NET as well

Aaronontheweb · 2022-03-04T13:36:37Z

+                WrittenMigrationMarker = writtenMigrationMarker;
+            }
+
+            public IImmutableSet<string> Shards { get; }


Should probably also be typed ShardId

Aaronontheweb · 2022-03-04T14:36:54Z

+        {
+            return new ShardsQueryResult<T>(
+                failed: ps.Where(i => i.task.IsCanceled || i.task.IsFaulted).Select(i => i.shard).ToImmutableHashSet(),
+                responses: ps.Where(i => !i.task.IsCanceled && !i.task.IsFaulted).Select(i => i.task.Result).ToImmutableList(),


Wondering about the Task.Result call here - I'll need to use the IDE for this, but I'd prefer it if we just passed the results of the task into this method rather than the Task themselves. Difficult to tell from the PR review dialog.

Aaronontheweb · 2022-03-04T14:37:59Z

@@ -21,6 +21,28 @@ namespace Akka.TestKit.Xunit2
    /// </summary>
    public class TestKit : TestKitBase , IDisposable
    {
+        private class PrefixedOutput : ITestOutputHelper


This is a great idea.

Aaronontheweb · 2022-03-04T14:39:04Z

+  # When 'remember-entities' is enabled and the state store mode is ddata this controls
+  # how the remembered entities and shards are stored. Possible values are "eventsourced" and "ddata"
+  # Default is ddata for backwards compatibility.
+  remember-entities-store = "ddata"


Makes sense

Aaronontheweb · 2022-03-04T14:42:08Z

 }

-message ShardRegionWithAddress{


I don't know enough about Protobuf here to say for certain if renaming a .proto file is a breaking change or not. I'm going to need to do some manual testing with these changes anyway.

Aaronontheweb · 2022-04-20T18:24:25Z

Merged via #5857

zbynek001 force-pushed the sharding-update2 branch 6 times, most recently from dafef2f to 05a658c Compare November 14, 2020 21:57

zbynek001 changed the title ~~[WIP] akka.cluster.sharding - storage api update~~ akka.cluster.sharding - storage api update Nov 18, 2020

zbynek001 force-pushed the sharding-update2 branch from 765616c to 1956620 Compare November 18, 2020 10:58

Aaronontheweb self-requested a review November 18, 2020 17:42

zbynek001 added 18 commits November 25, 2020 15:42

Add Dropped to Akka.Actor (migrated partially from akka/akka-core#27160)

9919ddb

Log Dropped from DeadLetterListener

MessageBuffer implementations

220dd0b

TestKit logger with prefix

c2d789f

sharding update

cc80e4b

sharding tests

55b4537

sharding multinode tests

ba28c66

api approval

5cffe62

replace sqlite with MemoryJournalShared and local snapshot store

df19783

tests

fb6ecc8

snapshot inmem

5cb90af

backwards compatible PersistenceId for PersistentShardCoordinator

ca3a3fa

test fix

4d67fc2

SnapshotStoreProxy & MemorySnapshotStoreShared

2547e32

test snapshot store switched to shared inmem

a0defd1

ExternalShardAllocationStrategy & tests

02e2994

ExternalShardAllocationStrategy API approval

2bb32cb

test timing fix

aee94d3

zbynek001 force-pushed the sharding-update2 branch from 1956620 to aee94d3 Compare December 1, 2020 08:47

fixed compilation error from rebase

03abd3d

Aaronontheweb reviewed Feb 10, 2022

View reviewed changes

zbynek001 added 3 commits February 10, 2022 13:40

switch RememberEntitiesShardIdExtractorChangeSpec from ddata to persi…

37af7c2

…stence

disable durable storage on PersistentShardingMigrationSpec

645e3f6

clean up leveldb configuration

2bfaf8d

Aaronontheweb mentioned this pull request Feb 10, 2022

replaced cref="..." with cref="@...` #5623

Closed

Aaronontheweb reviewed Feb 10, 2022

View reviewed changes

Aaronontheweb modified the milestones: 1.4.33, 1.4.34 Feb 14, 2022

Aaronontheweb reviewed Feb 15, 2022

View reviewed changes

Merge branch 'dev' into sharding-update2

56304f9

Aaronontheweb reviewed Mar 4, 2022

View reviewed changes

Aaronontheweb approved these changes Mar 4, 2022

View reviewed changes

Aaronontheweb modified the milestones: 1.4.34, 1.4.35 Mar 7, 2022

Aaronontheweb modified the milestones: 1.4.35, 1.4.36 Mar 18, 2022

Arkatufus added 2 commits April 5, 2022 22:03

Merge branch 'dev' into sharding-update2

ae4750b

Merge branch 'dev' into sharding-update2

c416295

Aaronontheweb modified the milestones: 1.4.36, 1.4.37, 1.4.38, 1.5.0 Apr 14, 2022

Aaronontheweb mentioned this pull request Apr 20, 2022

Zbynek001 sharding update2 #5857

Merged

Aaronontheweb closed this Apr 20, 2022

	public override int GetHashCode()
	{
	unchecked
	{
	return Shard?.GetHashCode() ?? 0;
	}
	}

	public sealed class ExternalShardAllocation : IExtension
	[ApiMayChange]
	public sealed class ExternalShardAllocation : IExtension

Conversation

zbynek001 commented Nov 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zbynek001 commented Nov 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Aaronontheweb commented Dec 15, 2020

Uh oh!

Aaronontheweb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Aaronontheweb commented Feb 10, 2022

Uh oh!

Aaronontheweb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Aaronontheweb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Aaronontheweb commented Feb 15, 2022

Uh oh!

Aaronontheweb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Aaronontheweb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zbynek001 commented Nov 12, 2020 •

edited

Loading

zbynek001 commented Nov 18, 2020 •

edited

Loading