How does shard rebalancing work in a cluster according to Deployment Strategy in Kubernetes?

[civilizeddev]

Hi, @Tochemey

Today, I have some questions about how the cluster and shards behave when there is a change in the members that make up the cluster.

There are mainly two deployment strategies that can be used to update apps in a cluster:

• Rolling Update
• Blue/Green Update

In the following case, when the cluster detects changes in membership and rebalances shards,

At such moments, it feels like there might be sudden fluctuations. I'm also curious if this leads to a chain of actor termination and respawn among shards.

Is the recommended deployment strategy in this case likely to be Blue/Green Update?

Thank you.

[Tochemey]

@civilizeddev Thanks for your question. Let me explain the current design:

• The cluster implementation simply relies on a distributed sharded key/value store engine. It handles the rebalance and the sharding mechanism.

• A single instance of an actor is created when cluster mode is enabled. The actor identifier is propagated across the cluster.

• Each node in the cluster has a copy of the created actor identifier and can easily refers to the location of the actor. Now that I am explaining this I just discover an issue :). When the node where the given actor is created dies, then the other nodes in the cluster are aware of the event. However, at the moment, Go-Akt is not doing actor redeployment which is the issue we need to address. It is an issue because that actor cannot be found when its node dies.

I hope I clarify the cluster implementation a bit.

I have created an issue for that #192

[civilizeddev]

@Tochemey

In #192 (comment)

Are you saying that you're considering abandoning the use of Olric as a cluster engine and looking into Raft as an alternative? I'm not really familiar with either. What are the shortcomings that necessitate a replacement? If you could explain it simply, I think I could learn.

[Tochemey]

@civilizeddev
The current cluster implementation does work as expected. One can access actors from any node as long as the actor is alive or the node hosting the actor is not dead(a.k.a left the cluster). Therefore to tackle actors' redeployment when the hosting node is dead, I am saying I may consider other technologies or protocols in case Olric does not meet the required needs. This is still WIP.
Olric is a distributed cluster in-memory store. You can read more on their readme.
Regarding the raft protocol, I will suggest this lovely Wikipedia page. In a nutshell, it is a protocol that helps build distributed and clustered-based applications.