HDDS-5148. Bump ratis version to 2.1.0-ff8aa66-SNAPSHOT

[GlenGeng-awx]

What changes were proposed in this pull request?

Bump ratis version to 2.1.0-ff8aa66-SNAPSHOT

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-5148

How was this patch tested?

CI

[GlenGeng-awx]

cc @runzhiwang Please take a look at this PR. Thanks !

[bharatviswa504]

+1.

[elek]

I started a discussion on the mailing list and would like to continue the discussion o community meeting.

I propose to avoid any usage of SNAPSHOT Ratis versions as last release is delayed because of similar reasons.

Releasing 2.0.1 from Ratis seems to be very easy as Ratis is TLP.

[elek]

Let me make it clear: It's not a blocking comment as we already have a SNAPSHOT dependency, feel free to go ahead. But I am planning to suggest 2.0.1 Ratis release.

Can you please share what is the Ratis commit which was required here? (Just to be sure it's included if we do a patch release..)

[bharatviswa504]

@elek for SCM HA snapshot implementation to work we require RATIS-1326, RATIS-1356, and RATIS-1369.
The commit related to this is 00f0c85

[adoroszlai]

@GlenGeng I think pushing empty commit is better for triggering new CI run than closing/reopening. Most importantly it keeps the PR's build history easily available. We also gain more data points (regarding CI failures), because it triggers two builds: PR and push. It is also slightly less noisy.

[GlenGeng-awx]

Thanks @elek for looking at this PR, and overall I agree with your suggestion. Thanks @bharatviswa504 for the explanation and the sharing of the commit and the related context.

I think pushing empty commit is better for triggering new CI run than closing/reopening

Thanks @adoroszlai for the suggestion. I will follow it.

[bharatviswa504]

@GlenGeng Can you check if test failure is related I see one of the test related to InstallSnapshot is failing.

[GlenGeng-awx]

@bharatviswa504 After installSnapshot, the lifecycle of follower's SCMStateMachine is in PAUSED state.

          stateMachine.followerEvent().notifyInstallSnapshotFromLeader(roleInfoProto, firstAvailableLogTermIndex)
              .whenComplete((reply, exception) -> {
                if (exception != null) {
                  LOG.warn("{}: Failed to notify StateMachine to InstallSnapshot. Exception: {}",
                      getMemberId(), exception.getMessage());
                  inProgressInstallSnapshotRequest.compareAndSet(firstAvailableLogTermIndex, null);
                  return;
                }

                if (reply != null) {
                  LOG.info("{}: StateMachine successfully installed snapshot index {}. Reloading the StateMachine.",
                      getMemberId(), reply.getIndex());
                  stateMachine.pause();
                  state.updateInstalledSnapshotIndex(reply);
                  state.reloadStateMachine(reply.getIndex());
                }
                inProgressInstallSnapshotRequest.compareAndSet(firstAvailableLogTermIndex, null);
              });
        } 

The reason is, after notifyInstallSnapshotFromLeader is called, ratis call stateMachine.pause(); again.
I am checking it.

[bharatviswa504]

cc @hanishakoneru does OmStateMachine also need similar code?

[bharatviswa504]

Question: How these tests are working before? (Not able to understand that? can you shed some info)

[GlenGeng-awx]

After notifyInstallSnapshotFromLeader is called, ratis calls

                if (reply != null) {
                  LOG.info("{}: StateMachine successfully installed snapshot index {}. Reloading the StateMachine.",
                      getMemberId(), reply.getIndex());
                  stateMachine.pause();
                  state.updateInstalledSnapshotIndex(reply);
                  state.reloadStateMachine(reply.getIndex());
                }

stateMachine.pause(); will make SM to be in PAUSED state, state.reloadStateMachine(reply.getIndex()) will trigger StateMachineUpdater#reload() to be called, which will then call stateMachine.reinitialize();.

This is the reason of the fix.

As far as I known, at the end of TestSCMInstallSnapshotWithHA#testInstallSnapshot, the followerSCM is also in PAUSED state, which is not checked before.

And for testInstallOldCheckpointFailure, notifyInstallSnapshotFromLeader is not really called, since without the fix in RATIS-1369, downloading snapshot taken at index 0 is ignore by follower SCM.

[bshashikant]

I think, in Ratis we need to fix the applyTransaction behaviour as well. If the stateMachine is in paused state, the applyIndex should not increase in the StateMachineUpdater. This is not happening right now.

Then, as part of reloading the stateMachine as part of reinitializing the stateMachine, it can call unpause().

[bshashikant]

Have filed https://issues.apache.org/jira/browse/RATIS-1370 for ratis issue.

[bshashikant]

Thanks @GlenGeng for the work and others for the reviews.