HDDS-4580. Datanode can be stuck in leader not ready state after restart

[lokeshj1703]

What changes were proposed in this pull request?

On restart the transactions are reapplied for an existing ratis pipeline. ContainerStateMachine#applyTransaction while processing future can throw NullPointerException leading to the future being completed exceptionally.

      future.thenApply(r -> {
        if (trx.getServerRole() == RaftPeerRole.LEADER) {
          long startTime = (long) trx.getStateMachineContext();
          metrics.incPipelineLatency(cmdType,
              Time.monotonicNowNanos() - startTime);
        }

In the above code snippet trx.getStateMachineContext() will be null during restart and this fails the future itself without updating the applyTransactionCompletionMap. As a result the lastAppliedIndex is not updated for the server and server is stuck in leader not ready state.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-4580

How was this patch tested?

Existing UT

[xiaoyuyao]

LGTM, +1.