HDDS-4754. Make scm heartbeat rpc retry interval configurable

[Xushaohong]

What changes were proposed in this pull request?

Background:
The current retry policy of DN is to retry sending with a 1s interval. Given at some time-point, all the DNs lost connection with the SCM at the same time, due to the restart of SCM, all DNs will send container report to SCM nearly at the same time, which is a ContainerReport Storm.

Solution:
Manually adjust the rpc-retry-interval with rpc-retry-count could mitigate extreme cases such as OOM, when facing up a huge cluster.
Make the rpc-retry-interval configurable.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-4754

How was this patch tested?

CI

[Xushaohong]

Please take a look @GlenGeng

[linyiqun]

Can we just reuse default DN heartbeat interval(HddsConfigKeys#HDDS_HEARTBEAT_INTERVAL_DEFAULT, 30s) rather than defined a new rpc retry interval here? Would this a better way?

[Xushaohong]

Can we just reuse default DN heartbeat interval(HddsConfigKeys#HDDS_HEARTBEAT_INTERVAL_DEFAULT, 30s) rather than defined a new rpc retry interval here? Would this a better way?

The retry interval is only 1 sec now, which is for quickly connecting the scm. The default HB interval may be too long.
Actually, the retry count is not working, since the DatanodeStateMachine keeps retrying after 15 retries finish.
The current retry policy seems still needs to be changed.

[linyiqun]

Okay, get it.
There is another place that also can be updated to use getScmRpcRetryInterval(conf) in this class. Can you update this (SCMConnectionManager.java#L200)?

  /**
   * Adds a new Recon server to the set of endpoints.
   * @param address Recon address.
   * @throws IOException
   */
  public void addReconServer(InetSocketAddress address) throws IOException {
    LOG.info("Adding Recon Server : {}", address.toString());
    writeLock();
    try {
      if (scmMachines.containsKey(address)) {
        LOG.warn("Trying to add an existing SCM Machine to Machines group. " +
            "Ignoring the request.");
        return;
      }
      Configuration hadoopConfig =
          LegacyHadoopConfigurationSource.asHadoopConfiguration(this.conf);
      RPC.setProtocolEngine(hadoopConfig, ReconDatanodeProtocolPB.class,
          ProtobufRpcEngine.class);
      long version =
          RPC.getProtocolVersion(ReconDatanodeProtocolPB.class);

      RetryPolicy retryPolicy =
          RetryPolicies.retryUpToMaximumCountWithFixedSleep(
              getScmRpcRetryCount(conf),
              1000, TimeUnit.MILLISECONDS);  <======
...
}

[linyiqun]

Thanks for addressing the comment, @Xushaohong .
LGTM, +1. Let's wait @GlenGeng to have a final review for this PR, : ).

[GlenGeng-awx]

+1

Thanks @Xushaohong for the work. Thanks @linyiqun for the review.

This is a preliminary work for the SCM OOM issue. Future proposal will be throttling the on-going reports at both SCM side and DN side, e.g., 1) SCM drops the reports if it has queued too many reports, 2) DN reduces the number of reports by recording a lease for its Container(recommended by @xiaoyuyao ).

[Xushaohong]

@runzhiwang pls take a look and help merge :)

[runzhiwang]

LGTM. @Xushaohong Thanks the patch. @linyiqun @GlenGeng Thanks for review.