HDDS-7606. Add SCM HA support in intellij run

[ChenSammi]

https://issues.apache.org/jira/browse/HDDS-7606

[kerneltime]

@Galsza can you please take a look?

[captainzmc]

Thanks for @ChenSammi's patch. I have locally verified that after the HA cluster is started, the volume creation, bucket creation and put key are all success.
The status of Recon may not be correct at present. After I started recon, I found that it has been trying to connect port 9860 (this is the port of SCM clinet of non-HA).
I think we'd better add a Recon-ha that starts with ozone-site-ha.xml.

[captainzmc]

If possible, we'd better add an Ozoneshell-ha as well. Currently, OzoneShell is also non-HA by default. Users need to manually change its ozone-site.xml to ozone-site-ha.xml if they want to operate the HA cluster

[neils-dev]

Thanks @ChenSammi for adding SCM HA support to the intellij IDE dev environment. I've run the HA configuration in the following manner through the run configs:
i. PrimordialSCMInit-ha
ii. PrimordialSCM-ha
iii. OzoneManagerInit-ha
iv. OzoneManager-ha
for the scm 2 & 3
v. Scm(2or3)Bootstrap-ha
vi.) Scm(2or3)-ha

Then the datanodesX-ha run configs. The ozone HA cluster came up with the scm first in safe mode then exiting safe mode without issue. With the datanodes I noticed some exceptions (timeout) in the log as follows:

java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at org.apache.hadoop.ozone.container.common.states.datanode.RunningDatanodeState.computeNextContainerState(RunningDatanodeState.java:199)

In addition, on starting the DNs in the cluster I found the following warnings generated from the SCMs:
2022-12-16 16:31:20,543 WARN events.EventQueue (EventQueue.java:fireEvent(216)) - No event handler registered for event TypedEvent{payloadType=DatanodeDetails, name='Datanode_Command_Queue

The cluster in the dev environment successfully ran shell file system operations with both the OzoneSell and OzoneFsShell. Also ran freon load tests, okbg, without issue.

For the ScmRoles run from the run config , receiving an error message

Unmatched argument at index 1: 'scm'
Usage: ozone admin [-hV] [--verbose] [-conf=<configurationPath>] ...

Do I need to add something to the runConfiguration for it to work? Running it as is and getting the error.

[captainzmc]

i. PrimordialSCMInit-ha
ii. PrimordialSCM-ha
iii. OzoneManagerInit-ha
iv. OzoneManager-ha
for the scm 2 & 3
v. Scm(2or3)Bootstrap-ha
vi.) Scm(2or3)-ha

Hi @neils-dev, The correct boot sequence is:

1. PrimordialSCMInit-ha
2. PrimordialSCM-ha
3. Scm2 Bootstrap-ha
4. Scm2-ha
5. Scm3 Bootstrap-ha
6. Scm3-ha
7. OzoneManagerInit-ha
8. OzoneManager-ha
9. Recon (Should change conf from ozone-site.xml to ozone-site-ha.xml)
10. Datanode(1~3)-ha

[captainzmc]

+1 The change looks good.

[captainzmc]

Let's merge this, Thanks @ChenSammi for the patch and thanks @neils-dev for the check.

[ChenSammi]

Thanks @captainzmc and @neils-dev for the code review.