HDDS-2278. Run S3 test suite on OM HA cluster.

[bharatviswa504]

Add a docker compose OM HA cluster with S3.
Run the S3 test suite on the OM HA cluster.
https://issues.apache.org/jira/browse/HDDS-2278

[elek]

+1 thanks the patch @bharatviswa504

Note: While I am happy to have more and more environments, I feel that it's harder and harder to manage them and some of them are confusing. I am thinking about to merge some of the environments, for example include s3 in the simple ozone (and ozone-om-ha) and delete ozone-s3.

S3 doesn't have huge resource requirement and very core of our provided functionality.

This is not related to this patch (as this patch follows the current practice) but I am very interested about your opinion.

[elek]

UPDATE: it's failing during my local test all the time. the ozones3 is working but not the new tests. Let's try to request a new test run...

[elek]

/retest

[bharatviswa504]

Thank You @elek for the review.
I see newly acceptance tests are passing. Let me know if you still see errors in your local run.

[bharatviswa504]

+1 thanks the patch @bharatviswa504

Note: While I am happy to have more and more environments, I feel that it's harder and harder to manage them and some of them are confusing. I am thinking about to merge some of the environments, for example include s3 in the simple ozone (and ozone-om-ha) and delete ozone-s3.

S3 doesn't have huge resource requirement and very core of our provided functionality.

This is not related to this patch (as this patch follows the current practice) but I am very interested about your opinion.

Yes, I am with you on this, S3 now became core functionality, we don't need a separate environment for it, we can include S3 in Ozone compose. And also this helps in reducing run time of our acceptance test suite.

[bharatviswa504]

@elek can we commit this or still some issues you are seeing?

[elek]

I see a problem locally, which is passed on the server. Don't know what is the exact problem...

[adoroszlai]

@bharatviswa504 @elek I think the problem is with the upgraded Ratis (#26 (review)). Leader election problem affects OM HA.

1. The pull request source branch is based on 920dde9, which precedes Ratis upgrade 546e36b. Test passes locally when checking out this branch. This is what CI is testing.
2. Applying the patch on current master (2529cee), the test fails locally. Log is full of leader election attempts:

om1_1       | 2019-10-18 15:55:51,883 INFO impl.RaftServerImpl: om1@group-562213E44849: changes role from CANDIDATE to FOLLOWER at term 3601 for DISCOVERED_A_NEW_TERM
...
om1_1       | 2019-10-18 15:55:52,026 INFO impl.RaftServerImpl: om1@group-562213E44849: changes role from  FOLLOWER to CANDIDATE at term 3601 for changeToCandidate
...
om1_1       | 2019-10-18 15:55:52,038 INFO impl.RaftServerImpl: om1@group-562213E44849: changes role from CANDIDATE to FOLLOWER at term 3602 for DISCOVERED_A_NEW_TERM

[bharatviswa504]

Thank You @adoroszlai for the analysis.
I will wait until we upgrade to the latest Ratis version to commit this.

[bharatviswa504]

Rebased with the latest master(after ratis version upgrade)

[bharatviswa504]

/retest

[bharatviswa504]

Newly added tests are passing, test failures are not related to this patch.

[dineshchitlangia]

+1 LGTM, failures are unrelated

[dineshchitlangia]

Thanks @elek and @adoroszlai for reviews. Thanks @bharatviswa504 for this contribution. Committed this change.