HDDS-10612. Add Robot test to verify Container Balancer for RATIS containers

[afilpp]

What changes were proposed in this pull request?

HDDS-10612. Add Robot test to verify Container Balancer for RATIS containers

Currently there are only unit tests for Container Balancer and no acceptance tests at all. At a minimum, we should add a Robot test to verify Container Balancer for RATIS containers. And probably in the future we should add robot test for EC case.

Test case:

1. Move 1 datanode to maintenance mode (we use 4 datanodes in this test)
2. Create multiple keys (after loading the data, we check that 3 datanodes are ~60% busy, and the one that is in maintenance mode is empty)
3. Start datanode recommission (wait until datanode recommissioning is completed)
4. Start container balancer (wait until container balancer is completed)
5. Check results (after balancing on all 4 datanodes, we should see approximately the same data distribution.)

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10612

How was this patch tested?

Added Robot test

[adoroszlai]

Thanks @afilpp for the patch. Overall looks good, added some minor comments.

It would be nice to create the environment as an add-on for ozone-ha instead of a completely separate one, but we can check if it's feasible in a follow-up task.

[adoroszlai]

Thanks @afilpp for updating the patch, LGTM.

[myskov]

LGTM, @siddhantsangwan please take a look

[siddhantsangwan]

The robot test logic LGTM.

[ivandika3]

Seems there is an intermittent failure on the acceptance test

https://github.com/apache/ozone/actions/runs/8546730074/job/23418032793

@afilpp Could you take a look?

Edit: Can refer to the comment in HDDS-10612 for possible root cause.