Skip to content

HDDS-10644. Intermittent failure in testBalancer.robot #6481

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
myskov merged 1 commit into from
Apr 5, 2024
Merged

HDDS-10644. Intermittent failure in testBalancer.robot #6481

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions hadoop-ozone/dist/src/main/compose/ozone-balancer/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,9 +51,6 @@ OZONE-SITE.XML_ozone.scm.primordial.node.id=scm1
OZONE-SITE.XML_hdds.container.report.interval=30s
OZONE-SITE.XML_ozone.om.s3.grpc.server_enabled=true
OZONE-SITE.XML_ozone.recon.db.dir=/data/metadata/recon
OZONE-SITE.XML_ozone.recon.address=recon:9891
OZONE-SITE.XML_ozone.recon.http-address=0.0.0.0:9888
OZONE-SITE.XML_ozone.recon.https-address=0.0.0.0:9889
OZONE-SITE.XML_dfs.container.ratis.datastream.enabled=true

OZONE_CONF_DIR=/etc/hadoop
Expand Down
7 changes: 4 additions & 3 deletions hadoop-ozone/dist/src/main/smoketest/balancer/testBalancer.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,15 +95,16 @@ Get Uuid
Close All Containers
FOR ${INDEX} IN RANGE 15
${container} = Execute ozone admin container list --state OPEN | jq -r 'select(.replicationConfig.replicationFactor == "THREE") | .containerID' | head -1
EXIT FOR LOOP IF "${container}" == ""
Execute ozone admin container close "${container}"
EXIT FOR LOOP IF "${container}" == "${EMPTY}"
${message} = Execute And Ignore Error ozone admin container close "${container}"
Run Keyword If '${message}' != '${EMPTY}' Should Contain ${message} is in closing state
${output} = Execute ozone admin container info "${container}"
Should contain ${output} CLOS
Copy link
Contributor

@ivandika3 ivandika3 Apr 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason of CLOS is to include both CLOSING and CLOSED?

Copy link
Contributor Author

@afilpp afilpp Apr 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes for this test we don't need to wait for the containers to be completely closed (this may take too long). Therefore we are happy with both “closed” and “closing” statuses.

Copy link
Contributor

@ivandika3 ivandika3 Apr 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the explanation.

END
Wait until keyword succeeds 3min 10sec All container is closed

All container is closed
${output} = Execute ozone admin container list --state OPEN
${output} = Execute ozone admin container list --state OPEN
Should Be Empty ${output}
Comment on lines -106 to 108
Copy link
Contributor

@ivandika3 ivandika3 Apr 5, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From ContainerBalancerSelectionCriteria#shouldBeExcluded, the container chosen needs to be CLOSED (ContainerBalancerCriteria#isContainerClosed). The "All container is closed" check whether there is no OPEN containers. Therefore, I think there is a chance where the container is still CLOSING when the container balancer start, and the container will be excluded during the container balancer iteration.

However, I think the chance is very small, since the time pass between the close container command and the start of the container balancer should be large enough for the container to be closed. So it should be fine as it is now.


Get Datanode Ozone Used Bytes Info
Expand Down