Skip to content

HDDS-8985. Intermittent timeout exiting safe mode in HA secure tests #5033

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ChenSammi merged 1 commit into from
Jul 9, 2023
Merged

HDDS-8985. Intermittent timeout exiting safe mode in HA secure tests #5033

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 7 additions & 31 deletions hadoop-ozone/dist/src/main/compose/testlib.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,38 +72,14 @@ find_tests(){

## @description wait until safemode exit (or 240 seconds)
wait_for_safemode_exit(){
# version-dependent
: ${OZONE_SAFEMODE_STATUS_COMMAND:=ozone admin safemode status --verbose}

#Reset the timer
SECONDS=0

#Don't give it up until 240 seconds
while [[ $SECONDS -lt 240 ]]; do

#This line checks the safemode status in scm
local command="${OZONE_SAFEMODE_STATUS_COMMAND}"
if [[ "${SECURITY_ENABLED}" == 'true' ]]; then
status=$(docker-compose exec -T ${SCM} bash -c "kinit -k HTTP/[email protected] -t /etc/security/keytabs/HTTP.keytab && $command" || true)
else
status=$(docker-compose exec -T ${SCM} bash -c "$command")
fi

echo "SECONDS: $SECONDS"

echo $status
if [[ "$status" ]]; then
if [[ ${status} == "SCM is out of safe mode." ]]; then
#Safemode exits. Let's return from the function.
echo "Safe mode is off"
return
fi
fi
local cmd="ozone admin safemode wait -t 240"
if [[ "${SECURITY_ENABLED}" == 'true' ]]; then
wait_for_port kdc 88 60
cmd="kinit -k HTTP/[email protected] -t /etc/security/keytabs/HTTP.keytab && $cmd"
fi

sleep 2
done
echo "WARNING! Safemode is still on. Please check the docker-compose files"
return 1
wait_for_port ${SCM} 9860 120
execute_commands_in_container ${SCM} "$cmd"
}

## @description wait until OM leader is elected (or 120 seconds)
Expand Down