Skip to content

Fix how we wait on routers - cluster ingress operator uses daemonsets #2001

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ramr wants to merge 1 commit into from
Closed

Fix how we wait on routers - cluster ingress operator uses daemonsets #2001

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 17 additions & 3 deletions ci-operator/templates/openshift/installer/cluster-launch-installer-e2e.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,12 +151,26 @@ objects:
break
done

function isRouterReady() {
local deployment
deployment=${1:-"router/deploy"}
local ns
ns=${2:-"openshift-ingress"}
if [[ "${deployment}" == "ds/router-default" ]]; then
ready=$(oc get "${deployment}" -n "${ns}" -o go-template='{{ne "0" (print .status.numberReady)}}')
[[ "${ready}" == "true" ]]
return
fi

oc wait "${deployment}" -n "${ns}" --for condition=available --timeout=10m
Copy link
Contributor

@ironcladlou ironcladlou Oct 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this --timeout cause each iteration to wait 10 minutes in addition to the 60 seconds in the loop?!

Copy link
Contributor Author

@ramr ramr Oct 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that was the original wait code, it was (oc wait 10 mins + 1 min sleep) * 10 retries.

Copy link
Contributor

@ironcladlou ironcladlou Oct 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder why? Seems like a good way to waste like 10 minutes during setup... it takes a matter of seconds for the router to deploy

Copy link
Contributor

@ironcladlou ironcladlou Oct 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or does this have to account for a large portion of the general cluster setup?

Copy link
Contributor Author

@ramr ramr Oct 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well it would depend on how fast the deployment occurs ... this will return back (fail) if there is no deployment object but once the deployment object exists, it would still need to wait on it becoming available. So I guess rather than loop and retry the oc wait operation, just waiting on it inline makes sense.
And it wouldn't necessarily be 10 minutes (that's the worst case scenario) - if the object is available it would return back prior to the timeout.

Copy link
Contributor

@smarterclayton smarterclayton Oct 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We were looping waiting because of a bug in wait.

Copy link
Member

@wking wking Oct 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is a sketch of something that may be more robust for looping on blocking calls.

}

i=0
MAX_RETRIES=10
until oc wait "${ROUTER_DEPLOYMENT}" -n "${ROUTER_NAMESPACE}" --for condition=available --timeout=10m || [ $i -eq $MAX_RETRIES ]; do
until isRouterReady "${ROUTER_DEPLOYMENT}" "${ROUTER_NAMESPACE}" || [ $i -eq $MAX_RETRIES ]; do
i=$((i+1))
[ $i -eq $MAX_RETRIES ] && echo "timeout waiting for router to be available" && exit 1
echo "error deploy/router did not come up"
[ $i -eq $MAX_RETRIES ] && echo "timeout waiting for ${ROUTER_NAMESPACE}/${ROUTER_DEPLOYMENT} to be available" && exit 1
echo "error ${ROUTER_NAMESPACE}/${ROUTER_DEPLOYMENT} did not come up"
sleep 60
done

Expand Down