Skip to content

DR snapshot restore: assemble etcd connection string on first master #3919

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
openshift-merge-robot merged 1 commit into from
Jun 5, 2019
Merged

DR snapshot restore: assemble etcd connection string on first master #3919

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 12 additions & 13 deletions ci-operator/templates/openshift/installer/cluster-launch-installer-e2e.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,12 +165,12 @@ objects:
local rc=0
shift
for i in $(seq 0 $((ATTEMPTS-1))); do
echo "${@}"
echo "--> ${@}"
set +e
"${@}"
rc="$?"
set -e
echo "exit code: $rc"
echo "--> exit code: $rc"
test "${rc}" = 0 && break
sleep 10
done
Expand Down Expand Up @@ -255,20 +255,23 @@ objects:
done

echo "Collect etcd names"
ETCD_CONN_STRING_TEMP=""
for master in "${MASTERS[@]}"
do
ETCD_NAME="etcd-member-$(bastion_ssh core@${master} hostname -f)"
ETCD_URI=$(bastion_ssh "core@${master}" 'source /run/etcd/environment && echo "https://${ETCD_DNS_NAME}:2380"')
ETCD_CONN_STRING_TEMP="${ETCD_CONN_STRING_TEMP}${ETCD_NAME}=${ETCD_URI},"
bastion_ssh "core@${master}" 'echo "etcd-member-$(hostname -f)" > /tmp/etcd_name && source /run/etcd/environment && echo "https://${ETCD_DNS_NAME}:2380" > /tmp/etcd_uri'
bastion_ssh "core@${FIRST_MASTER}" "mkdir -p /tmp/etcd/${master} && scp -o StrictHostKeyChecking=no core@${master}:/tmp/etcd_name /tmp/etcd/${master}/etcd_name && scp -o StrictHostKeyChecking=no core@${master}:/tmp/etcd_uri /tmp/etcd/${master}/etcd_uri"
bastion_ssh "core@${FIRST_MASTER}" "cat /tmp/etcd/${master}/etcd_name"
bastion_ssh "core@${FIRST_MASTER}" "cat /tmp/etcd/${master}/etcd_uri"
done
ETCD_CONN_STRING=${ETCD_CONN_STRING_TEMP::-1}

echo "Assemble etcd connection string"
bastion_ssh "core@${FIRST_MASTER}" 'rm -rf /tmp/etcd/connstring && mapfile -t MASTERS < <(ls /tmp/etcd) && echo ${MASTERS[@]} && for master in "${MASTERS[@]}"; do echo -n "$(cat /tmp/etcd/${master}/etcd_name)=$(cat /tmp/etcd/${master}/etcd_uri)," >> /tmp/etcd/connstring; done && sed -i '"'$ s/.$//'"' /tmp/etcd/connstring'

Copy link
Contributor

@hexfusion hexfusion Jun 5, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since etcd is running we can use it to generate this list. We don't need to change it now but just a talking point. Not perfect but this is one idea.

etcdctl member list -w json | jq -r '.members[] | [.name,.peerURLs[0]] | "(.[0])=(.[1])" ' | xargs | sed -e 's/ /,/g'

Copy link
Contributor Author

@vrutkovs vrutkovs Jun 5, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, so I'll be rewriting this into a proper golang e2e test - hopefully we'd have better instruments than cat and would be able to rework this

echo "Restore etcd cluster from snapshot"
for master in "${MASTERS[@]}"
do
echo "Running /usr/local/bin/etcd-snapshot-restore.sh on ${master}"
bastion_ssh "core@${master}" "sudo -i /bin/bash -x /usr/local/bin/etcd-snapshot-restore.sh /tmp/snapshot.db ${ETCD_CONN_STRING}"
bastion_ssh "core@${FIRST_MASTER}" "scp -o StrictHostKeyChecking=no /tmp/etcd/connstring core@${master}:/tmp/etcd_connstring"
bastion_ssh "core@${master}" 'sudo -i /bin/bash -x /usr/local/bin/etcd-snapshot-restore.sh /tmp/snapshot.db $(cat /tmp/etcd_connstring)'
done

echo "Wait for API server to come up"
Expand Down Expand Up @@ -300,11 +303,7 @@ objects:
rc=0
for master in "${MASTERS[@]}"
do
ROLLBACK="$(bastion_ssh core@${master} "sudo -i cat /etc/rollback-test")"
if [[ "${ROLLBACK}" != "A" ]]; then
echo "Master ${master} rollback contents was ${ROLLBACK}"
rc=1
fi
bastion_ssh core@${master} 'sudo -i test "$(cat /etc/rollback-test)" == "A"'
done

if [[ "${rc}" == "1" ]]; then exit 1; fi
Expand Down