DR snapshot restore: assemble etcd connection string on first master #3919

vrutkovs · 2019-05-30T18:00:20Z

bastion_ssh output may be mangled, so it should not be used to set var
values. Instead first master should collect necessary info without
passing it back to test container

hexfusion · 2019-06-01T00:56:30Z

/retest

vrutkovs · 2019-06-01T13:33:40Z

/test pj-rehearse

ssh-bastion remove failed

vrutkovs · 2019-06-02T13:38:32Z

/test pj-rehearse

vrutkovs · 2019-06-02T22:06:23Z

Flakes

/test pj-rehearse

vrutkovs · 2019-06-03T07:59:40Z

/test pj-rehearse

vrutkovs · 2019-06-03T10:52:16Z

Additional masters didn't come up

/test pj-rehearse

vrutkovs · 2019-06-03T12:07:46Z

/test pj-rehearse

vrutkovs · 2019-06-03T20:11:21Z

Failing tests:

[Feature:Platform] Managed cluster should have no crashlooping pods in core namespaces over two minutes [Suite:openshift/conformance/parallel]
[Feature:Prometheus][Conformance] Prometheus when installed on the cluster should report less than two alerts in firing or pending state [Suite:openshift/conformance/parallel/minimal]
[sig-instrumentation] MetricsGrabber should grab all metrics from a Kubelet. [Suite:openshift/conformance/parallel] [Suite:k8s]

/test pj-rehearse

openshift-ci-robot · 2019-06-03T22:03:12Z

@vrutkovs: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/rehearse/openshift/machine-config-operator/master/e2e-restore-cluster-state	a014af5b4ef0be22cf4c4417d18cff76d0fb7ffb	link	`/test pj-rehearse`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

vrutkovs · 2019-06-04T07:08:32Z

/cc @hexfusion @runcom

PTAL, this fixes etcd restore tests - see https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_release/3919/rehearse-3919-pull-ci-openshift-machine-config-operator-master-e2e-restore-cluster-state/30

[Feature:Platform] Managed cluster should have no crashlooping pods in core namespaces over two minutes [Suite:openshift/conformance/parallel] is still failing - I'll address that in the followup PR

`bastion_ssh` output may be mangled, so it should not be used to set var values. Instead first master should collect necessary info without passing it back to `test` container

hexfusion · 2019-06-05T14:51:16Z

ci-operator/templates/openshift/installer/cluster-launch-installer-e2e.yaml

+
+          echo "Assemble etcd connection string"
+          bastion_ssh "core@${FIRST_MASTER}" 'rm -rf /tmp/etcd/connstring && mapfile -t MASTERS < <(ls /tmp/etcd) && echo ${MASTERS[@]} && for master in "${MASTERS[@]}"; do echo -n "$(cat /tmp/etcd/${master}/etcd_name)=$(cat /tmp/etcd/${master}/etcd_uri)," >> /tmp/etcd/connstring; done && sed -i '"'$ s/.$//'"' /tmp/etcd/connstring'



Since etcd is running we can use it to generate this list. We don't need to change it now but just a talking point. Not perfect but this is one idea.

etcdctl member list -w json | jq -r '.members[] | [.name,.peerURLs[0]] | "(.[0])=(.[1])" ' | xargs | sed -e 's/ /,/g'

Yeah, so I'll be rewriting this into a proper golang e2e test - hopefully we'd have better instruments than cat and would be able to rework this

hexfusion · 2019-06-05T14:52:51Z

/lgtm

abhinavdahiya

/approve

openshift-ci-robot · 2019-06-05T15:37:41Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavdahiya, hexfusion, vrutkovs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~ci-operator/templates/openshift/installer/OWNERS~~ [abhinavdahiya]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2019-06-05T15:41:13Z

@vrutkovs: Updated the following 2 configmaps:

prow-job-cluster-launch-installer-e2e configmap in namespace ci using the following files:
- key cluster-launch-installer-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-e2e.yaml
prow-job-cluster-launch-installer-e2e configmap in namespace ci-stg using the following files:
- key cluster-launch-installer-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-e2e.yaml

Details

In response to this:

bastion_ssh output may be mangled, so it should not be used to set var
values. Instead first master should collect necessary info without
passing it back to test container

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label May 30, 2019

openshift-ci-robot requested review from LorbusChris and aaronlevy May 30, 2019 18:01

vrutkovs force-pushed the dr-etcd-restore-assemble-fix branch 10 times, most recently from 7b50694 to fcb7649 Compare May 31, 2019 13:37

vrutkovs mentioned this pull request May 31, 2019

MCO: always run restore-cluster-state DR test #3917

Closed

1 task

vrutkovs force-pushed the dr-etcd-restore-assemble-fix branch 3 times, most recently from ce1db6b to 7c94cc3 Compare May 31, 2019 19:00

openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 31, 2019

vrutkovs force-pushed the dr-etcd-restore-assemble-fix branch from 7c94cc3 to a6d8c5c Compare June 1, 2019 08:24

openshift-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 1, 2019

openshift-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jun 2, 2019

vrutkovs force-pushed the dr-etcd-restore-assemble-fix branch from c02234d to 701d530 Compare June 2, 2019 12:15

openshift-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 2, 2019

vrutkovs force-pushed the dr-etcd-restore-assemble-fix branch 5 times, most recently from 252acaf to a014af5 Compare June 3, 2019 15:47

openshift-ci-robot requested review from hexfusion and runcom June 4, 2019 07:08

DR snapshot restore: assemble etcd connection string on first master

4d491ba

`bastion_ssh` output may be mangled, so it should not be used to set var values. Instead first master should collect necessary info without passing it back to `test` container

vrutkovs force-pushed the dr-etcd-restore-assemble-fix branch from a014af5 to 4d491ba Compare June 4, 2019 07:09

vrutkovs mentioned this pull request Jun 4, 2019

DR: use param to populate etcd name openshift/machine-config-operator#804

Merged

hexfusion reviewed Jun 5, 2019

View reviewed changes

openshift-ci-robot assigned hexfusion Jun 5, 2019

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jun 5, 2019

abhinavdahiya approved these changes Jun 5, 2019

View reviewed changes

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 5, 2019

openshift-merge-robot merged commit ccd42f5 into openshift:master Jun 5, 2019


		echo "Assemble etcd connection string"
		bastion_ssh "core@${FIRST_MASTER}" 'rm -rf /tmp/etcd/connstring && mapfile -t MASTERS < <(ls /tmp/etcd) && echo ${MASTERS[@]} && for master in "${MASTERS[@]}"; do echo -n "$(cat /tmp/etcd/${master}/etcd_name)=$(cat /tmp/etcd/${master}/etcd_uri)," >> /tmp/etcd/connstring; done && sed -i '"'$ s/.$//'"' /tmp/etcd/connstring'

DR snapshot restore: assemble etcd connection string on first master #3919

DR snapshot restore: assemble etcd connection string on first master #3919

Uh oh!

Conversation

vrutkovs commented May 30, 2019

Uh oh!

hexfusion commented Jun 1, 2019

Uh oh!

vrutkovs commented Jun 1, 2019

Uh oh!

vrutkovs commented Jun 2, 2019

Uh oh!

vrutkovs commented Jun 2, 2019

Uh oh!

vrutkovs commented Jun 3, 2019

Uh oh!

vrutkovs commented Jun 3, 2019

Uh oh!

vrutkovs commented Jun 3, 2019

Uh oh!

vrutkovs commented Jun 3, 2019

Uh oh!

openshift-ci-robot commented Jun 3, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vrutkovs commented Jun 4, 2019

Uh oh!

hexfusion Jun 5, 2019

Choose a reason for hiding this comment

Uh oh!

vrutkovs Jun 5, 2019

Choose a reason for hiding this comment

Uh oh!

hexfusion commented Jun 5, 2019

Uh oh!

abhinavdahiya left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci-robot commented Jun 5, 2019

Uh oh!

openshift-ci-robot commented Jun 5, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

openshift-ci-robot commented Jun 3, 2019 •

edited

Loading