Bug 1788707: Backport Azure private DNS to 4.2 branch #2825

jhixson74 · 2019-12-16T20:04:33Z

This brings Azure private DNS into the 4.2 branch

https://issues.redhat.com/browse/CORS-1316

…ndle private_dns zone Using the upstream azurerm provider is not possible for now because of following reasons: 1) There is not srv record resource for private dns zone 2) The version of provider that has the private dns zone resources `1.34.0` has a lot of bugs like * hashicorp/terraform-provider-azurerm#4452 * hashicorp/terraform-provider-azurerm#4453 * hashicorp/terraform-provider-azurerm#4501 Some of these bugs are fixed, and some are in flight. Another reason moving to `1.36.0` which might have all the fixes we need is the provider has moved to using `standalone terraform plugin SDK v1.1.1` [1]. Because we vendor both terraform and providers, this causes errors like `panic: gob: registering duplicate types for "github.com/zclconf/go-cty/cty.primitiveType": cty.primitiveType != cty.primitiveType` Therefore, we would have to move towards a single vendor for terraform and plugins for correct inter-operation, which is tricker due to conflicts elsewhere A simple 4 resource plugin that re-uses the already vendored azurerm provider as library and carries over the required resources seems like an easy fix for now. [1]: hashicorp/terraform-provider-azurerm#4474 (cherry picked from commit af00810)

Using the Private DNS Zone also allows us to remove our previous hack [1] [1]: openshift@8ac9ab4 (cherry picked from commit c1e24af) Conflicts: data/data/azure/master/variables.tf

the ingress-operator can handle the Private DNS Zone since [1] [1]: openshift/cluster-ingress-operator#300 (cherry picked from commit 7d3119f)

(cherry picked from commit e80fde1)

… deleting public records Updates the destroy to look for both DNS Zones type Private and Private DNS Zones to find the private records and the corresponding DNS Zone type Public. Since the zone name for both types of private dns zone is the same, we can try both to calculate the private records and then re-use the same codepath to delete the public records. (cherry picked from commit 65111a0)

openshift-ci-robot · 2019-12-16T20:04:35Z

@jhixson74: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

Details

In response to this:

WIP: backport Azure private DNS to 4.2 branch

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2019-12-16T20:09:57Z

@jhixson74: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

Details

In response to this:

Backport Azure private DNS to 4.2 branch

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jhixson74 · 2020-01-06T18:43:28Z

/test e2e-azure

openshift-ci-robot · 2020-01-07T21:35:37Z

@jhixson74: This pull request references Bugzilla bug 1788707, which is invalid:

expected the bug to target the "4.2.z" release, but it targets "---" instead
expected Bugzilla bug 1788707 to depend on a bug in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but no dependents were found

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Bug 1788707: Backport Azure private DNS to 4.2 branch

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2020-01-07T21:36:05Z

@jhixson74: This pull request references Bugzilla bug 1788707, which is invalid:

expected the bug to target the "4.2.z" release, but it targets "---" instead
expected Bugzilla bug 1788707 to depend on a bug in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but no dependents were found

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jhixson74 · 2020-01-07T21:37:52Z

/bugzilla refresh

openshift-ci-robot · 2020-01-07T21:37:54Z

@jhixson74: This pull request references Bugzilla bug 1788707, which is invalid:

expected the bug to target the "4.2.z" release, but it targets "---" instead
expected Bugzilla bug 1788707 to depend on a bug in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but no dependents were found

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jhixson74 · 2020-01-07T21:39:52Z

/bugzilla refresh

openshift-ci-robot · 2020-01-07T21:39:54Z

@jhixson74: This pull request references Bugzilla bug 1788707, which is invalid:

expected Bugzilla bug 1788707 to depend on a bug in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but no dependents were found

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jhixson74 · 2020-01-07T21:53:59Z

/bugzilla refresh

openshift-ci-robot · 2020-01-07T21:54:02Z

@jhixson74: This pull request references Bugzilla bug 1788707, which is invalid:

expected the bug to be in one of the following states: NEW, ASSIGNED, ON_DEV, POST, POST, but it is VERIFIED instead
expected Bugzilla bug 1788707 to depend on a bug in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but no dependents were found

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mtnbikenc · 2020-01-08T20:18:34Z

/bugzilla refresh

openshift-ci-robot · 2020-01-08T20:18:40Z

@mtnbikenc: This pull request references Bugzilla bug 1788707, which is valid. The bug has been moved to the POST state.

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jhixson74 · 2020-01-09T17:22:24Z

/test e2e-azure

wking · 2020-01-09T23:29:19Z

There's no 4.3 analog to this installer PR, and the 4.3 rhbz#1772804 is VERIFIED with no installer changes. Is that intentional? Can you explain why 4.3+ don't need this installer change?

wking · 2020-01-09T23:31:48Z

Azure job died with:

fail [github.com/openshift/origin/test/extended/prometheus/prometheus_builds.go:135]: Expected
    <map[string]error | len:1>: {
        "ALERTS{alertname!=\"Watchdog\",alertstate=\"firing\"} >= 1": {
            s: "promQL query: ALERTS{alertname!=\"Watchdog\",alertstate=\"firing\"} >= 1 had reported incorrect results: ALERTS{alertname=\"ClusterMonitoringOperatorDown\", alertstate=\"firing\", severity=\"critical\"} => 1 @[1578596289.224]\nALERTS{alertname=\"MachineAPIOperatorDown\", alertstate=\"firing\", severity=\"critical\"} => 1 @[1578596289.224]\nALERTS{alertname=\"TargetDown\", alertstate=\"firing\", job=\"api\", namespace=\"openshift-apiserver\", service=\"api\", severity=\"warning\"} => 1 @[1578596289.224]\nALERTS{alertname=\"TargetDown\", alertstate=\"firing\", job=\"catalog-operator-metrics\", namespace=\"openshift-operator-lifecycle-manager\", service=\"catalog-operator-metrics\", severity=\"warning\"} => 1 @[1578596289.224]\nALERTS{alertname=\"TargetDown\", alertstate=\"firing\", job=\"cluster-monitoring-operator\", namespace=\"openshift-monitoring\", service=\"cluster-monitoring-operator\", severity=\"warning\"} => 1 @[1578596289.224]\nALERTS{alertname=\"TargetDown\", alertstate=\"firing\", job=\"controller-manager\", namespace=\"openshift-controller-manager\", service=\"controller-manager\", severity=\"warning\"} => 1 @[1578596289.224]\nALERTS{alertname=\"TargetDown\", alertstate=\"firing\", job=\"dns-default\", namespace=\"openshift-dns\", service=\"dns-default\", severity=\"warning\"} => 1 @[1578596289.224]\nALERTS{alertname=\"TargetDown\", alertstate=\"firing\", job=\"machine-api-operator\", namespace=\"openshift-machine-api\", service=\"machine-api-operator\", severity=\"warning\"} => 1 @[1578596289.224]\nALERTS{alertname=\"TargetDown\", alertstate=\"firing\", job=\"metrics\", namespace=\"openshift-apiserver-operator\", service=\"metrics\", severity=\"warning\"} => 1 @[1578596289.224]\nALERTS{alertname=\"TargetDown\", alertstate=\"firing\", job=\"metrics\", namespace=\"openshift-authentication-operator\", service=\"metrics\", severity=\"warning\"} => 1 @[1578596289.224]\nALERTS{alertname=\"TargetDown\", alertstate=\"firing\", job=\"metrics\", namespace=\"openshift-controller-manager-operator\", service=\"metrics\", severity=\"warning\"} => 1 @[1578596289.224]\nALERTS{alertname=\"TargetDown\", alertstate=\"firing\", job=\"metrics\", namespace=\"openshift-kube-apiserver-operator\", service=\"metrics\", severity=\"warning\"} => 1 @[1578596289.224]\nALERTS{alertname=\"TargetDown\", alertstate=\"firing\", job=\"metrics\", namespace=\"openshift-kube-controller-manager-operator\", service=\"metrics\", severity=\"warning\"} => 1 @[1578596289.224]\nALERTS{alertname=\"TargetDown\", alertstate=\"firing\", job=\"metrics\", namespace=\"openshift-kube-scheduler-operator\", service=\"metrics\", severity=\"warning\"} => 1 @[1578596289.224]\nALERTS{alertname=\"TargetDown\", alertstate=\"firing\", job=\"olm-operator-metrics\", namespace=\"openshift-operator-lifecycle-manager\", service=\"olm-operator-metrics\", severity=\"warning\"} => 1 @[1578596289.224]",
        },
    }
to be empty

That's a MachineAPIOperatorDown and a bunch of TargetDown. I didn't dig deeper to see if they were related to this PR or not.

jhixson74 · 2020-01-09T23:41:00Z

There's no 4.3 analog to this installer PR, and the 4.3 rhbz#1772804 is VERIFIED with no installer changes. Is that intentional? Can you explain why 4.3+ don't need this installer change?

This code is a feature in 4.3. The same commits in this PR are already in 4.3. This PR is for back porting the changes to 4.2.

jhixson74 · 2020-01-09T23:45:42Z

Azure job died with:

fail [github.com/openshift/origin/test/extended/prometheus/prometheus_builds.go:135]: Expected
    <map[string]error | len:1>: {
        "ALERTS{alertname!=\"Watchdog\",alertstate=\"firing\"} >= 1": {
            s: "promQL query: ALERTS{alertname!=\"Watchdog\",alertstate=\"firing\"} >= 1 had reported incorrect results: ALERTS{alertname=\"ClusterMonitoringOperatorDown\", alertstate=\"firing\", severity=\"critical\"} => 1 @[1578596289.224]\nALERTS{alertname=\"MachineAPIOperatorDown\", alertstate=\"firing\", severity=\"critical\"} => 1 @[1578596289.224]\nALERTS{alertname=\"TargetDown\", alertstate=\"firing\", job=\"api\", namespace=\"openshift-apiserver\", service=\"api\", severity=\"warning\"} => 1 @[1578596289.224]\nALERTS{alertname=\"TargetDown\", alertstate=\"firing\", job=\"catalog-operator-metrics\", namespace=\"openshift-operator-lifecycle-manager\", service=\"catalog-operator-metrics\", severity=\"warning\"} => 1 @[1578596289.224]\nALERTS{alertname=\"TargetDown\", alertstate=\"firing\", job=\"cluster-monitoring-operator\", namespace=\"openshift-monitoring\", service=\"cluster-monitoring-operator\", severity=\"warning\"} => 1 @[1578596289.224]\nALERTS{alertname=\"TargetDown\", alertstate=\"firing\", job=\"controller-manager\", namespace=\"openshift-controller-manager\", service=\"controller-manager\", severity=\"warning\"} => 1 @[1578596289.224]\nALERTS{alertname=\"TargetDown\", alertstate=\"firing\", job=\"dns-default\", namespace=\"openshift-dns\", service=\"dns-default\", severity=\"warning\"} => 1 @[1578596289.224]\nALERTS{alertname=\"TargetDown\", alertstate=\"firing\", job=\"machine-api-operator\", namespace=\"openshift-machine-api\", service=\"machine-api-operator\", severity=\"warning\"} => 1 @[1578596289.224]\nALERTS{alertname=\"TargetDown\", alertstate=\"firing\", job=\"metrics\", namespace=\"openshift-apiserver-operator\", service=\"metrics\", severity=\"warning\"} => 1 @[1578596289.224]\nALERTS{alertname=\"TargetDown\", alertstate=\"firing\", job=\"metrics\", namespace=\"openshift-authentication-operator\", service=\"metrics\", severity=\"warning\"} => 1 @[1578596289.224]\nALERTS{alertname=\"TargetDown\", alertstate=\"firing\", job=\"metrics\", namespace=\"openshift-controller-manager-operator\", service=\"metrics\", severity=\"warning\"} => 1 @[1578596289.224]\nALERTS{alertname=\"TargetDown\", alertstate=\"firing\", job=\"metrics\", namespace=\"openshift-kube-apiserver-operator\", service=\"metrics\", severity=\"warning\"} => 1 @[1578596289.224]\nALERTS{alertname=\"TargetDown\", alertstate=\"firing\", job=\"metrics\", namespace=\"openshift-kube-controller-manager-operator\", service=\"metrics\", severity=\"warning\"} => 1 @[1578596289.224]\nALERTS{alertname=\"TargetDown\", alertstate=\"firing\", job=\"metrics\", namespace=\"openshift-kube-scheduler-operator\", service=\"metrics\", severity=\"warning\"} => 1 @[1578596289.224]\nALERTS{alertname=\"TargetDown\", alertstate=\"firing\", job=\"olm-operator-metrics\", namespace=\"openshift-operator-lifecycle-manager\", service=\"olm-operator-metrics\", severity=\"warning\"} => 1 @[1578596289.224]",
        },
    }
to be empty

That's a MachineAPIOperatorDown and a bunch of TargetDown. I didn't dig deeper to see if they were related to this PR or not.

The only problem I am aware of as to why the Azure test will fail is the ingress operator not having support for the new Azure private DNS in 4.2. The PR for that (openshift/cluster-ingress-operator#344) merged today, so the test should now work. I am not clear on if this change is immediate or if there is a delay before it is available for CI.

jhixson74 · 2020-01-10T18:57:23Z

/test e2e-azure

wking · 2020-01-10T23:11:21Z

I am not clear on if this change is immediate or if there is a delay before it is available for CI.

PR merging should be all that is needed before changes appear in subsequent CI jobs. But if you want to check for a particular job, can drill in like:

Details link for the e2e-azure job link from this PR to get to https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_installer/2825/pull-ci-openshift-installer-release-4.2-e2e-azure/85
Artifacts link to get to https://gcsweb-ci.svc.ci.openshift.org/gcs/origin-ci-test/pr-logs/pull/openshift_installer/2825/pull-ci-openshift-installer-release-4.2-e2e-azure/85/
Walk artifacts -> release-latest -> release-payload-latest -> image-references to get to https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_installer/2825/pull-ci-openshift-installer-release-4.2-e2e-azure/85/artifacts/release-latest/release-payload-latest/image-references , which has:
```
      {
        "name": "cluster-ingress-operator",
        "annotations": {
          "io.openshift.build.commit.id": "c0a5e5eb5a9e80dd304f4e49066e4bf52708be4a",
```
which is that PR landing: openshift/cluster-ingress-operator@c0a5e5e

wking · 2020-01-10T23:19:06Z

This code is a feature in 4.3.

So looks like this mostly backports #2470, but with 36e431a and 0ec99cc unique to the backport? 0ec99cc overlaps with, but is not the same as 48da378 (#2429). Oh, I see you have some commit references in your cherry-pick commit references. Looking...

wking · 2020-01-10T23:26:50Z

af00810 -> fa5412a is clean.
c1e24af -> 3c1118f drops the master private_dns_zone_id removal. I'm not clear on why, because that variable is in 4.2. Ah, then you clean it out later in 0ec99cc. You could squash that private_dns_zone_id removal down into 3c1118f to make the cherry-picks clearer, but the end effect is the same.

wking · 2020-01-10T23:32:13Z

7d3119f -> eb1884d is clean
e80fde1 -> 36e431a is clean
65111a0 -> 5e8e8a5 is clean
I don't understand 0ec99cc (except for the private_dns_zone_id recovery mentioned above).

jhixson74 · 2020-01-10T23:56:01Z

* [7d3119f](https://github.com/openshift/installer/commit/7d3119f8ba3ecefba5b1ec7168de526fdab81b6d) -> [eb1884d](https://github.com/openshift/installer/commit/eb1884da336e9b5d276f49378a1f9beaa9b5801f) is clean

* [e80fde1](https://github.com/openshift/installer/commit/e80fde1336b2c5feb400428b74364fc455b78c57) -> [36e431a](https://github.com/openshift/installer/commit/36e431a54bf18a472a1ea912d2ccac21820df22e) is clean

* [65111a0](https://github.com/openshift/installer/commit/65111a04a42cf864900086acdbb46ee86a8f9724) -> [5e8e8a5](https://github.com/openshift/installer/commit/5e8e8a5816ee6c50d9850cdfd73d10e05c800d8b) is clean

* I don't understand [0ec99cc](https://github.com/openshift/installer/commit/0ec99ccc4f6d1b68670fc8ac4b008a87a7822893) (except for the `private_dns_zone_id` recovery mentioned above).

master_subnet_cidr was nuked in 4.3 except in the vnet module, where it has become a local. The cherry pick relies on this, so that commit removed the older 4.2 references.

jstuever

/lgtm

wking · 2020-01-13T22:57:43Z

/approve

wking · 2020-01-14T05:00:17Z

/approve

Trying again.

jhixson74 · 2020-01-14T05:37:58Z

/assign @jhixson74

wking · 2020-01-14T15:09:18Z

/approve

Eventually Prow will hear about this ;).

openshift-ci-robot · 2020-01-14T15:09:53Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jstuever, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2020-01-21T03:38:45Z

@jhixson74: All pull requests linked via external trackers have merged. Bugzilla bug 1788707 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1788707: Backport Azure private DNS to 4.2 branch

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

abhinavdahiya added 5 commits December 16, 2019 11:59

data/azure: use azureprivatedns provider to create private records

3c1118f

Using the Private DNS Zone also allows us to remove our previous hack [1] [1]: openshift@8ac9ab4 (cherry picked from commit c1e24af) Conflicts: data/data/azure/master/variables.tf

asset: update dns manifest to use Private DNS Zone

eb1884d

the ingress-operator can handle the Private DNS Zone since [1] [1]: openshift/cluster-ingress-operator#300 (cherry picked from commit 7d3119f)

vendor: re-sync to include privatedns models

36e431a

(cherry picked from commit e80fde1)

openshift-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Dec 16, 2019

openshift-ci-robot requested review from jstuever and mtnbikenc December 16, 2019 20:05

jhixson74 changed the title ~~WIP: backport Azure private DNS to 4.2 branch~~ Backport Azure private DNS to 4.2 branch Dec 16, 2019

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 16, 2019

azure: remove references to var.master_subnet_cidr

0ec99cc

jhixson74 changed the title ~~Backport Azure private DNS to 4.2 branch~~ Bug 1788707: Backport Azure private DNS to 4.2 branch Jan 7, 2020

openshift-ci-robot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Jan 7, 2020

openshift-ci-robot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Jan 8, 2020

jstuever approved these changes Jan 13, 2020

View reviewed changes

openshift-ci-robot assigned jstuever Jan 13, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jan 13, 2020

jstuever removed their assignment Jan 13, 2020

openshift-ci-robot assigned jhixson74 Jan 14, 2020

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 14, 2020

crawford added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Jan 21, 2020

openshift-merge-robot merged commit 23438fc into openshift:release-4.2 Jan 21, 2020

Bug 1788707: Backport Azure private DNS to 4.2 branch #2825

Bug 1788707: Backport Azure private DNS to 4.2 branch #2825

Uh oh!

Conversation

jhixson74 commented Dec 16, 2019

Uh oh!

openshift-ci-robot commented Dec 16, 2019

Uh oh!

openshift-ci-robot commented Dec 16, 2019

Uh oh!

jhixson74 commented Jan 6, 2020

Uh oh!

openshift-ci-robot commented Jan 7, 2020

Uh oh!

openshift-ci-robot commented Jan 7, 2020

Uh oh!

jhixson74 commented Jan 7, 2020

Uh oh!

openshift-ci-robot commented Jan 7, 2020

Uh oh!

jhixson74 commented Jan 7, 2020

Uh oh!

openshift-ci-robot commented Jan 7, 2020

Uh oh!

jhixson74 commented Jan 7, 2020

Uh oh!

openshift-ci-robot commented Jan 7, 2020

Uh oh!

mtnbikenc commented Jan 8, 2020

Uh oh!

openshift-ci-robot commented Jan 8, 2020

Uh oh!

jhixson74 commented Jan 9, 2020

Uh oh!

wking commented Jan 9, 2020

Uh oh!

wking commented Jan 9, 2020

Uh oh!

jhixson74 commented Jan 9, 2020

Uh oh!

jhixson74 commented Jan 9, 2020

Uh oh!

jhixson74 commented Jan 10, 2020

Uh oh!

wking commented Jan 10, 2020

Uh oh!

wking commented Jan 10, 2020

Uh oh!

wking commented Jan 10, 2020

Uh oh!

wking commented Jan 10, 2020

Uh oh!

jhixson74 commented Jan 10, 2020

Uh oh!

jstuever left a comment

Choose a reason for hiding this comment

Uh oh!

wking commented Jan 13, 2020

Uh oh!

wking commented Jan 14, 2020

Uh oh!

jhixson74 commented Jan 14, 2020

Uh oh!

wking commented Jan 14, 2020

Uh oh!

openshift-ci-robot commented Jan 14, 2020

Uh oh!

openshift-ci-robot commented Jan 21, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants