Bug 1942657: Ingress operator stays degraded after privateZone fixed in DNS #641

lmzuccarelli · 2021-08-05T12:02:20Z

This bugfix addresses the BZ https://bugzilla.redhat.com/show_bug.cgi?id=1942657

Verification Steps

Create cluster via cluster-bot version 4.8.0
Start local dev - make run-local
Verify ingress is working

$oc get clusteroperators/ingress

NAME      VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
ingress   4.8.0     True        False         False      20m

Change tags.name (prefix with lmz)

$oc edit dns.config.openshift.io cluster

apiVersion: config.openshift.io/v1
  7 kind: DNS
  8 metadata:
  9   creationTimestamp: "2021-08-05T10:49:47Z"
 10   generation: 1
 11   name: cluster
 12   resourceVersion: "653"
 13   uid: d47dab3f-1955-418b-942b-ca002e7a3f99
 14 spec:
 15   baseDomain: ci-ln-tdn0hi2-d5d6b.origin-ci-int-aws.dev.rhcloud.com
 16   privateZone:
 17     tags:
 18       Name: lmz-ci-ln-tdn0hi2-d5d6b-kth95-int
 19       kubernetes.io/cluster/ci-ln-tdn0hi2-d5d6b-kth95: owned
 20   publicZone:
 21     id: Z2GYOLTZHS5VK
 22 status: {}

dns.config.openshift.io/cluster edited

Check ingress status has degraded

$ oc get clusteroperators/ingress
NAME      VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
ingress   4.8.0     False       True          True       4m12s

Check logs

2021-08-05T12:32:34.414+0100	INFO	operator.dns_controller	dns/controller.go:190	skipping zone to which the DNS record is already published	{"record": {"dnsName":"*.apps.ci-ln-tdn0hi2-d5d6b.origin-ci-int-aws.dev.rhcloud.com.","targets":["ac3079d0e08d7428b8aa6536d7c1057e-1686588591.us-west-2.elb.amazonaws.com"],"recordType":"CNAME","recordTTL":30}, "dnszone": {"id":"Z2GYOLTZHS5VK"}}
2021-08-05T12:33:04.415+0100	INFO	operator.dns_controller	controller/controller.go:298	reconciling	{"request": "openshift-ingress-operator/default-wildcard"}
2021-08-05T12:33:05.384+0100	ERROR	operator.dns_controller	dns/controller.go:190	failed to publish DNS record to zone	{"record": {"dnsName":"*.apps.ci-ln-tdn0hi2-d5d6b.origin-ci-int-aws.dev.rhcloud.com.","targets":["ac3079d0e08d7428b8aa6536d7c1057e-1686588591.us-west-2.elb.amazonaws.com"],"recordType":"CNAME","recordTTL":30}, "dnszone": {"tags":{"Name":"lmz-ci-ln-tdn0hi2-d5d6b-kth95-int","kubernetes.io/cluster/ci-ln-tdn0hi2-d5d6b-kth95":"owned"}}, "error": "failed to find hosted zone for record: no matching hosted zone found"}

Change the dns private zone config to original

$oc edit dns.config.openshift.io cluster
   
apiVersion: config.openshift.io/v1
kind: DNS
metadata:
  creationTimestamp: "2021-08-05T10:49:47Z"
  generation: 1
  name: cluster
  resourceVersion: "653"
  uid: d47dab3f-1955-418b-942b-ca002e7a3f99
spec:
  baseDomain: ci-ln-tdn0hi2-d5d6b.origin-ci-int-aws.dev.rhcloud.com
  privateZone:
    tags:
      Name: ci-ln-tdn0hi2-d5d6b-kth95-int
      kubernetes.io/cluster/ci-ln-tdn0hi2-d5d6b-kth95: owned
  publicZone:
    id: Z2GYOLTZHS5VK
status: {}

dns.config.openshift.io/cluster edited

Verify logs

2021-08-05T12:35:09.903+0100	INFO	operator.dns_controller	dns/controller.go:190	skipping zone to which the DNS record is already published	{"record": {"dnsName":"*.apps.ci-ln-tdn0hi2-d5d6b.origin-ci-int-aws.dev.rhcloud.com.","targets":["ac3079d0e08d7428b8aa6536d7c1057e-1686588591.us-west-2.elb.amazonaws.com"],"recordType":"CNAME","recordTTL":30}, "dnszone": {"tags":{"Name":"ci-ln-tdn0hi2-d5d6b-kth95-int","kubernetes.io/cluster/ci-ln-tdn0hi2-d5d6b-kth95":"owned"}}}
2021-08-05T12:35:09.903+0100	INFO	operator.dns_controller	dns/controller.go:190	skipping zone to which the DNS record is already published	{"record": {"dnsName":"*.apps.ci-ln-tdn0hi2-d5d6b.origin-ci-int-aws.dev.rhcloud.com.","targets":["ac3079d0e08d7428b8aa6536d7c1057e-1686588591.us-west-2.elb.amazonaws.com"],"recordType":"CNAME","recordTTL":30}, "dnszone": {"id":"Z2GYOLTZHS5VK"}}

Check ingress status

$ oc get clusteroperators/ingress
NAME      VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
ingress   4.8.0     True        False         False      83s

…in DNS

lmzuccarelli · 2021-08-05T13:12:19Z

/test e2e-upgrade

openshift-ci · 2021-08-05T13:12:27Z

@lmzuccarelli: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

/test e2e-aws
/test e2e-aws-operator
/test e2e-gcp-serial
/test e2e-upgrade
/test images
/test unit
/test verify

The following commands are available to trigger optional jobs:

/test e2e-aws-single-node
/test e2e-azure
/test e2e-azure-operator
/test e2e-gcp-operator

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-cluster-ingress-operator-master-e2e-aws
pull-ci-openshift-cluster-ingress-operator-master-e2e-aws-operator
pull-ci-openshift-cluster-ingress-operator-master-e2e-aws-single-node
pull-ci-openshift-cluster-ingress-operator-master-e2e-gcp-serial
pull-ci-openshift-cluster-ingress-operator-master-e2e-upgrade
pull-ci-openshift-cluster-ingress-operator-master-images
pull-ci-openshift-cluster-ingress-operator-master-unit
pull-ci-openshift-cluster-ingress-operator-master-verify

Details

In response to this:

/retest e2e-upgrade

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

lmzuccarelli · 2021-08-05T13:13:08Z

/test e2e-upgrade

frobware

I'd like to see a unit test for checkZoneInConfig.

pkg/operator/controller/ingress/status.go

frobware · 2021-08-05T13:51:19Z

Can we also create an e2e test for this? Looking at the PR description it seems the steps are straightforward.

lmzuccarelli · 2021-08-05T13:56:46Z

@frobware - thanks for the review

pkg/operator/controller/ingress/status_test.go

frobware

Overall LGTM. Just some things to tidy up in the e2e test so that we fail early.

test/e2e/dns_ingressdegrade_test.go

frobware · 2021-08-12T15:36:22Z

@lmzuccarelli thanks for the changes

/lgtm

frobware · 2021-08-12T15:38:42Z

/refresh

openshift-ci · 2021-08-12T15:38:42Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: frobware, lmzuccarelli

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [frobware]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2021-08-13T04:11:28Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-16T06:22:46Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-16T06:35:59Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-16T06:48:45Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-16T07:01:47Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-16T07:14:44Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-16T07:27:48Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-16T07:39:48Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-16T07:51:46Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-16T08:03:43Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-16T08:15:45Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

lmzuccarelli · 2021-08-16T08:25:51Z

/test e2e-aws-single-node

openshift-ci · 2021-08-16T09:14:07Z

@lmzuccarelli: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-aws-single-node	`5e47482`	link	`/test e2e-aws-single-node`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-bot · 2021-08-16T11:03:48Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-16T11:39:47Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

lmzuccarelli · 2021-08-16T15:42:11Z

/retest-required

rfredette · 2021-08-16T15:54:40Z

/skip

openshift-ci · 2021-08-16T15:58:17Z

@lmzuccarelli: This pull request references Bugzilla bug 1942657, which is invalid:

expected the bug to target the "4.9.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Bug 1942657: Ingress operator stays degraded after privateZone fixed in DNS

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

rfredette · 2021-08-16T15:59:30Z

/bugzilla refresh

openshift-ci · 2021-08-16T15:59:37Z

@rfredette: This pull request references Bugzilla bug 1942657, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.9.0) matches configured target release for branch (4.9.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @lihongan

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2021-08-16T17:40:38Z

@lmzuccarelli: All pull requests linked via external trackers have merged:

openshift/cluster-ingress-operator#641

Bugzilla bug 1942657 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1942657: Ingress operator stays degraded after privateZone fixed in DNS

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

lmzuccarelli added 4 commits June 21, 2021 13:45

Added lmzuccarelli to OWNERS

85f61cb

Merge branch 'openshift:master' into master

00a6bd8

Merge branch 'openshift:master' into master

519c742

Bug 1942657: Ingress operator stays degraded after privateZone fixed …

3fdb860

…in DNS

openshift-ci bot requested review from frobware and knobunc August 5, 2021 12:04

lmzuccarelli changed the title ~~Bugfix BZ1942657: Ingress operator stays degraded after privateZone fixed in DNS~~ Bug BZ1942657: Ingress operator stays degraded after privateZone fixed in DNS Aug 5, 2021

frobware requested changes Aug 5, 2021

View reviewed changes

Add e2e test for IngressStatus on DNS config change (degrade status)

9f32923

lmzuccarelli force-pushed the bugfix-BZ1942657 branch 2 times, most recently from d2f25e6 to 086e96d Compare August 9, 2021 08:28

frobware reviewed Aug 12, 2021

View reviewed changes

pkg/operator/controller/ingress/status_test.go Show resolved Hide resolved

frobware requested changes Aug 12, 2021

View reviewed changes

test/e2e/dns_ingressdegrade_test.go Outdated Show resolved Hide resolved

test/e2e/dns_ingressdegrade_test.go Outdated Show resolved Hide resolved

test/e2e/dns_ingressdegrade_test.go Outdated Show resolved Hide resolved

frobware reviewed Aug 12, 2021

View reviewed changes

test/e2e/dns_ingressdegrade_test.go Show resolved Hide resolved

lmzuccarelli force-pushed the bugfix-BZ1942657 branch from 07bd88c to a46910a Compare August 12, 2021 15:14

Add unit test for DNS degrade utility logic

5e47482

lmzuccarelli force-pushed the bugfix-BZ1942657 branch from a46910a to 5e47482 Compare August 12, 2021 15:33

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 12, 2021

frobware approved these changes Aug 12, 2021

View reviewed changes

openshift-ci bot assigned frobware Aug 12, 2021

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 12, 2021

lmzuccarelli changed the title ~~Bug BZ1942657: Ingress operator stays degraded after privateZone fixed in DNS~~ Bug 1942657: Ingress operator stays degraded after privateZone fixed in DNS Aug 16, 2021

openshift-ci bot added bugzilla/severity-low Referenced Bugzilla bug's severity is low for the branch this PR is targeting. bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Aug 16, 2021

openshift-ci bot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Aug 16, 2021

openshift-ci bot removed the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Aug 16, 2021

openshift-ci bot requested a review from lihongan August 16, 2021 15:59

openshift-ci bot merged commit e1a1618 into openshift:master Aug 16, 2021

Miciah mentioned this pull request Oct 15, 2021

Bug 2011972: Use fake dns provider with external cp topology only in IBM Cloud case #659

Merged

Miciah mentioned this pull request Jul 20, 2023

OCPBUGS-15978: Check public DNS zone when reporting status #967

Merged

Miciah mentioned this pull request Oct 10, 2024

OCPBUGS-43033: e2e/ingress_dns: support both private & public #1153

Merged

Bug 1942657: Ingress operator stays degraded after privateZone fixed in DNS #641

Bug 1942657: Ingress operator stays degraded after privateZone fixed in DNS #641

Uh oh!

Conversation

lmzuccarelli commented Aug 5, 2021

Uh oh!

lmzuccarelli commented Aug 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Aug 5, 2021

Uh oh!

lmzuccarelli commented Aug 5, 2021

Uh oh!

frobware left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

frobware commented Aug 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lmzuccarelli commented Aug 5, 2021

Uh oh!

Uh oh!

frobware left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

frobware commented Aug 12, 2021

Uh oh!

frobware commented Aug 12, 2021

Uh oh!

openshift-ci bot commented Aug 12, 2021

Uh oh!

openshift-bot commented Aug 13, 2021

Uh oh!

openshift-bot commented Aug 16, 2021

Uh oh!

openshift-bot commented Aug 16, 2021

Uh oh!

openshift-bot commented Aug 16, 2021

Uh oh!

openshift-bot commented Aug 16, 2021

Uh oh!

openshift-bot commented Aug 16, 2021

Uh oh!

openshift-bot commented Aug 16, 2021

Uh oh!

openshift-bot commented Aug 16, 2021

Uh oh!

openshift-bot commented Aug 16, 2021

Uh oh!

openshift-bot commented Aug 16, 2021

Uh oh!

openshift-bot commented Aug 16, 2021

Uh oh!

lmzuccarelli commented Aug 16, 2021

Uh oh!

openshift-ci bot commented Aug 16, 2021

Uh oh!

openshift-bot commented Aug 16, 2021

Uh oh!

openshift-bot commented Aug 16, 2021

Uh oh!

lmzuccarelli commented Aug 16, 2021

Uh oh!

rfredette commented Aug 16, 2021

Uh oh!

openshift-ci bot commented Aug 16, 2021

Uh oh!

rfredette commented Aug 16, 2021

Uh oh!

openshift-ci bot commented Aug 16, 2021

Uh oh!

openshift-ci bot commented Aug 16, 2021

lmzuccarelli commented Aug 5, 2021 •

edited

Loading

frobware commented Aug 5, 2021 •

edited

Loading