Skip to content

OCPBUGS-59645: [release-4.19] BGP related backports#2748

Merged
openshift-merge-bot[bot] merged 9 commits intoopenshift:release-4.19from
jcaamano:bgp-backports-4.19-bis
Jul 25, 2025
Merged

OCPBUGS-59645: [release-4.19] BGP related backports#2748
openshift-merge-bot[bot] merged 9 commits intoopenshift:release-4.19from
jcaamano:bgp-backports-4.19-bis

Conversation

@jcaamano
Copy link
Copy Markdown
Contributor

@jcaamano jcaamano commented Jul 17, 2025

cherry-picked

5cc28c623 FRR-K8s webhook: promote to priviledged
7056e67b7 FRRK8s CRDs: align to upstream
0b29885cc FRRK8s webhook: align to upstream
acdd04316 FRRK8s webhook: webhook liveness / readiness from metrics to webhook
0dae0a03e (origin/nad-validation-nameOrSpec, nad-validation-nameOrSpec) Validate NAD name and spec only in multus admission controller
401f7b46b frr-k8s: stop listening for incoming connection in the bgp daemon
bc5f08ed5 Pass '--gateway-mode' flag for ovnkube-cluster-manager
acdd04316 FRRK8s webhook: webhook liveness / readiness from metrics to webhook
942c9c295 Configure use_ct_inv_match=false in nbdb when route advertisement is enabled

/hold
waiting for #2714 adn/or #2735 unless we decide otherwise

pliurh and others added 3 commits July 17, 2025 11:56
Some code (EgressSVC and BGP) in cluster-manager needs to know the
gateway mode.

Signed-off-by: Peng Liu <pliu@redhat.com>
(cherry picked from commit bc5f08e)
frr-k8s will not listen to port 179, and BGP peering can only be
established from OCP to external.

Signed-off-by: Konstantinos Karampogias <karampok@gmail.com>
(cherry picked from commit 401f7b4)
This should save up calls to the webhook and reduce latency when a NAD
is updated to add labels or annotations.

The root cause of the change is the OVNK BGP feature: when BGP is
enabled for the cluster default network, reconfiguration might cause
temporary disruptions. As part of this reconfiguration and necessary to
complete it, OVNK depends on annotating an internal NAD. We want to
avoid having to reach the webhook for this annotation because the
temporary disruption might prevent it and in that case the
reconfiguration won't complete.

Another possibility would be to filter out from validation the specific
internal NAD but this current approach might be more beneficial overall.

Signed-off-by: Jaime Caamaño Ruiz <jcaamano@redhat.com>
(cherry picked from commit 0dae0a0)
@openshift-ci openshift-ci Bot requested a review from fedepaol July 17, 2025 11:58
@openshift-ci openshift-ci Bot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jul 17, 2025
Comment thread bindata/network/frr-k8s/webhook.yaml Outdated
ports:
- containerPort: 7572
name: monitoring
name: frr-k8s-webhook-server
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'd still need - --webhook-port=9443 here as we are hostnetworking the pod and it needs to be a reserved one (for which I still have openshift/enhancements#1815 dangling)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

9123 you mean?
Also, you are saying passing --webhook-port arg would work for this PR?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

another go, PTAL!

@jcaamano jcaamano force-pushed the bgp-backports-4.19-bis branch from ead0114 to 4eea398 Compare July 17, 2025 14:38
fedepaol added 4 commits July 17, 2025 16:02
Instead of relying on the metrics port to understand when the webhook is
ready, we try to hit the endpoint of the webhook itself, so the webhook
pod is going to be ready only when it is effectively able to serve.

This is aligned with the frr-k8s binary change where we add a new
endpoint "healtz" to the webhook server.

Also, pinning the webhook port parameter and not relying on the default,
making the change more convenient.

Signed-off-by: Federico Paolinelli <fpaoline@redhat.com>
(cherry picked from commit acdd043)
Aligning to upstream and:
- move the webhook deployment to hostnetworked, so that the api can
  still be served if an offending FRRConfiguration is applied
- openshift only: change the webhook port to one in the allowed range
- remove the metrics listening port, as no service monitor was deployed

Signed-off-by: Federico Paolinelli <fpaoline@redhat.com>
(cherry picked from commit 0b29885)
Aligning to upstream and bringing the deprecation of the disableMP flag
and the introduction of a "dualStackAddressFamily" flag to bring the
behavior back to allow backward compatibility.

The default behavior was inconsistent in case of dual stack clusters, as
frr was being configured to advertise both ip families over a single
session, without being instructed properly to what next hop set for the
ip family not corresponding to the ip family of the session.

The dualStackAddressFamily flag is introduced to allow users relying on
that behavior to keep working. Note that both flags are not documented
nor supported d/s.

More details in the upstream metallb issue
metallb/metallb#2704

Signed-off-by: Federico Paolinelli <fpaoline@redhat.com>
(cherry picked from commit 7056e67)
We need to run the webhook as hostnetworked, so it requires the
privileged scc.

Signed-off-by: Federico Paolinelli <fpaoline@redhat.com>
(cherry picked from commit 5cc28c6)
@jcaamano jcaamano force-pushed the bgp-backports-4.19-bis branch from 4eea398 to 28d53ad Compare July 17, 2025 16:02
@karampok
Copy link
Copy Markdown
Contributor

fyi there is this #2732 (not sure if we should close it or merge it first)

We need this to be able to backport [1] after which we don't need it
anymore.

1 metallb/frr-k8s@963d15c

Signed-off-by: Jaime Caamaño Ruiz <jcaamano@redhat.com>
@jcaamano
Copy link
Copy Markdown
Contributor Author

fyi there is this #2732 (not sure if we should close it or merge it first)

Thanks for letting me know. Let's keep both open but give this one preference. If we hit a blocker here we can merge the other one.

@jcaamano
Copy link
Copy Markdown
Contributor Author

/test ?

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Jul 21, 2025

@jcaamano: The following commands are available to trigger required jobs:

/test 4.19-upgrade-from-stable-4.18-images
/test e2e-aws-ovn-hypershift-conformance
/test e2e-aws-ovn-upgrade
/test e2e-aws-ovn-upgrade-ipsec
/test e2e-aws-ovn-windows
/test e2e-azure-ovn-upgrade
/test e2e-gcp-ovn
/test e2e-gcp-ovn-techpreview
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-metal-ipi-ovn-ipv6-ipsec
/test e2e-ovn-ipsec-step-registry
/test hypershift-e2e-aks
/test images
/test lint
/test okd-scos-images
/test unit
/test verify
/test verify-deps

The following commands are available to trigger optional jobs:

/test 4.19-upgrade-from-stable-4.18-e2e-aws-ovn-upgrade
/test 4.19-upgrade-from-stable-4.18-e2e-aws-ovn-upgrade-ipsec
/test 4.19-upgrade-from-stable-4.18-e2e-azure-ovn-upgrade
/test 4.19-upgrade-from-stable-4.18-e2e-gcp-ovn-upgrade
/test e2e-aws-hypershift-ovn-kubevirt
/test e2e-aws-ovn-local-to-shared-gateway-mode-migration
/test e2e-aws-ovn-serial
/test e2e-aws-ovn-serial-ipsec
/test e2e-aws-ovn-shared-to-local-gateway-mode-migration
/test e2e-aws-ovn-single-node
/test e2e-aws-ovn-techpreview-serial
/test e2e-azure-ovn
/test e2e-azure-ovn-dualstack
/test e2e-azure-ovn-manual-oidc
/test e2e-metal-ipi-ovn-dualstack-bgp-local-gw-techpreview
/test e2e-metal-ipi-ovn-dualstack-bgp-techpreview
/test e2e-network-mtu-migration-ovn-ipv4
/test e2e-network-mtu-migration-ovn-ipv6
/test e2e-openstack-ovn
/test e2e-ovn-hybrid-step-registry
/test e2e-ovn-step-registry
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-dualstack
/test e2e-vsphere-ovn-dualstack-primaryv6
/test e2e-vsphere-ovn-windows
/test frrk8s-e2e
/test okd-scos-e2e-aws-ovn
/test qe-perfscale-aws-ovn-medium-cluster-density
/test qe-perfscale-aws-ovn-medium-node-density-cni
/test qe-perfscale-aws-ovn-small-cluster-density
/test qe-perfscale-aws-ovn-small-node-density-cni
/test security

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-cluster-network-operator-release-4.19-4.19-upgrade-from-stable-4.18-e2e-aws-ovn-upgrade
pull-ci-openshift-cluster-network-operator-release-4.19-4.19-upgrade-from-stable-4.18-e2e-aws-ovn-upgrade-ipsec
pull-ci-openshift-cluster-network-operator-release-4.19-4.19-upgrade-from-stable-4.18-e2e-azure-ovn-upgrade
pull-ci-openshift-cluster-network-operator-release-4.19-4.19-upgrade-from-stable-4.18-e2e-gcp-ovn-upgrade
pull-ci-openshift-cluster-network-operator-release-4.19-4.19-upgrade-from-stable-4.18-images
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-aws-hypershift-ovn-kubevirt
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-aws-ovn-hypershift-conformance
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-aws-ovn-local-to-shared-gateway-mode-migration
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-aws-ovn-serial
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-aws-ovn-serial-ipsec
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-aws-ovn-shared-to-local-gateway-mode-migration
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-aws-ovn-single-node
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-aws-ovn-upgrade
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-aws-ovn-upgrade-ipsec
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-aws-ovn-windows
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-azure-ovn
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-azure-ovn-upgrade
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-gcp-ovn
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-gcp-ovn-upgrade
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-metal-ipi-ovn-dualstack-bgp-local-gw-techpreview
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-metal-ipi-ovn-dualstack-bgp-techpreview
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-metal-ipi-ovn-ipv6
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-metal-ipi-ovn-ipv6-ipsec
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-network-mtu-migration-ovn-ipv4
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-network-mtu-migration-ovn-ipv6
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-openstack-ovn
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-ovn-hybrid-step-registry
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-ovn-ipsec-step-registry
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-ovn-step-registry
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-vsphere-ovn
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-vsphere-ovn-dualstack
pull-ci-openshift-cluster-network-operator-release-4.19-e2e-vsphere-ovn-dualstack-primaryv6
pull-ci-openshift-cluster-network-operator-release-4.19-frrk8s-e2e
pull-ci-openshift-cluster-network-operator-release-4.19-hypershift-e2e-aks
pull-ci-openshift-cluster-network-operator-release-4.19-images
pull-ci-openshift-cluster-network-operator-release-4.19-lint
pull-ci-openshift-cluster-network-operator-release-4.19-okd-scos-e2e-aws-ovn
pull-ci-openshift-cluster-network-operator-release-4.19-okd-scos-images
pull-ci-openshift-cluster-network-operator-release-4.19-security
pull-ci-openshift-cluster-network-operator-release-4.19-unit
pull-ci-openshift-cluster-network-operator-release-4.19-verify
pull-ci-openshift-cluster-network-operator-release-4.19-verify-deps
Details

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@jcaamano
Copy link
Copy Markdown
Contributor Author

/test e2e-metal-ipi-ovn-dualstack-bgp-local-gw-techpreview
/test e2e-metal-ipi-ovn-dualstack-bgp-techpreview

@jcaamano
Copy link
Copy Markdown
Contributor Author

/test e2e-metal-ipi-ovn-dualstack-bgp-local-gw-techpreview

…enabled

This is necessary to enable assymetric traffic for advertised Layer2 networks
running in local gateway mode.

Signed-off-by: Patryk Diak <pdiak@redhat.com>
(cherry picked from commit 942c9c2)
@jcaamano
Copy link
Copy Markdown
Contributor Author

/retitle OCPBUGS-59643: [release-4.19] BGP related backports

@openshift-ci openshift-ci Bot changed the title [release-4.19] BGP related backports OCPBUGS-59643: [release-4.19] BGP related backports Jul 22, 2025
@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jul 22, 2025
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@jcaamano: This pull request references Jira Issue OCPBUGS-59643, which is invalid:

  • expected the bug to target the "4.19.z" version, but no target version was set
  • release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
  • expected Jira Issue OCPBUGS-59643 to depend on a bug targeting a version in 4.20.0 and in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

cherry-picked

5cc28c623 FRR-K8s webhook: promote to priviledged
7056e67b7 FRRK8s CRDs: align to upstream
0b29885cc FRRK8s webhook: align to upstream
acdd04316 FRRK8s webhook: webhook liveness / readiness from metrics to webhook
0dae0a03e (origin/nad-validation-nameOrSpec, nad-validation-nameOrSpec) Validate NAD name and spec only in multus admission controller
401f7b46b frr-k8s: stop listening for incoming connection in the bgp daemon
bc5f08ed5 Pass '--gateway-mode' flag for ovnkube-cluster-manager
acdd04316 FRRK8s webhook: webhook liveness / readiness from metrics to webhook

/hold
waiting for #2714 adn/or #2735 unless we decide otherwise

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@jcaamano
Copy link
Copy Markdown
Contributor Author

/retitle OCPBUGS-59645: [release-4.19] BGP related backports

@openshift-ci openshift-ci Bot changed the title OCPBUGS-59643: [release-4.19] BGP related backports OCPBUGS-59645: [release-4.19] BGP related backports Jul 22, 2025
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@jcaamano: This pull request references Jira Issue OCPBUGS-59645, which is invalid:

  • release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
  • expected dependent Jira Issue OCPBUGS-59643 to target a version in 4.20.0, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

cherry-picked

5cc28c623 FRR-K8s webhook: promote to priviledged
7056e67b7 FRRK8s CRDs: align to upstream
0b29885cc FRRK8s webhook: align to upstream
acdd04316 FRRK8s webhook: webhook liveness / readiness from metrics to webhook
0dae0a03e (origin/nad-validation-nameOrSpec, nad-validation-nameOrSpec) Validate NAD name and spec only in multus admission controller
401f7b46b frr-k8s: stop listening for incoming connection in the bgp daemon
bc5f08ed5 Pass '--gateway-mode' flag for ovnkube-cluster-manager
acdd04316 FRRK8s webhook: webhook liveness / readiness from metrics to webhook

/hold
waiting for #2714 adn/or #2735 unless we decide otherwise

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@jcaamano
Copy link
Copy Markdown
Contributor Author

/hold cancel

@jcaamano
Copy link
Copy Markdown
Contributor Author

/retest

@jcaamano
Copy link
Copy Markdown
Contributor Author

/test hypershift-e2e-aks

2 similar comments
@jcaamano
Copy link
Copy Markdown
Contributor Author

/test hypershift-e2e-aks

@kyrtapz
Copy link
Copy Markdown
Contributor

kyrtapz commented Jul 24, 2025

/test hypershift-e2e-aks

@kyrtapz
Copy link
Copy Markdown
Contributor

kyrtapz commented Jul 24, 2025

- --namespace=$(NAMESPACE)
- --metrics-bind-address=:7572
- --metrics-bind-address=0
- --webhook-port=9123
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup, I have a dangling pr here, see my comment #2748 (comment)

Not sure who do I need to ask for merge though

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤦 sorry, completely missed that.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

np!

@kyrtapz
Copy link
Copy Markdown
Contributor

kyrtapz commented Jul 24, 2025

/retest

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Jul 24, 2025

@jcaamano: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/4.19-upgrade-from-stable-4.18-e2e-azure-ovn-upgrade 8b72568 link false /test 4.19-upgrade-from-stable-4.18-e2e-azure-ovn-upgrade
ci/prow/e2e-aws-hypershift-ovn-kubevirt 8b72568 link false /test e2e-aws-hypershift-ovn-kubevirt
ci/prow/e2e-aws-ovn-single-node 8b72568 link false /test e2e-aws-ovn-single-node
ci/prow/security 8b72568 link false /test security
ci/prow/4.19-upgrade-from-stable-4.18-e2e-gcp-ovn-upgrade 8b72568 link false /test 4.19-upgrade-from-stable-4.18-e2e-gcp-ovn-upgrade
ci/prow/e2e-vsphere-ovn-dualstack-primaryv6 8b72568 link false /test e2e-vsphere-ovn-dualstack-primaryv6

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@kyrtapz
Copy link
Copy Markdown
Contributor

kyrtapz commented Jul 25, 2025

/test e2e-aws-ovn-single-node

@kyrtapz
Copy link
Copy Markdown
Contributor

kyrtapz commented Jul 25, 2025

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 25, 2025
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Jul 25, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jcaamano, kyrtapz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kyrtapz
Copy link
Copy Markdown
Contributor

kyrtapz commented Jul 25, 2025

/label backport-risk-assessed

@openshift-ci openshift-ci Bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Jul 25, 2025
@openshift-merge-bot openshift-merge-bot Bot merged commit fd80a4a into openshift:release-4.19 Jul 25, 2025
37 of 43 checks passed
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@jcaamano: Jira Issue OCPBUGS-59645: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-59645 has been moved to the MODIFIED state.

Details

In response to this:

cherry-picked

5cc28c623 FRR-K8s webhook: promote to priviledged
7056e67b7 FRRK8s CRDs: align to upstream
0b29885cc FRRK8s webhook: align to upstream
acdd04316 FRRK8s webhook: webhook liveness / readiness from metrics to webhook
0dae0a03e (origin/nad-validation-nameOrSpec, nad-validation-nameOrSpec) Validate NAD name and spec only in multus admission controller
401f7b46b frr-k8s: stop listening for incoming connection in the bgp daemon
bc5f08ed5 Pass '--gateway-mode' flag for ovnkube-cluster-manager
acdd04316 FRRK8s webhook: webhook liveness / readiness from metrics to webhook
942c9c295 Configure use_ct_inv_match=false in nbdb when route advertisement is enabled

/hold
waiting for #2714 adn/or #2735 unless we decide otherwise

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-bot
Copy link
Copy Markdown
Contributor

[ART PR BUILD NOTIFIER]

Distgit: cluster-network-operator
This PR has been included in build cluster-network-operator-container-v4.19.0-202507251009.p0.gfd80a4a.assembly.stream.el9.
All builds following this will include this PR.

@openshift-merge-robot
Copy link
Copy Markdown
Contributor

Fix included in accepted release 4.19.0-0.nightly-2025-07-28-070511

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.