NE-1517: Set EIP for NLB Ingress controller. #1593

miheer · 2024-03-12T11:04:02Z

This EP describes how to set EIP for NLB ingress controller.

Epic link: https://issues.redhat.com/browse/NE-1274

openshift-ci-robot · 2024-03-22T16:06:53Z

@miheer: This pull request references NE-1517 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set.

Details

In response to this:

This EP describes how to set EIP for NLB ingress controller.

Epic link: https://issues.redhat.com/browse/NE-1274

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

candita · 2024-03-27T15:38:26Z

/assign @frobware
/assign @Miciah
/assign

frobware

My main concerns revolve around the expectation that updating EIP allocation IDs will automatically update an existing NLB. From my brief experimentation, it seems that the system only recognises these allocations, as specified by the annotation, at the time of creation. The document implies at several points that changing EIP allocation IDs is possible and that such changes will be reconciled with the existing ELB without necessitating its recreation. It would be beneficial to conduct further tests to determine if modifications to the annotation result in immediate updates on a functioning ELB.

frobware · 2024-03-28T11:36:25Z

enhancements/ingress/set_eip_nlb_ingress.md

+This is a feature request to enhance the IngressController API to be able to support static IPs during
+- Install time
+- Custom NLB ingress controller creation 
+- Reconfiguration of the router.


As elastic IP allocations can only be added at creation time, what does reconfiguration imply recreation?

I don't understand how the router is involved at all. Should this say "Updating the IngressController"? Andy's question then applies.

Updating the IngressController is accurate.

frobware · 2024-03-28T11:37:26Z

enhancements/ingress/set_eip_nlb_ingress.md

+
+- As an administrator, I want to provision EIPs and use them with an NLB IngressController.
+- As a user or admin, I want to ensure EIPs are used with NLB on default router at install time.
+- As a user, I want to reconfigure default router to use EIPs.


As per my previous comment: reconfiguration would have to be destruction followed by creation as elastic allocation IDs need to be done at creation time.

Yes the workflow has changed. We will recreate the LB.

frobware · 2024-03-28T11:41:35Z

enhancements/ingress/set_eip_nlb_ingress.md

+        eip-allocations: eipalloc-<redacted>,eipalloc-<redacted>,eipalloc-<redacted>,eipalloc-<redacted>,eipalloc-<redacted> 
+    subnets:
+    - subnet-1
+    - subnet-2
+    - subnet-3


In the proposed design, will there be a check to ensure the count of EIP allocation IDs matches the number of public subnets? Without this, one might only discover that the ingress controller cannot deploy after much progress in the installation process...

One option is we can add a field to mention the number of public subnets. Then using CEL at API level we could try to compare the number of eip-allocations they provided and the the number of public subnets

Or I think we may put this as a Non-Goal for this EP.

Andy said "in the installation process". Validation to ensure installation succeeds can be done by the installer.

I added this point in the Open Questions

frobware · 2024-03-28T11:42:03Z

enhancements/ingress/set_eip_nlb_ingress.md

+ # Please edit the object below. Lines beginning with a '#' will be ignored,
+ # and an empty file will abort the edit. If an error occurs while saving this file will be
+ # reopened with the relevant failures.
+ #


Nit: Maybe remove the noise?

frobware · 2024-03-28T11:44:07Z

enhancements/ingress/set_eip_nlb_ingress.md

+       aws:
+         type: NLB
+           networkLoadBalancer:
+             eip-allocations: eipalloc-<redacted>,eipalloc-<redacted>,eipalloc-<redacted>,eipalloc-<redacted>,eipalloc-<redacted>


Could we update the example to use sequential numbers for the EIP allocation IDs? This would make it clearer. Suggesting: "eipalloc-12345, eipalloc-12346, eipalloc-12347, eipalloc-12348, eipalloc-12349". Ditto for the other usage where we use .

frobware · 2024-03-28T11:56:41Z

enhancements/ingress/set_eip_nlb_ingress.md

+- The API field for `eip-allocations` field in the Ingress Controller CR needs to be set as follows
+  in the file `operator/v1/types_ingress.go`:
+```go
+// AWSNetworkLoadBalancerParameters holds configuration parameters for an
+// AWS Network load balancer.
+type AWSNetworkLoadBalancerParameters struct {
+    EIPAllocations EIPAllocations `json:"eip-allocations,omitempty"`        
+}
+
+type EIPAllocations string
+
+```


This appears to be duplication. Aren't the types also described above?

We can do something like this

// AWSNetworkLoadBalancerParameters holds configuration parameters for an // AWS Network load balancer. type AWSNetworkLoadBalancerParameters struct { EIPAllocations configv1.EIPAllocations `json:"eip-allocations,omitempty"` }

In operator/v1/types_ingress.go we already have a type ProviderLoadBalancerParameters with an AWSLoadBalancerParameters. Is that where this is supposed to go? https://github.com/openshift/api/blob/master/operator/v1/types_ingress.go#L473

not sure what you mean please check openshift/api@261093f#diff-891d2cdcb01603ef2a2c1bfb2f6ef32cf2acf8b1471871a5a6e27308d50050f9R641

Okay, my confusion was because the struct was empty to begin with. I thought you were adding a new struct type. Btw, your api PR shows this as a slice:

type AWSNetworkLoadBalancerParameters struct { EIPAllocations []configv1.EIPAllocations `json:"eipAllocations,omitempty"` }

OK so I hope your question is answered.

frobware · 2024-03-28T11:58:26Z

enhancements/ingress/set_eip_nlb_ingress.md

+   The admin sets the `eip-allocations` in the Ingress Controller CR. Then same process
+   as per point 1.c.ii and 1.c.iii is followed by the ingress operator. 
+
+#### 3. Update the existing EIP for the IngressController:


As previously mentioned, will changes to EIP allocations actually be reconciled after creation?

#1593 (comment)

@frobware we need to recreate the LB service.

frobware · 2024-03-28T12:00:11Z

enhancements/ingress/set_eip_nlb_ingress.md

+#### 3. Update the existing EIP for the IngressController:
+
+   When the admin changes the eip-allocations field in the Ingress Controller CR, the watch on the
+   `eip-allocations` field by the ingress operator scales a new deployment for the Ingress Controller CR.


Oh, so if you change the EIP allocations then you get a "brand new" ingresscontroller? That might be surprising as it will drop all traffic. Seems subtle.

@frobware I need to test again. It looks like if the service is updated with the annotation the kcm should update the eips. https://github.com/brooksgarrett/kubernetes/blob/72dd54ec4856ccb7bd2b08258d95f3d0b932fb11/pkg/cloudprovider/providers/aws/aws.go#L4157

It looks like https://github.com/openshift/enhancements/compare/45ad01df72b72174f9347fcb54179af4661cbd19..1137a8acba5a4c72ae496c34fa93b97dd18d3990 resolved this question by removing "scales a new deployment" and instead describing the workflow for recreating the service.

frobware · 2024-03-28T12:01:14Z

enhancements/ingress/set_eip_nlb_ingress.md

+  in the Ingress Controller CR and then test if the Ingress Controller
+  with service type load balancer with the annotation `service.beta.kubernetes.io/aws-load-balancer-eip-allocations`
+  has value of `eip-allocations` field from the Ingress Controller CR was updated.
+


Is there a plan for testing when the EIP allocations are plumbed in the install-config.yaml?

There will be unit tests in the installer.

frobware · 2024-03-28T12:02:19Z

enhancements/ingress/set_eip_nlb_ingress.md

+
+## Upgrade / Downgrade Strategy
+
+During upgrade, user should be able to populate the eip_allocations_field via installer


If EIP allocations are only reconciled at creation time can this work? The cluster is upgrading which implies it has a deployed ingresscontroller.

Yes you are right. During the upgrade we won't have this option for populating the field. So, the cluster admin will have to set the eip-allocations field once the cluster is installed.

Miciah · 2024-03-28T16:03:42Z

enhancements/ingress/set_eip_nlb_ingress.md

+- As an administrator, I want to provision EIPs and use them with an NLB IngressController.
+- As a user or admin, I want to ensure EIPs are used with NLB on default router at install time.
+- As a user, I want to reconfigure default router to use EIPs.
+- As a user of OpenShift on AWS (ROSA), I want to use static IPs (and therefore AWS Elastic IPs) so that 


These roles are ambiguous. Is the administrator or user in all cases the cluster administrator? If there are multiple roles, can you define them?

OK. From my understanding we shall have only one user i.e the cluster-admin as he is the one who is responsible for creating routers.

Miciah · 2024-03-28T16:14:30Z

enhancements/ingress/set_eip_nlb_ingress.md

+### User Stories
+
+- As an administrator, I want to provision EIPs and use them with an NLB IngressController.
+- As a user or admin, I want to ensure EIPs are used with NLB on default router at install time.
+- As a user, I want to reconfigure default router to use EIPs.
+- As a user of OpenShift on AWS (ROSA), I want to use static IPs (and therefore AWS Elastic IPs) so that 
+  I can configure appropriate firewall rules.
+  I want the default AWS Load Balancer that they use (NLB) for their router to use these EIPs.


Another one:

As a cluster-admin who has purchased a block of public IPv4 addresses, I want to use these IP addresses, so that I can avoid Amazon's new charge for public IP addresses.

miheer · 2024-03-29T20:21:11Z

My main concerns revolve around the expectation that updating EIP allocation IDs will automatically update an existing NLB. From my brief experimentation, it seems that the system only recognises these allocations, as specified by the annotation, at the time of creation. The document implies at several points that changing EIP allocation IDs is possible and that such changes will be reconciled with the existing ELB without necessitating its recreation. It would be beneficial to conduct further tests to determine if modifications to the annotation result in immediate updates on a functioning ELB.

@frobware #1593 (comment)

candita · 2024-04-01T14:59:18Z

enhancements/ingress/set_eip_nlb_ingress.md

+  - ""
+---
+
+# Set AWS EIP For NLB Ingress Controller


Please define EIP and NLB at least once in this proposal.

Yes added under "Proposal"

candita · 2024-04-01T15:38:30Z

enhancements/ingress/set_eip_nlb_ingress.md

+### Goals
+- Users are able to use EIP for a NLB Ingress Controller.
+- Users are able to create a NLB default Ingress Controller during install time with the EIPs specified during install.
+- Any existing ingress controller should be able to be reconfigured to use EIPs. 


As mentioned earlier, "reconfigure" should really be "update with restart", right?

I see that https://github.com/openshift/enhancements/compare/1137a8acba5a4c72ae496c34fa93b97dd18d3990..79d098963268d49c7e4b7d64ffe1e25878c2b6c0 implemented this suggestion.

To me, "restart" implies that the router pod has to be restarted. How about this?

- The ELB of an existing IngressController should be able to be reconfigured to use specific EIPs by updating the IngressController CR and recreating the ELB.

added with slight changes.

candita · 2024-04-01T17:18:17Z

enhancements/ingress/set_eip_nlb_ingress.md

+In this enhancement we would be adding API fields in the installer and the ingress controller CRD
+to set the AWS EIP for the NLB Ingress Controller. The cluster ingress operator will then create 
+the ingress controller and get the EIP assigned to the service LoadBalancer with the annotation 
+`service.beta.kubernetes.io/aws-load-balancer-eip-allocations`


Can you add a note about why the annotation is in kubernetes.io rather than openshift.io?

Looks like this was addressed in https://github.com/openshift/enhancements/compare/1137a8acba5a4c72ae496c34fa93b97dd18d3990..79d098963268d49c7e4b7d64ffe1e25878c2b6c0. However, that change uses service.beta.kubernetes.io/aws-load-balancer-eip-allocations as the target of a link, which will not work. Better to use https://kubernetes.io/docs/reference/labels-annotations-taints/#service-beta-kubernetes-io-aws-load-balancer-eip-allocations as a documentation reference.

candita · 2024-04-01T17:19:18Z

enhancements/ingress/set_eip_nlb_ingress.md

+cluster. A cluster administrator also has the rights to modify the cluster level components.
+
+#### Set EIP using an installer:
+1. Cluster administrator creates adds the following in the install-config.yaml


Suggested change

1. Cluster administrator creates adds the following in the install-config.yaml

1. Cluster administrator creates or adds the following in the install-config.yaml

updated in a different way

candita · 2024-11-06T18:21:22Z

enhancements/ingress/set_eip_nlb_ingress.md

+1. Shall we support updating EIP in the default Classic `IngressController` after installation ?
+   Please refer to [section](#updating-eip-in-the-default-classic-ingresscontroller-after-installation).


Suggested change

1. Shall we support updating EIP in the default Classic `IngressController` after installation ?

Please refer to [section](#updating-eip-in-the-default-classic-ingresscontroller-after-installation).

1. Shall we support updating EIP in the default Classic `IngressController` after installation ?

Yes. Please refer to [section](#updating-eip-in-the-default-classic-ingresscontroller-after-installation).

candita · 2024-11-06T18:22:35Z

enhancements/ingress/set_eip_nlb_ingress.md

+   The load balancer would be created with no eipAllocations and the service annotation `service.beta.kubernetes.io/aws-load-balancer-eip-allocations` will be removed.
+   For more check [this](#cluster-admin-wants-operator-to-automatically-effectuate-the-load-balancer-service-for-updated-ingress-controller-eipallocations-)
+   AWS will randomly assign the NLB's IP addresses from the IP address pools available.


Rearrange the sentences:

Suggested change

The load balancer would be created with no eipAllocations and the service annotation `service.beta.kubernetes.io/aws-load-balancer-eip-allocations` will be removed.

For more check [this](#cluster-admin-wants-operator-to-automatically-effectuate-the-load-balancer-service-for-updated-ingress-controller-eipallocations-)

AWS will randomly assign the NLB's IP addresses from the IP address pools available.

The load balancer would be created with no eipAllocations and the service annotation `service.beta.kubernetes.io/aws-load-balancer-eip-allocations` will be removed. AWS will randomly assign the NLB's IP addresses from the IP address pools available.

For more details, see [this](#cluster-admin-wants-operator-to-automatically-effectuate-the-load-balancer-service-for-updated-ingress-controller-eipallocations-)

candita · 2024-11-06T18:23:56Z

enhancements/ingress/set_eip_nlb_ingress.md

+5. Can you assign `eipAllocations` to an `IngressController` with scope `Internal` ?
+
+    No, eipAllocations can be provided only for an `IngressController` with scope `External`.
+    The default IC is set to [External](https://github.com/openshift/cluster-ingress-operator/blob/master/pkg/operator/operator.go#L476)


This link isn't correct, it doesn't point to scope.

candita · 2024-11-06T18:25:21Z

enhancements/ingress/set_eip_nlb_ingress.md

+    However, we will validate the `IngressController` CR at API using CEL to check if `eipAllocations` are provided only when `scope` is 
+    set to `External`.


Suggested change

However, we will validate the `IngressController` CR at API using CEL to check if `eipAllocations` are provided only when `scope` is

set to `External`.

The `IngressController` CR cannot be saved if `eipAllocations` are provided and `scope` is not

set to `External`.

candita · 2024-11-06T18:35:25Z

enhancements/ingress/set_eip_nlb_ingress.md

+3. **Future Possibilities**: Should the AWS CCM eventually supports reliably modifying subnets without deleting the
+   LB-type service, we can then easily adapt the Ingress Operator to immediately effectuate the `eipAllocations` and eliminate
+   the need for a cluster admin to delete the LB-type service.


This is not a drawback, and isn't related to immutability. It is really its own Alternative.

Suggested change

3. **Future Possibilities**: Should the AWS CCM eventually supports reliably modifying subnets without deleting the

LB-type service, we can then easily adapt the Ingress Operator to immediately effectuate the `eipAllocations` and eliminate

the need for a cluster admin to delete the LB-type service.

### Future Possibility based on changes to AWS CCM.

Should the AWS CCM eventually supports reliably modifying subnets without deleting the

LB-type service, we can then easily adapt the Ingress Operator to immediately effectuate the `eipAllocations` and eliminate the need for a cluster admin to delete the LB-type service.

candita · 2024-11-06T18:39:26Z

/remove-lifecycle-rotten

Hi @miheer, I have completed a final review on this EP. It's mostly nits, but some areas need bigger changes. Afaik none of my suggestions would need to be propagated to the implementation. If/when you address my suggestions, can you please either put a 👍🏼 on each, or explain why you didn't accept the suggestion. That will make it easier for me to add the lgtm.

candita · 2024-11-06T18:55:46Z

/remove-lifecycle rotten

openshift-bot · 2024-12-05T01:15:45Z

Inactive enhancement proposals go stale after 28d of inactivity.

See https://github.com/openshift/enhancements#life-cycle for details.

Mark the proposal as fresh by commenting /remove-lifecycle stale.
Stale proposals rot after an additional 7d of inactivity and eventually close.
Exclude this proposal from closing by commenting /lifecycle frozen.

If this proposal is safe to close now please do so with /close.

/lifecycle stale

lihongan · 2024-12-05T01:24:28Z

/remove-lifecycle stale

openshift-bot · 2025-01-02T09:15:10Z

Inactive enhancement proposals go stale after 28d of inactivity.

See https://github.com/openshift/enhancements#life-cycle for details.

Mark the proposal as fresh by commenting /remove-lifecycle stale.
Stale proposals rot after an additional 7d of inactivity and eventually close.
Exclude this proposal from closing by commenting /lifecycle frozen.

If this proposal is safe to close now please do so with /close.

/lifecycle stale

openshift-bot · 2025-01-10T00:45:19Z

Stale enhancement proposals rot after 7d of inactivity.

See https://github.com/openshift/enhancements#life-cycle for details.

Mark the proposal as fresh by commenting /remove-lifecycle rotten.
Rotten proposals close after an additional 7d of inactivity.
Exclude this proposal from closing by commenting /lifecycle frozen.

If this proposal is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

openshift-bot · 2025-01-17T08:15:15Z

Rotten enhancement proposals close after 7d of inactivity.

See https://github.com/openshift/enhancements#life-cycle for details.

Reopen the proposal by commenting /reopen.
Mark the proposal as fresh by commenting /remove-lifecycle rotten.
Exclude this proposal from closing again by commenting /lifecycle frozen.

/close

openshift-ci · 2025-01-17T08:15:25Z

@openshift-bot: Closed this PR.

Details

In response to this:

Rotten enhancement proposals close after 7d of inactivity.

See https://github.com/openshift/enhancements#life-cycle for details.

Reopen the proposal by commenting /reopen.
Mark the proposal as fresh by commenting /remove-lifecycle rotten.
Exclude this proposal from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

candita · 2025-01-27T17:18:19Z

/reopen

openshift-ci-robot · 2025-01-27T17:18:43Z

@miheer: This pull request references NE-1517 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

Details

In response to this:

This EP describes how to set EIP for NLB ingress controller.

Epic link: https://issues.redhat.com/browse/NE-1274

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2025-01-27T17:18:44Z

@candita: Reopened this PR.

Details

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

candita · 2025-01-27T19:40:58Z

Team decided to go ahead with this change, disregarding the final commentary that was not incorporated.
/lgtm
/approve

openshift-ci · 2025-01-27T19:41:42Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: candita

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~enhancements/ingress/OWNERS~~ [candita]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2025-01-27T20:12:54Z

@miheer: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 12, 2024

openshift-ci bot requested review from candita and frobware March 12, 2024 11:04

miheer force-pushed the eip branch 7 times, most recently from bc40070 to 4b14bb9 Compare March 22, 2024 09:46

miheer changed the title ~~WIP: Set EIP for NLB Ingress controller.~~ Set EIP for NLB Ingress controller. Mar 22, 2024

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 22, 2024

miheer changed the title ~~Set EIP for NLB Ingress controller.~~ NE-1517: Set EIP for NLB Ingress controller. Mar 22, 2024

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Mar 22, 2024

miheer force-pushed the eip branch 3 times, most recently from 9e62c06 to 45ad01d Compare March 24, 2024 20:52

openshift-ci bot assigned candita, frobware and Miciah Mar 27, 2024

frobware reviewed Mar 28, 2024

View reviewed changes

Miciah reviewed Mar 28, 2024

View reviewed changes

candita reviewed Apr 1, 2024

View reviewed changes

candita reviewed Nov 6, 2024

View reviewed changes

openshift-ci bot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Nov 6, 2024

openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 5, 2024

openshift-ci bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 5, 2024

openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 2, 2025

openshift-ci bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jan 10, 2025

openshift-ci bot closed this Jan 17, 2025

openshift-ci bot reopened this Jan 27, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jan 27, 2025

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 27, 2025

openshift-merge-bot bot merged commit 5840551 into openshift:master Jan 27, 2025
2 checks passed


		## Upgrade / Downgrade Strategy

		During upgrade, user should be able to populate the eip_allocations_field via installer

	1. Cluster administrator creates adds the following in the install-config.yaml
	1. Cluster administrator creates or adds the following in the install-config.yaml

		1. Shall we support updating EIP in the default Classic `IngressController` after installation ?
		Please refer to [section](#updating-eip-in-the-default-classic-ingresscontroller-after-installation).

		However, we will validate the `IngressController` CR at API using CEL to check if `eipAllocations` are provided only when `scope` is
		set to `External`.

NE-1517: Set EIP for NLB Ingress controller. #1593

NE-1517: Set EIP for NLB Ingress controller. #1593

Uh oh!

Conversation

miheer commented Mar 12, 2024

Uh oh!

openshift-ci-robot commented Mar 22, 2024 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

candita commented Mar 27, 2024

Uh oh!

frobware left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

frobware Mar 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

miheer Mar 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

candita Apr 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

openshift-ci-robot commented Mar 22, 2024 •

edited by openshift-ci bot

Loading

frobware left a comment •

edited

Loading

frobware Mar 28, 2024 •

edited

Loading

miheer Mar 29, 2024 •

edited

Loading

candita Apr 1, 2024 •

edited

Loading

miheer commented Mar 29, 2024 •

edited

Loading

candita Apr 1, 2024 •

edited

Loading