✨ ROSANetwork: new CRD & reconciler to provision network infrastructure for ROSA-HCP #5464

mzazrivec · 2025-04-09T19:27:34Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

This pull request implements CRD and a controller for provisioning complete networking infrastructure required to install a ROSA-HCP cluster in AWS. The proposal for this implementation has been described in #5381.

Under the hood, the implementation uses cloudformation stack and a static (i.e. no possibility of customization) cloudformation template from rosa-cli

This pull request depends on openshift/rosa#2904 (now merged).

Quick howto:

$ export ROSA_NETWORK_NAME=rosa-net-01
$ export AWS_REGION=us-west-2
$ export AVAILABILITY_ZONE_COUNT=2
$ export CIDR_BLOCK=10.0.0.0/16
$ clusterctl generate yaml --from templates/rosa-network.yaml > rosa-net-01.yaml
$ kubectl apply -f rosa-net-01.yaml

To use the ROSANetwork from ROSA control plane:

apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: ROSAControlPlane
metadata:
  name: rosa-hcp01-control-plane
  namespace: default
spec:
  rosaNetworkRef:
    name: rosa-net01

and skip / remove subnets and availability zones from the CP spec.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Checklist:

Release note:

New API for provisioning network infrastructure for ROSA clusters

k8s-ci-robot · 2025-04-09T19:27:44Z

Hi @mzazrivec. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

serngawy · 2025-04-10T14:39:00Z

config/crd/patches/webhook_in_rosanetworks.yaml

+    webhookClientConfig:
+      # this is "\n" used as a placeholder, otherwise it will be rejected by the apiserver for being blank,
+      # but we're going to set it later using the cert-manager (or potentially a patch if not using cert-manager)
+      caBundle: Cg==


No need to add the caBundle.

serngawy · 2025-04-10T14:45:18Z

exp/api/v1beta2/rosanetwork_types.go

+	Resource string `json:"resource"`
+
+	// Identified of the created resource. Will be filled in once the resource is created & ready
+	ID string `json:"ID"`


Suggested change

ID string `json:"ID"`

Id string `json:"id"`

Or resourceId

serngawy · 2025-04-10T14:46:01Z

exp/api/v1beta2/rosanetwork_types.go

+// CFResource groups information pertaining to a resource created as a part of a cloudformation stack
+type CFResource struct {
+	// Name of the created resource: NATGateway1, VPC, SecurityGroup, ...
+	Resource string `json:"resource"`


Suggested change

Resource string `json:"resource"`

Name string `json:"name"`

OR resourceName

serngawy · 2025-04-10T14:49:19Z

exp/api/v1beta2/rosanetwork_types.go

+	Status string `json:"status"`
+
+	// Message pertaining to the status of the resource
+	Reason string `json:"reason"`


message is better I guess ?

Suggested change

Reason string `json:"reason"`

Message string `json:"message"`

serngawy · 2025-04-10T14:51:37Z

exp/api/v1beta2/rosanetwork_types.go

+	// Availability zone of the subnet pair
+	AvailabilityZone string `json:"availabilityZone"`
+
+	// ID of the public subnet


Suggested change

// ID of the public subnet

// Public subnet Id ex; subnet-xxxxxxxxxx

serngawy · 2025-04-10T14:53:59Z

main.go

 		}
 	}

+	// TODO: feature gates?


I don't think we need a new feature gate, we can have it under ROSA feature gate

Yes. I did not mean a new feature gate here, just the existing rosa FG.

serngawy · 2025-04-23T11:46:57Z

you also need to update the ValidatingWebhookConfiguration and MutatingWebhookConfiguration here

serngawy

/ok-to-test

serngawy · 2025-04-24T13:43:59Z

exp/api/v1beta2/rosanetwork_types.go

+	// If no identity is specified, the default identity for this controller will be used.
+	//
+	// +optional
+	IdentityRef *infrav1.AWSIdentityReference `json:"identityRef,omitempty"`


Not sure, if we want to provide this option to end user. We don't do that with RosaControlPlane only default aws identity. However, we should provide OCM identityRef

Why shouldn't we provide this option to the end user? We need to specify the ref to the aws secret somehow. Here I'm just reusing existing structures & code.

What do you mean by OCM identity ref? OCM will not be involved here in any way.

well, to use openshift/rosa and establish ocm client you need to have ocm authentication. Is this not the case with the RosaNetwork CF stack creation?

No. No OCM credentials are needed for rosanet, just AWS credentials.

@serngawy Are you satisfied with the answers here?

@mzazrivec I do remember we discuss that, but after checking the ROSANetwork cloud formation stack template , there are tags added as rosa_hcp_policy and roas service here.
Those tags I think is used to check for privileges ?
I think we have to authenticate the ocm credential. Even if we don't need to create the CF stack but enduser must be a valid OCM user.

@serngawy What does creating VPC with certain tags and checking OCM credentials have to do with each other?

okay, as we discussed no need to have ocm authentication.

serngawy · 2025-08-26T11:51:02Z

config/crd/kustomization.yaml

 - patches/webhook_in_awsmanagedcontrolplanetemplates.yaml
 - patches/webhook_in_eksconfigs.yaml
 - patches/webhook_in_eksconfigtemplates.yaml
+#- patches/webhook_in_rosanetworks.yaml


remove comment line

serngawy · 2025-09-08T16:00:54Z

exp/controllers/rosanetwork_controller.go

+	subnets := make(map[string]*expinfrav1.ROSANetworkSubnet)
+
+	for _, resource := range rosaNet.Status.Resources {
+		if resource.ResourceType != "AWS::EC2::Subnet" { // Skip all non subnets
+			continue
+		}
+
+		az, err := r.awsClient.GetSubnetAvailabilityZone(resource.PhysicalID)
+		if err != nil {
+			return err
+		}
+
+		if subnets[az] == nil {
+			subnets[az] = &expinfrav1.ROSANetworkSubnet{
+				AvailabilityZone: az,
+				PublicSubnet:     "",
+				PrivateSubnet:    "",
+			}
+		}
+
+		if strings.HasPrefix(resource.LogicalID, "SubnetPrivate") {
+			subnets[az].PrivateSubnet = resource.PhysicalID
+		} else {
+			subnets[az].PublicSubnet = resource.PhysicalID
+		}
+	}
+
+	rosaNet.Status.Subnets = make([]expinfrav1.ROSANetworkSubnet, len(subnets))
+	for i, v := range slices.Collect(maps.Values(subnets)) {
+		rosaNet.Status.Subnets[i] = *v
+	}
+
+	return nil


Better golang style and avoiding extra loop

Suggested change

subnets := make(map[string]*expinfrav1.ROSANetworkSubnet)

for _, resource := range rosaNet.Status.Resources {

if resource.ResourceType != "AWS::EC2::Subnet" { // Skip all non subnets

continue

}

az, err := r.awsClient.GetSubnetAvailabilityZone(resource.PhysicalID)

if err != nil {

return err

}

if subnets[az] == nil {

subnets[az] = &expinfrav1.ROSANetworkSubnet{

AvailabilityZone: az,

PublicSubnet: "",

PrivateSubnet: "",

}

}

if strings.HasPrefix(resource.LogicalID, "SubnetPrivate") {

subnets[az].PrivateSubnet = resource.PhysicalID

} else {

subnets[az].PublicSubnet = resource.PhysicalID

}

}

rosaNet.Status.Subnets = make([]expinfrav1.ROSANetworkSubnet, len(subnets))

for i, v := range slices.Collect(maps.Values(subnets)) {

rosaNet.Status.Subnets[i] = *v

}

return nil

subnets := make(map[string]expinfrav1.ROSANetworkSubnet)

for _, resource := range rosaNet.Status.Resources {

if resource.ResourceType != "AWS::EC2::Subnet" {

continue

}

az, err := r.awsClient.GetSubnetAvailabilityZone(resource.PhysicalID)

if err != nil {

return fmt.Errorf("failed to get AZ for subnet %s: %w", resource.PhysicalID, err)

}

subnet := subnets[az]

subnet.AvailabilityZone = az

if strings.HasPrefix(resource.LogicalID, "SubnetPrivate") {

subnet.PrivateSubnet = resource.PhysicalID

} else {

subnet.PublicSubnet = resource.PhysicalID

}

subnets[az] = subnet

}

rosaNet.Status.Subnets = slices.Collect(maps.Values(subnets))

return nil

serngawy · 2025-09-18T12:23:44Z

main.go


+		setupLog.Debug("enabling ROSA network controller")
+		if err = (&expcontrollers.ROSANetworkReconciler{
+			Client: mgr.GetClient(),


please add watchFilterValue string

Suggested change

Client: mgr.GetClient(),

Client: mgr.GetClient(),

WatchFilterValue: watchFilterValue,

serngawy · 2025-09-18T16:38:23Z

exp/controllers/rosanetwork_controller.go

+// ROSANetworkReconciler reconciles a ROSANetwork object.
+type ROSANetworkReconciler struct {
+	client.Client
+	Log       logr.Logger


Please add the WatchFilterValue to the struct

Suggested change

Log logr.Logger

Log logr.Logger

WatchFilterValue string

serngawy · 2025-09-18T16:40:39Z

config/crd/patches/webhook_in_rosanetworks.yaml

No need for the webhook patch file as it is not included in kustomization.yaml

damdo · 2025-09-23T08:08:43Z

/assign @nrb @damdo @richardcase

damdo · 2025-09-30T09:44:56Z

/label tide/merge-method-squash

controlplane/rosa/api/v1beta2/rosacontrolplane_types.go

damdo · 2025-09-30T10:11:19Z

pkg/cloud/scope/session.go

 func getSessionName(region string, clusterScoper cloud.SessionMetadata) string {
-	return fmt.Sprintf("%s-%s-%s", region, clusterScoper.InfraClusterName(), clusterScoper.Namespace())
+	return fmt.Sprintf("%s-%s-%s-%s", region, clusterScoper.ControllerName(), clusterScoper.InfraClusterName(), clusterScoper.Namespace())
 }


This changes the session name in all the places already using this (not only ROSA ones). Is this a backward compatible change? (cc. @richardcase @punkwalker)

I guess it should be okay as when the capa-manager update happen all cache sessions will be re-created with the new session name.
Does the cache stored some how even when the pod re-created, we need to fail safe when loading session fail ?

controlplane/rosa/api/v1beta2/rosacontrolplane_types.go

controlplane/rosa/api/v1beta2/rosacontrolplane_webhook.go

damdo · 2025-09-30T10:19:52Z

controlplane/rosa/controllers/rosacontrolplane_controller.go

+		// Is the referenced ROSANetwork ready yet?
+		if !conditions.IsTrue(rosaNet, expinfrav1.ROSANetworkReadyCondition) {
+			rosaScope.Info(fmt.Sprintf("referenced ROSANetwork %s is not ready", rosaNet.Name))
+			return ctrl.Result{RequeueAfter: time.Minute}, nil


A minute seems quite a lot here, no?

It takes about 5 minutes to create the cloudformation stack, i.e. approximately 5 cycles through the reconciliation loop. I'm fine with making it smaller (suggestions welcome), but not quite sure how it would help or improve the situation.

Ok and do we need to requeue after or are we watching and we should get a reconciliation event anyway?

exp/controllers/rosanetwork_controller.go

k8s-ci-robot · 2025-10-01T13:47:32Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from nrb. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

damdo

/lgtm

Thanks for addressing my comments

/assign @nrb @richardcase

k8s-ci-robot · 2025-10-02T16:10:20Z

LGTM label has been added.

Git tree hash: 5f64d3fa3341e2ef3426645208e69a234b4147e3

k8s-ci-robot requested review from faiq and serngawy April 9, 2025 19:27

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Apr 9, 2025

mzazrivec force-pushed the rosa_network branch from 3435873 to 22dea0b Compare April 9, 2025 19:32

serngawy reviewed Apr 10, 2025

View reviewed changes

mzazrivec force-pushed the rosa_network branch 4 times, most recently from 5907fb1 to 24a5950 Compare April 24, 2025 13:20

mzazrivec force-pushed the rosa_network branch 3 times, most recently from a947563 to a255790 Compare May 19, 2025 13:43

mzazrivec force-pushed the rosa_network branch from a255790 to d923d6f Compare May 28, 2025 12:17

serngawy reviewed May 28, 2025

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 28, 2025

mzazrivec force-pushed the rosa_network branch 4 times, most recently from d2534a7 to dcc599d Compare June 9, 2025 08:19

mzazrivec force-pushed the rosa_network branch 2 times, most recently from 023f99b to 1729dff Compare June 27, 2025 12:31

mzazrivec force-pushed the rosa_network branch 10 times, most recently from 8d886df to c6b826d Compare September 3, 2025 15:25

serngawy reviewed Sep 8, 2025

View reviewed changes

mzazrivec force-pushed the rosa_network branch from c6b826d to dffef26 Compare September 9, 2025 09:40

serngawy approved these changes Sep 11, 2025

View reviewed changes

serngawy suggested changes Sep 18, 2025

View reviewed changes

mzazrivec force-pushed the rosa_network branch 2 times, most recently from 95b12df to bd3b722 Compare September 19, 2025 19:43

k8s-ci-robot assigned damdo, nrb and richardcase Sep 23, 2025

k8s-ci-robot added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Sep 30, 2025

damdo reviewed Sep 30, 2025

View reviewed changes

mzazrivec added 2 commits September 30, 2025 16:36

RosaNetwork: new CRD & reconciler to provision net infra for ROSA-HCP

343f1e0

ROSANetwork: tests

974568c

mzazrivec force-pushed the rosa_network branch from bd3b722 to 974568c Compare October 1, 2025 13:47

damdo reviewed Oct 2, 2025

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 2, 2025

	Reason string `json:"reason"`
	Message string `json:"message"`

	// ID of the public subnet
	// Public subnet Id ex; subnet-xxxxxxxxxx

	Client: mgr.GetClient(),
	Client: mgr.GetClient(),
	WatchFilterValue: watchFilterValue,

✨ ROSANetwork: new CRD & reconciler to provision network infrastructure for ROSA-HCP #5464

Are you sure you want to change the base?

✨ ROSANetwork: new CRD & reconciler to provision network infrastructure for ROSA-HCP #5464

Conversation

mzazrivec commented Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Apr 9, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

serngawy commented Apr 23, 2025

Uh oh!

serngawy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

serngawy Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

damdo commented Sep 23, 2025

Uh oh!

damdo commented Sep 30, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

mzazrivec commented Apr 9, 2025 •

edited

Loading

serngawy Aug 18, 2025 •

edited

Loading