Skip to content

feat: add nodepool scoped maestro readonly bundles controllers - #4599

Merged
openshift-merge-bot[bot] merged 16 commits into
mainfrom
msoriano-add-maestro-readonlynodepools-controllers
Apr 20, 2026
Merged

feat: add nodepool scoped maestro readonly bundles controllers#4599
openshift-merge-bot[bot] merged 16 commits into
mainfrom
msoriano-add-maestro-readonlynodepools-controllers

Conversation

@miguelsorianod

@miguelsorianod Miguel Soriano (miguelsorianod) commented Mar 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

We add nodepool-scoped controllers in backend that are able to manage maestro readonly bundles.
With it, we introduce the ability to read the Hypershift's NodePool CRs associated to the node pools and persist it in Cosmos.

The Cosmos ManagementClusterContent type is a direct child of the node pool Cosmos type. Documents within the ManagementClusterContent Cosmos type are different readonly bundles internal references. A new ManagementClusterContent document per each Node Pool of the cluster is added. The maestro bundle internal name references associated to the hypershift nodepools is readonlyHypershiftNodePool.

@miguelsorianod

Miguel Soriano (miguelsorianod) commented Mar 24, 2026

Copy link
Copy Markdown
Collaborator Author

/hold

@miguelsorianod
Miguel Soriano (miguelsorianod) force-pushed the msoriano-add-maestro-readonlynodepools-controllers branch from 7bdf0d9 to 60ec28b Compare March 24, 2026 17:56
@machi1990

Manyanda Chitimbo (machi1990) commented Mar 24, 2026

Copy link
Copy Markdown
Collaborator

Alba Hita (@ahitacat) this might interest you
cc Jameel Briones (@JameelB) as well

@miguelsorianod
Miguel Soriano (miguelsorianod) force-pushed the msoriano-add-maestro-readonlynodepools-controllers branch from 60ec28b to 2cb3525 Compare March 30, 2026 12:34
@miguelsorianod

Miguel Soriano (miguelsorianod) commented Mar 30, 2026

Copy link
Copy Markdown
Collaborator Author

I've updated the PR and now the maestro readonly bundle of the nodepool's hypershift nodepool CR belongs to a Cosmos ManagementClusterContent named readonlyHypershiftNodePool, where ManagementClusterContent is a child resource of NodePool. With this, we have Cosmos ManagementClusterContent resources at both the cluster and the nodepool level:

/subscriptions/<subscription>/resourceGroups/<rg>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/<clustername>/managementClusterContents/<managementClusterContentName>

/subscriptions/<subscription>/resourceGroups/<rg>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/<clustername>/nodePools/<nodePoolName>/managementClusterContents/<managementClusterContentName>.

In that way:

  • We avoid potential collisions of the managementClusterContent name between nodepools in the same cluster, as well as cross cluster
  • We get the benefits of hierarchical Cosmos DB deletion when its parent resource (cluster or nodepool) is deleted

@miguelsorianod

Miguel Soriano (miguelsorianod) commented Mar 30, 2026

Copy link
Copy Markdown
Collaborator Author

In https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/Azure_ARO-HCP/4599/pull-ci-Azure-ARO-HCP-main-e2e-parallel/2038595829597474816 an example execution can be found.

Taking an example the ARO-HCP Cluster /subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster and one of its Node Pools /subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/nodePools/arm64-vm-np-1, we can observe how their corresponding maestro readonly bundles and management cluster contents are created, as well as their deletion:

cluster maestro readonly bundle 
"timestamp": 2026-03-30T13:34:46.870Z,
"log": {
	"time": "2026-03-30T13:34:46.8708793Z",
	"level": "INFO",
	"source": {
		"function": "github.com/Azure/ARO-HCP/internal/serverutils.DumpDataToLogger-range1",
		"file": "/app/internal/serverutils/dump_data.go",
		"line": 53
	},
	"msg": "dumping resourceID /subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/managementClusterContents/readonlyHypershiftHostedCluster",
	"controller_name": "datadump",
	"subscription_id": "974ebd46-8ad3-41e3-afef-7ef25fd5c371",
	"resource_group": "arm64-vm-cluster-n2ktdt",
	"resourceType": "microsoft.redhatopenshift/hcpopenshiftclusters",
	"resource_name": "arm64-vm-hcp-cluster",
	"resource_id": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourcegroups/arm64-vm-cluster-n2ktdt/providers/microsoft.redhatopenshift/hcpopenshiftclusters/arm64-vm-hcp-cluster",
	"hcp_cluster_name": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourcegroups/arm64-vm-cluster-n2ktdt/providers/microsoft.redhatopenshift/hcpopenshiftclusters/arm64-vm-hcp-cluster",
	"currentResourceID": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/managementClusterContents/readonlyHypershiftHostedCluster",
	"content": {
		"resourceType": "Microsoft.RedHatOpenShift/hcpOpenShiftClusters/managementClusterContents",
		"id": "ba832971-30eb-588f-a1be-1e532958b2a6",
		"_rid": "425DAJSnY-BgAQAAAAAAAA==",
		"_self": "dbs/425DAA==/colls/425DAJSnY-A=/docs/425DAJSnY-BgAQAAAAAAAA==/",
		"_etag": "\"26000783-0000-4d00-0000-69ca7bc30000\"",
		"_attachments": "attachments/",
		"_ts": 1774877635,
		"partitionKey": "974ebd46-8ad3-41e3-afef-7ef25fd5c371",
		"resourceID": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/managementClusterContents/readonlyHypershiftHostedCluster",
		"properties": {
			"status": {
				"conditions": [
					{
						"type": "Degraded",
						"status": "False",
						"lastTransitionTime": "2026-03-30T13:33:55.9314012Z",
						"reason": "NoErrors",
						"message": "As expected."
					}
				],
				"kubeContent": {
					"metadata": {},
					"items": [
						{
							"status": {
								"version": {
									"observedGeneration": 1,
									"availableUpdates": null,
									"desired": {
										"version": "",
										"image": "quay.io/openshift-release-dev/ocp-release@sha256:3fac61ad81b3b209c7b2e617ad73f91f79c83692b1cb8196cdeb8567543c4b43"
									},
									"history": [
										{
											"version": "",
											"state": "Partial",
											"image": "quay.io/openshift-release-dev/ocp-release@sha256:3fac61ad81b3b209c7b2e617ad73f91f79c83692b1cb8196cdeb8567543c4b43",
											"completionTime": null,
											"startedTime": "2026-03-30T13:31:06.0000000Z",
											"verified": false
										}
									]
								},
								"conditions": [
									{
										"type": "ClusterSizeComputed",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:31:06.0000000Z",
										"reason": "e2e_minimal",
										"message": "The HostedCluster has transitioned to a new t-shirt size."
									},
									{
										"type": "ClusterSizeTransitionPending",
										"status": "False",
										"lastTransitionTime": "2026-03-30T13:31:06.0000000Z",
										"reason": "ClusterSizeTransitioned",
										"message": "The HostedCluster has transitioned to a new t-shirt size."
									},
									{
										"type": "ClusterSizeTransitionRequired",
										"status": "False",
										"lastTransitionTime": "2026-03-30T13:31:06.0000000Z",
										"reason": "AsExpected",
										"message": "The HostedCluster has transitioned to a new t-shirt size."
									},
									{
										"type": "ClusterVersionSucceeding",
										"status": "Unknown",
										"lastTransitionTime": "2026-03-30T13:31:06.0000000Z",
										"reason": "StatusUnknown",
										"message": "Condition not found in the CVO.",
										"observedGeneration": 1
									},
									{
										"type": "ClusterVersionProgressing",
										"status": "Unknown",
										"lastTransitionTime": "2026-03-30T13:31:06.0000000Z",
										"reason": "StatusUnknown",
										"message": "Condition not found in the CVO.",
										"observedGeneration": 1
									},
									{
										"type": "ClusterVersionReleaseAccepted",
										"status": "Unknown",
										"lastTransitionTime": "2026-03-30T13:31:06.0000000Z",
										"reason": "StatusUnknown",
										"message": "Condition not found in the CVO.",
										"observedGeneration": 1
									},
									{
										"type": "ClusterVersionUpgradeable",
										"status": "Unknown",
										"lastTransitionTime": "2026-03-30T13:31:06.0000000Z",
										"reason": "StatusUnknown",
										"message": "Condition not found in the CVO.",
										"observedGeneration": 1
									},
									{
										"type": "ClusterVersionAvailable",
										"status": "Unknown",
										"lastTransitionTime": "2026-03-30T13:31:06.0000000Z",
										"reason": "StatusUnknown",
										"message": "Condition not found in the CVO.",
										"observedGeneration": 1
									},
									{
										"type": "Degraded",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:33:14.0000000Z",
										"reason": "UnavailableReplicas",
										"message": "[catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]",
										"observedGeneration": 1
									},
									{
										"type": "EtcdAvailable",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:32:31.0000000Z",
										"reason": "QuorumAvailable",
										"message": "",
										"observedGeneration": 1
									},
									{
										"type": "KubeAPIServerAvailable",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:33:01.0000000Z",
										"reason": "AsExpected",
										"message": "Kube APIServer deployment is available",
										"observedGeneration": 1
									},
									{
										"type": "InfrastructureReady",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:31:39.0000000Z",
										"reason": "AsExpected",
										"message": "All is well",
										"observedGeneration": 1
									},
									{
										"type": "ExternalDNSReachable",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:31:27.0000000Z",
										"reason": "AsExpected",
										"message": "All is well",
										"observedGeneration": 1
									},
									{
										"type": "ValidHostedControlPlaneConfiguration",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:31:27.0000000Z",
										"reason": "AsExpected",
										"message": "Configuration passes validation",
										"observedGeneration": 1
									},
									{
										"type": "ValidReleaseInfo",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:31:37.0000000Z",
										"reason": "AsExpected",
										"message": "All is well",
										"observedGeneration": 1
									},
									{
										"type": "ValidIDPConfiguration",
										"status": "Unknown",
										"lastTransitionTime": "2026-03-30T13:31:06.0000000Z",
										"reason": "StatusUnknown",
										"message": "Condition not found in the HCP",
										"observedGeneration": 1
									},
									{
										"type": "HostedClusterRestoredFromBackup",
										"status": "Unknown",
										"lastTransitionTime": "2026-03-30T13:31:06.0000000Z",
										"reason": "StatusUnknown",
										"message": "Condition not found in the HCP",
										"observedGeneration": 1
									},
									{
										"type": "DataPlaneConnectionAvailable",
										"status": "Unknown",
										"lastTransitionTime": "2026-03-30T13:31:06.0000000Z",
										"reason": "StatusUnknown",
										"message": "Condition not found in the HCP",
										"observedGeneration": 1
									},
									{
										"type": "ControlPlaneConnectionAvailable",
										"status": "Unknown",
										"lastTransitionTime": "2026-03-30T13:31:06.0000000Z",
										"reason": "StatusUnknown",
										"message": "Condition not found in the HCP",
										"observedGeneration": 1
									},
									{
										"type": "Available",
										"status": "False",
										"lastTransitionTime": "2026-03-30T13:31:06.0000000Z",
										"reason": "ComponentsNotAvailable",
										"message": "Waiting for components to be available: olm-operator, redhat-marketplace-catalog, community-operators-catalog, openshift-controller-manager, openshift-route-controller-manager, cluster-node-tuning-operator, konnectivity-agent, hosted-cluster-config-operator, cluster-storage-operator, redhat-operators-catalog, csi-snapshot-controller-operator, catalog-operator, cluster-policy-controller, dns-operator, cluster-image-registry-operator, ingress-operator, cluster-version-operator, cluster-network-operator, packageserver, certified-operators-catalog",
										"observedGeneration": 1
									},
									{
										"type": "ValidConfiguration",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:31:07.0000000Z",
										"reason": "AsExpected",
										"message": "Configuration passes validation",
										"observedGeneration": 1
									},
									{
										"type": "SupportedHostedCluster",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:31:06.0000000Z",
										"reason": "AsExpected",
										"message": "HostedCluster is supported by operator configuration",
										"observedGeneration": 1
									},
									{
										"type": "ValidProxyConfiguration",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:31:06.0000000Z",
										"reason": "AsExpected",
										"message": "No proxy CA bundle configured",
										"observedGeneration": 1
									},
									{
										"type": "IgnitionEndpointAvailable",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:33:16.0000000Z",
										"reason": "AsExpected",
										"message": "Ignition server deployment is available",
										"observedGeneration": 1
									},
									{
										"type": "ReconciliationActive",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:31:06.0000000Z",
										"reason": "AsExpected",
										"message": "Reconciliation active on resource",
										"observedGeneration": 1
									},
									{
										"type": "ValidReleaseImage",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:31:06.0000000Z",
										"reason": "AsExpected",
										"message": "Release image is valid",
										"observedGeneration": 1
									},
									{
										"type": "Progressing",
										"status": "False",
										"lastTransitionTime": "2026-03-30T13:31:06.0000000Z",
										"reason": "AsExpected",
										"message": "HostedCluster is at expected version",
										"observedGeneration": 1
									},
									{
										"type": "ReconciliationSucceeded",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:31:21.0000000Z",
										"reason": "ReconciliatonSucceeded",
										"message": "Reconciliation completed successfully",
										"observedGeneration": 1
									},
									{
										"type": "PlatformCredentialsFound",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:31:07.0000000Z",
										"reason": "AsExpected",
										"message": "Required platform credentials are found",
										"observedGeneration": 1
									},
									{
										"type": "ValidAzureKMSConfig",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:31:27.0000000Z",
										"reason": "AsExpected",
										"message": "All is well",
										"observedGeneration": 1
									}
								],
								"controlPlaneEndpoint": {
									"host": "api-int.g9s1j1d6a6y5b9t.iuh0.j7474816.hcp.osadev.cloud",
									"port": 443
								},
								"customKubeconfig": {
									"name": "g9s1j1d6a6y5b9t-custom-admin-kubeconfig"
								},
								"ignitionEndpoint": "ignition-server.g9s1j1d6a6y5b9t.iuh0.j7474816.hcp.osadev.cloud",
								"kubeconfig": {
									"name": "g9s1j1d6a6y5b9t-admin-kubeconfig"
								},
								"payloadArch": "Multi"
							},
							"spec": {
								"dns": {
									"baseDomainPrefix": "",
									"baseDomain": "aro.g9s1j1d6a6y5b9t.iuh0.j7474816.hcp.osadev.cloud",
									"publicZoneID": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt--managed/providers/Microsoft.Network/dnszones/aro.g9s1j1d6a6y5b9t.iuh0.j7474816.hcp.osadev.cloud"
								},
								"platform": {
									"type": "Azure",
									"azure": {
										"location": "westus3",
										"tenantID": "64dc69e4-d083-49fc-9569-ebece1dd1408",
										"azureAuthenticationConfig": {
											"azureAuthenticationConfigType": "ManagedIdentities",
											"managedIdentities": {
												"controlPlane": {
													"file": {
														"credentialsSecretName": "uamsi-2pc2p4sbgf0l2k3pp45jh8desquimj0f-file-csi-driver",
														"objectEncoding": "utf-8"
													},
													"network": {
														"credentialsSecretName": "uamsi-2pc2p4sbgf0l2k3pp45jh8desquimj0f-cloud-network-config",
														"objectEncoding": "utf-8"
													},
													"ingress": {
														"credentialsSecretName": "uamsi-2pc2p4sbgf0l2k3pp45jh8desquimj0f-ingress",
														"objectEncoding": "utf-8"
													},
													"cloudProvider": {
														"credentialsSecretName": "uamsi-2pc2p4sbgf0l2k3pp45jh8desquimj0f-cloud-controller-manager",
														"objectEncoding": "utf-8"
													},
													"controlPlaneOperator": {
														"credentialsSecretName": "uamsi-2pc2p4sbgf0l2k3pp45jh8desquimj0f-control-plane",
														"objectEncoding": "utf-8"
													},
													"disk": {
														"credentialsSecretName": "uamsi-2pc2p4sbgf0l2k3pp45jh8desquimj0f-disk-csi-driver",
														"objectEncoding": "utf-8"
													},
													"imageRegistry": {
														"credentialsSecretName": "uamsi-2pc2p4sbgf0l2k3pp45jh8desquimj0f-image-registry",
														"objectEncoding": "utf-8"
													},
													"managedIdentitiesKeyVault": {
														"name": "ah-prow-mi-j7474816-1",
														"tenantID": "64dc69e4-d083-49fc-9569-ebece1dd1408"
													},
													"nodePoolManagement": {
														"credentialsSecretName": "uamsi-2pc2p4sbgf0l2k3pp45jh8desquimj0f-cluster-api-azure",
														"objectEncoding": "utf-8"
													}
												},
												"dataPlane": {
													"diskMSIClientID": "40fda6b0-96fa-404a-a284-32d0e855b1f4",
													"fileMSIClientID": "4b4f7d77-f077-4344-b033-c2a9c0b71cb7",
													"imageRegistryMSIClientID": "0a28602b-7ad8-4501-8603-41cf6272fe25"
												}
											}
										},
										"cloud": "AzurePublicCloud",
										"resourceGroup": "arm64-vm-cluster-n2ktdt--managed",
										"securityGroupID": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/Microsoft.Network/networkSecurityGroups/arm64-vm-customer-nsg",
										"subnetID": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/Microsoft.Network/virtualNetworks/arm64-vm-customer-vnet/subnets/arm64-vm-subnet1",
										"subscriptionID": "974ebd46-8ad3-41e3-afef-7ef25fd5c371",
										"vnetID": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/Microsoft.Network/virtualNetworks/arm64-vm-customer-vnet"
									}
								},
								"autoscaling": {
									"podPriorityThreshold": -10,
									"maxNodeProvisionTime": "15m",
									"maxNodesTotal": 0,
									"maxPodGracePeriod": 600,
									"scaling": "ScaleUpAndScaleDown"
								},
								"etcd": {
									"managed": {
										"storage": {
											"type": "PersistentVolume",
											"persistentVolume": {
												"size": "32G"
											}
										}
									},
									"managementType": "Managed"
								},
								"labels": {
									"kubernetes.azure.com/managedby": "sub_974ebd46-8ad3-41e3-afef-7ef25fd5c371"
								},
								"capabilities": {},
								"channel": "stable-4.20",
								"clusterID": "549a9064-0ab8-4696-8e2e-dfd2fc0983de",
								"configuration": {
									"apiServer": {
										"audit": {
											"profile": "Default"
										},
										"clientCA": {
											"name": ""
										},
										"encryption": {},
										"servingCerts": {
											"namedCertificates": [
												{
													"names": [
														"api.g9s1j1d6a6y5b9t.iuh0.j7474816.hcp.osadev.cloud"
													],
													"servingCertificate": {
														"name": "kube-apiserver-tls-cert"
													}
												}
											]
										}
									},
									"authentication": {
										"type": "OIDC",
										"oauthMetadata": {
											"name": ""
										},
										"serviceAccountIssuer": ""
									}
								},
								"controllerAvailabilityPolicy": "HighlyAvailable",
								"fips": false,
								"imageContentSources": [
									{
										"source": "quay.io/openshift-release-dev/ocp-v4.0-art-dev",
										"mirrors": [
											"arohcpocpdev.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev"
										]
									},
									{
										"source": "quay.io/openshift-release-dev/ocp-release",
										"mirrors": [
											"arohcpocpdev.azurecr.io/openshift-release-dev/ocp-release"
										]
									},
									{
										"source": "quay.io/openshift-release-dev/ocp-release-nightly",
										"mirrors": [
											"arohcpocpdev.azurecr.io/openshift-release-dev/ocp-release-nightly"
										]
									}
								],
								"infraID": "2pc2p4sbgf0l2k3pp45jh8desquimj0f",
								"infrastructureAvailabilityPolicy": "HighlyAvailable",
								"issuerURL": "https://arohcpprowoidcj7474816.z1.web.core.windows.net/64dc69e4-d083-49fc-9569-ebece1dd1408/2pc2p4sbgf0l2k3pp45jh8desquimj0f",
								"kubeAPIServerDNSName": "api.g9s1j1d6a6y5b9t.iuh0.j7474816.hcp.osadev.cloud",
								"networking": {
									"networkType": "OVNKubernetes",
									"clusterNetwork": [
										{
											"hostPrefix": 23,
											"cidr": "10.128.0.0/14"
										}
									],
									"machineNetwork": [
										{
											"cidr": "10.0.0.0/16"
										}
									],
									"serviceNetwork": [
										{
											"cidr": "172.30.0.0/16"
										}
									]
								},
								"olmCatalogPlacement": "management",
								"pullSecret": {
									"name": "g9s1j1d6a6y5b9t-pull"
								},
								"release": {
									"image": "quay.io/openshift-release-dev/ocp-release@sha256:3fac61ad81b3b209c7b2e617ad73f91f79c83692b1cb8196cdeb8567543c4b43"
								},
								"secretEncryption": {
									"type": "kms",
									"kms": {
										"azure": {
											"kms": {
												"credentialsSecretName": "uamsi-2pc2p4sbgf0l2k3pp45jh8desquimj0f-kms",
												"objectEncoding": "utf-8",
												"clientID": "e8723db7-9b9e-46a4-9f7d-64d75c3534f0"
											},
											"activeKey": {
												"keyName": "etcd-data-kms-encryption-key",
												"keyVaultName": "cust-kv-5aic5v443asos",
												"keyVersion": "7805077d973a4b42b8f3f9a9c86b0d8f"
											},
											"keyVaultAccess": "Public"
										},
										"provider": "Azure"
									}
								},
								"serviceAccountSigningKey": {
									"name": "bound-service-account-signing-key"
								},
								"services": [
									{
										"service": "APIServer",
										"servicePublishingStrategy": {
											"type": "Route",
											"route": {
												"hostname": "api-int.g9s1j1d6a6y5b9t.iuh0.j7474816.hcp.osadev.cloud"
											}
										}
									},
									{
										"service": "OAuthServer",
										"servicePublishingStrategy": {
											"type": "Route",
											"route": {
												"hostname": "oauth.g9s1j1d6a6y5b9t.iuh0.j7474816.hcp.osadev.cloud"
											}
										}
									},
									{
										"service": "Konnectivity",
										"servicePublishingStrategy": {
											"type": "Route",
											"route": {
												"hostname": "konnectivity-server.g9s1j1d6a6y5b9t.iuh0.j7474816.hcp.osadev.cloud"
											}
										}
									},
									{
										"service": "Ignition",
										"servicePublishingStrategy": {
											"type": "Route",
											"route": {
												"hostname": "ignition-server.g9s1j1d6a6y5b9t.iuh0.j7474816.hcp.osadev.cloud"
											}
										}
									}
								],
								"sshKey": {
									"name": ""
								}
							},
							"metadata": {
								"name": "g9s1j1d6a6y5b9t",
								"labels": {
									"api.openshift.com/environment": "arohcpprow",
									"api.openshift.com/id": "2pc2p4sbgf0l2k3pp45jh8desquimj0f",
									"api.openshift.com/name": "arm64-vm-hcp-cluster",
									"hypershift.openshift.io/hosted-cluster-size": "e2e_minimal"
								},
								"annotations": {
									"azure.microsoft.com/hcp-cluster-azure-resource-id": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/Microsoft.RedHatOpenshift/hcpOpenShiftClusters/arm64-vm-hcp-cluster",
									"cluster.open-cluster-management.io/hypershiftdeployment": "ignore/ignore",
									"cluster.open-cluster-management.io/managedcluster-name": "2pc2p4sbgf0l2k3pp45jh8desquimj0f",
									"hypershift.openshift.io/cleanup-cloud-resources": "true",
									"hypershift.openshift.io/cluster-scheduled": "true",
									"hypershift.openshift.io/kube-apiserver-max-mutating-requests-inflight": "99999",
									"hypershift.openshift.io/kube-apiserver-max-requests-inflight": "99999",
									"hypershift.openshift.io/kube-apiserver-service-account-token-max-expiration": "24h",
									"hypershift.openshift.io/olm-catalogs-is-registry-overrides": "registry.redhat.io/redhat=arohcpocpdev.azurecr.io/redhat",
									"hypershift.openshift.io/pod-security-admission-label-override": "baseline",
									"hypershift.openshift.io/skip-release-image-validation": "true",
									"resource-request-override.hypershift.openshift.io/cluster-policy-controller.cluster-policy-controller": "memory=100Mi,cpu=100m",
									"resource-request-override.hypershift.openshift.io/etcd.etcd": "memory=100Mi,cpu=100m",
									"resource-request-override.hypershift.openshift.io/kube-apiserver.kube-apiserver": "memory=100Mi,cpu=100m",
									"resource-request-override.hypershift.openshift.io/kube-controller-manager.kube-controller-manager": "memory=100Mi,cpu=100m",
									"resource-request-override.hypershift.openshift.io/openshift-apiserver.openshift-apiserver": "memory=100Mi,cpu=100m",
									"resource-request-override.hypershift.openshift.io/openshift-controller-manager.openshift-controller-manager": "memory=100Mi,cpu=100m",
									"resource-request-override.hypershift.openshift.io/ovnkube-control-plane.ovnkube-control-plane": "memory=100Mi,cpu=100m",
									"hypershift.openshift.io/cluster-size-override": "e2e_minimal"
								},
								"creationTimestamp": "2026-03-30T13:31:06.0000000Z",
								"finalizers": [
									"hypershift.openshift.io/finalizer"
								],
								"generation": 1,
								"managedFields": [
									{
										"time": "2026-03-30T13:31:06.0000000Z",
										"apiVersion": "hypershift.openshift.io/v1beta1",
										"fieldsType": "FieldsV1",
										"fieldsV1": {
											"f:metadata": {
												"f:labels": {
													"f:hypershift.openshift.io/hosted-cluster-size": {}
												}
											}
										},
										"manager": "hostedclustersizing",
										"operation": "Apply"
									},
									{
										"time": "2026-03-30T13:31:06.0000000Z",
										"apiVersion": "hypershift.openshift.io/v1beta1",
										"fieldsType": "FieldsV1",
										"fieldsV1": {
											"f:status": {
												"f:conditions": {
													"k:{\"type\":\"ClusterSizeComputed\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {}
													},
													"k:{\"type\":\"ClusterSizeTransitionPending\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {}
													},
													"k:{\"type\":\"ClusterSizeTransitionRequired\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {}
													}
												}
											}
										},
										"manager": "hostedclustersizing",
										"operation": "Apply",
										"subresource": "status"
									},
									{
										"time": "2026-03-30T13:31:06.0000000Z",
										"apiVersion": "hypershift.openshift.io/v1beta1",
										"fieldsType": "FieldsV1",
										"fieldsV1": {
											"f:metadata": {
												"f:labels": {
													"f:api.openshift.com/environment": {},
													"f:api.openshift.com/id": {},
													"f:api.openshift.com/name": {}
												},
												"f:annotations": {
													"f:azure.microsoft.com/hcp-cluster-azure-resource-id": {},
													"f:cluster.open-cluster-management.io/hypershiftdeployment": {},
													"f:cluster.open-cluster-management.io/managedcluster-name": {},
													"f:hypershift.openshift.io/cleanup-cloud-resources": {},
													"f:hypershift.openshift.io/kube-apiserver-service-account-token-max-expiration": {},
													"f:hypershift.openshift.io/olm-catalogs-is-registry-overrides": {},
													"f:hypershift.openshift.io/pod-security-admission-label-override": {},
													"f:hypershift.openshift.io/skip-release-image-validation": {},
													"f:hypershift.openshift.io/cluster-size-override": {}
												}
											},
											"f:spec": {
												"f:labels": {
													"f:kubernetes.azure.com/managedby": {}
												},
												"f:autoscaling": {
													"f:maxNodeProvisionTime": {},
													"f:maxNodesTotal": {},
													"f:maxPodGracePeriod": {},
													"f:podPriorityThreshold": {}
												},
												"f:capabilities": {},
												"f:channel": {},
												"f:clusterID": {},
												"f:configuration": {
													"f:apiServer": {
														"f:audit": {},
														"f:clientCA": {
															"f:name": {}
														},
														"f:encryption": {},
														"f:servingCerts": {
															"f:namedCertificates": {}
														}
													},
													"f:authentication": {
														"f:type": {},
														"f:oauthMetadata": {
															"f:name": {}
														},
														"f:serviceAccountIssuer": {}
													}
												},
												"f:controllerAvailabilityPolicy": {},
												"f:dns": {
													"f:baseDomain": {},
													"f:baseDomainPrefix": {},
													"f:publicZoneID": {}
												},
												"f:etcd": {
													"f:managed": {
														"f:storage": {
															"f:type": {},
															"f:persistentVolume": {
																"f:size": {}
															}
														}
													},
													"f:managementType": {}
												},
												"f:fips": {},
												"f:imageContentSources": {},
												"f:infraID": {},
												"f:infrastructureAvailabilityPolicy": {},
												"f:issuerURL": {},
												"f:kubeAPIServerDNSName": {},
												"f:networking": {
													"f:clusterNetwork": {},
													"f:machineNetwork": {},
													"f:networkType": {},
													"f:serviceNetwork": {}
												},
												"f:olmCatalogPlacement": {},
												"f:platform": {
													"f:type": {},
													"f:azure": {
														"f:tenantID": {},
														"f:azureAuthenticationConfig": {
															"f:azureAuthenticationConfigType": {},
															"f:managedIdentities": {
																"f:controlPlane": {
																	"f:cloudProvider": {
																		"f:credentialsSecretName": {},
																		"f:objectEncoding": {}
																	},
																	"f:controlPlaneOperator": {
																		"f:credentialsSecretName": {},
																		"f:objectEncoding": {}
																	},
																	"f:disk": {
																		"f:credentialsSecretName": {},
																		"f:objectEncoding": {}
																	},
																	"f:file": {
																		"f:credentialsSecretName": {},
																		"f:objectEncoding": {}
																	},
																	"f:imageRegistry": {
																		"f:credentialsSecretName": {},
																		"f:objectEncoding": {}
																	},
																	"f:ingress": {
																		"f:credentialsSecretName": {},
																		"f:objectEncoding": {}
																	},
																	"f:managedIdentitiesKeyVault": {
																		"f:name": {},
																		"f:tenantID": {}
																	},
																	"f:network": {
																		"f:credentialsSecretName": {},
																		"f:objectEncoding": {}
																	},
																	"f:nodePoolManagement": {
																		"f:credentialsSecretName": {},
																		"f:objectEncoding": {}
																	}
																},
																"f:dataPlane": {
																	"f:diskMSIClientID": {},
																	"f:fileMSIClientID": {},
																	"f:imageRegistryMSIClientID": {}
																}
															}
														},
														"f:cloud": {},
														"f:location": {},
														"f:resourceGroup": {},
														"f:securityGroupID": {},
														"f:subnetID": {},
														"f:subscriptionID": {},
														"f:vnetID": {}
													}
												},
												"f:pullSecret": {},
												"f:release": {
													"f:image": {}
												},
												"f:secretEncryption": {
													"f:type": {},
													"f:kms": {
														"f:azure": {
															"f:activeKey": {
																"f:keyName": {},
																"f:keyVaultName": {},
																"f:keyVersion": {}
															},
															"f:keyVaultAccess": {},
															"f:kms": {
																"f:credentialsSecretName": {},
																"f:objectEncoding": {},
																"f:clientID": {}
															}
														},
														"f:provider": {}
													}
												},
												"f:serviceAccountSigningKey": {},
												"f:services": {},
												"f:sshKey": {}
											}
										},
										"manager": "work-agent",
										"operation": "Apply"
									},
									{
										"time": "2026-03-30T13:31:06.0000000Z",
										"apiVersion": "hypershift.openshift.io/v1beta1",
										"fieldsType": "FieldsV1",
										"fieldsV1": {
											"f:metadata": {
												"f:annotations": {
													"f:hypershift.openshift.io/cluster-scheduled": {},
													"f:hypershift.openshift.io/kube-apiserver-max-mutating-requests-inflight": {},
													"f:hypershift.openshift.io/kube-apiserver-max-requests-inflight": {},
													"f:resource-request-override.hypershift.openshift.io/cluster-policy-controller.cluster-policy-controller": {},
													"f:resource-request-override.hypershift.openshift.io/etcd.etcd": {},
													"f:resource-request-override.hypershift.openshift.io/kube-apiserver.kube-apiserver": {},
													"f:resource-request-override.hypershift.openshift.io/kube-controller-manager.kube-controller-manager": {},
													"f:resource-request-override.hypershift.openshift.io/openshift-apiserver.openshift-apiserver": {},
													"f:resource-request-override.hypershift.openshift.io/openshift-controller-manager.openshift-controller-manager": {},
													"f:resource-request-override.hypershift.openshift.io/ovnkube-control-plane.ovnkube-control-plane": {}
												},
												"f:finalizers": {
													".": {},
													"v:\"hypershift.openshift.io/finalizer\"": {}
												}
											}
										},
										"manager": "hypershift-operator-manager",
										"operation": "Update"
									},
									{
										"time": "2026-03-30T13:31:07.0000000Z",
										"apiVersion": "hypershift.openshift.io/v1beta1",
										"fieldsType": "FieldsV1",
										"fieldsV1": {
											"f:metadata": {
												"f:ownerReferences": {
													".": {},
													"k:{\"uid\":\"69cd35d3-f539-4872-94ee-cb02b2ec35f3\"}": {}
												}
											}
										},
										"manager": "work-agent",
										"operation": "Update"
									},
									{
										"time": "2026-03-30T13:33:43.0000000Z",
										"apiVersion": "hypershift.openshift.io/v1beta1",
										"fieldsType": "FieldsV1",
										"fieldsV1": {
											"f:status": {
												"f:conditions": {
													"k:{\"type\":\"Available\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													},
													"k:{\"type\":\"ClusterVersionAvailable\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													},
													"k:{\"type\":\"ClusterVersionProgressing\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													},
													"k:{\"type\":\"ClusterVersionReleaseAccepted\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													},
													"k:{\"type\":\"ClusterVersionSucceeding\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													},
													"k:{\"type\":\"ClusterVersionUpgradeable\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													},
													"k:{\"type\":\"ControlPlaneConnectionAvailable\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													},
													"k:{\"type\":\"DataPlaneConnectionAvailable\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													},
													"k:{\"type\":\"Degraded\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													},
													"k:{\"type\":\"EtcdAvailable\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													},
													"k:{\"type\":\"ExternalDNSReachable\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													},
													"k:{\"type\":\"HostedClusterRestoredFromBackup\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													},
													"k:{\"type\":\"IgnitionEndpointAvailable\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													},
													"k:{\"type\":\"InfrastructureReady\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													},
													"k:{\"type\":\"KubeAPIServerAvailable\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													},
													"k:{\"type\":\"PlatformCredentialsFound\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													},
													"k:{\"type\":\"Progressing\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													},
													"k:{\"type\":\"ReconciliationActive\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													},
													"k:{\"type\":\"ReconciliationSucceeded\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													},
													"k:{\"type\":\"SupportedHostedCluster\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													},
													"k:{\"type\":\"ValidAzureKMSConfig\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													},
													"k:{\"type\":\"ValidConfiguration\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													},
													"k:{\"type\":\"ValidHostedControlPlaneConfiguration\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													},
													"k:{\"type\":\"ValidIDPConfiguration\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													},
													"k:{\"type\":\"ValidProxyConfiguration\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													},
													"k:{\"type\":\"ValidReleaseImage\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													},
													"k:{\"type\":\"ValidReleaseInfo\"}": {
														".": {},
														"f:lastTransitionTime": {},
														"f:message": {},
														"f:reason": {},
														"f:status": {},
														"f:type": {},
														"f:observedGeneration": {}
													}
												},
												"f:controlPlaneEndpoint": {
													".": {},
													"f:host": {},
													"f:port": {}
												},
												"f:customKubeconfig": {},
												"f:ignitionEndpoint": {},
												"f:kubeconfig": {},
												"f:payloadArch": {},
												"f:version": {
													".": {},
													"f:observedGeneration": {},
													"f:availableUpdates": {},
													"f:desired": {
														".": {},
														"f:image": {},
														"f:version": {}
													},
													"f:history": {}
												}
											}
										},
										"manager": "hypershift-operator-manager",
										"operation": "Update",
										"subresource": "status"
									}
								],
								"namespace": "ocm-arohcpprow-2pc2p4sbgf0l2k3pp45jh8desquimj0f",
								"uid": "4cbad968-cdd9-499b-a2e2-bdc0d4358412",
								"ownerReferences": [
									{
										"name": "1710ce7a0d4dfc0db490b6b6747644ffbb86e00748dd9562d4bee835e077b083-2pc2p4sbgf0l2k3pp45jh8desquimj0f",
										"apiVersion": "work.open-cluster-management.io/v1",
										"uid": "69cd35d3-f539-4872-94ee-cb02b2ec35f3",
										"kind": "AppliedManifestWork"
									}
								],
								"resourceVersion": "31429"
							},
							"apiVersion": "hypershift.openshift.io/v1beta1",
							"kind": "HostedCluster"
						}
					]
				}
			},
			"resourceId": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/managementClusterContents/readonlyHypershiftHostedCluster",
			"cosmosMetadata": {
				"resourceID": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/managementClusterContents/readonlyHypershiftHostedCluster",
				"etag": "\"2600e082-0000-4d00-0000-69ca7bb80000\""
			}
		}
	}
},
"cluster": prow-j7474816-svc,
"namespace_name": aro-hcp,
"container_name": aro-hcp-backend
nodepool maestro readonly bundle 
"timestamp": 2026-03-30T13:44:39.644Z,
"log": {
	"time": "2026-03-30T13:44:39.6447811Z",
	"level": "INFO",
	"source": {
		"function": "github.com/Azure/ARO-HCP/internal/serverutils.DumpDataToLogger-range1",
		"file": "/app/internal/serverutils/dump_data.go",
		"line": 53
	},
	"msg": "dumping resourceID /subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/nodePools/arm64-vm-np-1/managementClusterContents/readonlyHypershiftNodePool",
	"controller_name": "datadump",
	"subscription_id": "974ebd46-8ad3-41e3-afef-7ef25fd5c371",
	"resource_group": "arm64-vm-cluster-n2ktdt",
	"resourceType": "microsoft.redhatopenshift/hcpopenshiftclusters",
	"resource_name": "arm64-vm-hcp-cluster",
	"resource_id": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourcegroups/arm64-vm-cluster-n2ktdt/providers/microsoft.redhatopenshift/hcpopenshiftclusters/arm64-vm-hcp-cluster",
	"hcp_cluster_name": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourcegroups/arm64-vm-cluster-n2ktdt/providers/microsoft.redhatopenshift/hcpopenshiftclusters/arm64-vm-hcp-cluster",
	"currentResourceID": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/nodePools/arm64-vm-np-1/managementClusterContents/readonlyHypershiftNodePool",
	"content": {
		"resourceType": "Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools/managementClusterContents",
		"id": "094a1329-92dd-5cdd-b3cb-2a302d4ec732",
		"_rid": "425DAJSnY-BOAgAAAAAAAA==",
		"_self": "dbs/425DAA==/colls/425DAJSnY-A=/docs/425DAJSnY-BOAgAAAAAAAA==/",
		"_etag": "\"26003f87-0000-4d00-0000-69ca7d850000\"",
		"_attachments": "attachments/",
		"_ts": 1774878085,
		"partitionKey": "974ebd46-8ad3-41e3-afef-7ef25fd5c371",
		"resourceID": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/nodePools/arm64-vm-np-1/managementClusterContents/readonlyHypershiftNodePool",
		"properties": {
			"status": {
				"conditions": [
					{
						"type": "Degraded",
						"status": "False",
						"lastTransitionTime": "2026-03-30T13:41:25.5576824Z",
						"reason": "NoErrors",
						"message": "As expected."
					}
				],
				"kubeContent": {
					"metadata": {},
					"items": [
						{
							"status": {
								"conditions": [
									{
										"type": "AutoscalingEnabled",
										"status": "False",
										"lastTransitionTime": "2026-03-30T13:35:43.0000000Z",
										"reason": "AsExpected",
										"observedGeneration": 1
									},
									{
										"type": "UpdateManagementEnabled",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:35:43.0000000Z",
										"reason": "AsExpected",
										"observedGeneration": 1
									},
									{
										"type": "ValidReleaseImage",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:35:44.0000000Z",
										"reason": "AsExpected",
										"message": "Using release image: quay.io/openshift-release-dev/ocp-release@sha256:7574c1d0774a2c76967cd71f77e9bb72dd9b0b30a49a1451d02f2bcc2ff6cce7",
										"observedGeneration": 1
									},
									{
										"type": "ValidArchPlatform",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:35:44.0000000Z",
										"reason": "AsExpected",
										"observedGeneration": 1
									},
									{
										"type": "ReconciliationActive",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:35:44.0000000Z",
										"reason": "ReconciliationActive",
										"message": "Reconciliation active on resource",
										"observedGeneration": 1
									},
									{
										"type": "SupportedVersionSkew",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:35:44.0000000Z",
										"reason": "AsExpected",
										"message": "Release image version is valid",
										"observedGeneration": 1
									},
									{
										"type": "ValidMachineConfig",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:35:44.0000000Z",
										"reason": "AsExpected",
										"observedGeneration": 1
									},
									{
										"type": "UpdatingConfig",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:35:44.0000000Z",
										"reason": "AsExpected",
										"message": "Updating config in progress. Target config: 5811ab4a",
										"observedGeneration": 1
									},
									{
										"type": "UpdatingVersion",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:35:44.0000000Z",
										"reason": "AsExpected",
										"message": "Updating version in progress. Target version: 4.20.15",
										"observedGeneration": 1
									},
									{
										"type": "ValidGeneratedPayload",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:36:39.0000000Z",
										"reason": "AsExpected",
										"message": "Payload generated successfully",
										"observedGeneration": 1
									},
									{
										"type": "ReachedIgnitionEndpoint",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:36:43.0000000Z",
										"reason": "AsExpected",
										"observedGeneration": 1
									},
									{
										"type": "AllMachinesReady",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:36:53.0000000Z",
										"reason": "AsExpected",
										"message": "All is well",
										"observedGeneration": 1
									},
									{
										"type": "AllNodesHealthy",
										"status": "False",
										"lastTransitionTime": "2026-03-30T13:35:44.0000000Z",
										"reason": "NodeProvisioning",
										"message": "Machine g9s1j1d6a6y5b9t-arm64-vm-np-1-z4mdr-pxfp2: NodeProvisioning\nMachine g9s1j1d6a6y5b9t-arm64-vm-np-1-z4mdr-r8pfz: NodeProvisioning\n",
										"observedGeneration": 1
									},
									{
										"type": "ValidPlatformConfig",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:35:44.0000000Z",
										"reason": "AsExpected",
										"message": "All is well",
										"observedGeneration": 1
									},
									{
										"type": "ValidTuningConfig",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:35:44.0000000Z",
										"reason": "AsExpected",
										"observedGeneration": 1
									},
									{
										"type": "UpdatingPlatformMachineTemplate",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:35:44.0000000Z",
										"reason": "AsExpected",
										"message": "platform machine template update in progress. Target template: g9s1j1d6a6y5b9t-arm64-vm-np-1-e5c02a96",
										"observedGeneration": 1
									},
									{
										"type": "Ready",
										"status": "False",
										"lastTransitionTime": "2026-03-30T13:35:44.0000000Z",
										"reason": "WaitingForAvailableMachines",
										"message": "Minimum availability requires 2 replicas, current 0 available",
										"observedGeneration": 1
									},
									{
										"type": "AutorepairEnabled",
										"status": "True",
										"lastTransitionTime": "2026-03-30T13:36:43.0000000Z",
										"reason": "AsExpected",
										"observedGeneration": 1
									}
								]
							},
							"spec": {
								"platform": {
									"type": "Azure",
									"azure": {
										"image": {
											"type": "AzureMarketplace",
											"azureMarketplace": {
												"version": "9.6.20251015",
												"imageGeneration": "Gen2",
												"offer": "aro4",
												"publisher": "azureopenshift",
												"sku": "420-arm"
											}
										},
										"subnetID": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/Microsoft.Network/virtualNetworks/arm64-vm-customer-vnet/subnets/arm64-vm-subnet1",
										"vmSize": "Standard_D2pls_v6",
										"osDisk": {
											"sizeGiB": 64,
											"diskStorageAccountType": "StandardSSD_LRS",
											"persistence": "Persistent"
										},
										"diagnostics": {
											"storageAccountType": "Managed"
										},
										"encryptionAtHost": "Disabled"
									}
								},
								"release": {
									"image": "quay.io/openshift-release-dev/ocp-release@sha256:7574c1d0774a2c76967cd71f77e9bb72dd9b0b30a49a1451d02f2bcc2ff6cce7"
								},
								"replicas": 2,
								"arch": "arm64",
								"clusterName": "g9s1j1d6a6y5b9t",
								"management": {
									"autoRepair": true,
									"replace": {
										"rollingUpdate": {
											"maxSurge": 1,
											"maxUnavailable": 0
										},
										"strategy": "RollingUpdate"
									},
									"upgradeType": "Replace"
								}
							},
							"metadata": {
								"name": "g9s1j1d6a6y5b9t-arm64-vm-np-1",
								"labels": {
									"api.openshift.com/environment": "arohcpprow",
									"api.openshift.com/id": "2pc2p4sbgf0l2k3pp45jh8desquimj0f",
									"api.openshift.com/name": "arm64-vm-hcp-cluster"
								},
								"creationTimestamp": "2026-03-30T13:35:43.0000000Z",
								"finalizers": [
									"hypershift.openshift.io/finalizer"
								],
								"generation": 1,
								"managedFields": [
									{
										"time": "2026-03-30T13:35:43.0000000Z",
										"apiVersion": "hypershift.openshift.io/v1beta1",
										"fieldsType": "FieldsV1",
										"fieldsV1": {
											"f:metadata": {
												"f:labels": {
													"f:api.openshift.com/environment": {},
													"f:api.openshift.com/id": {},
													"f:api.openshift.com/name": {}
												}
											},
											"f:spec": {
												"f:platform": {
													"f:type": {},
													"f:azure": {
														"f:subnetID": {},
														"f:image": {
															"f:type": {},
															"f:azureMarketplace": {
																"f:version": {},
																"f:offer": {},
																"f:publisher": {},
																"f:sku": {}
															}
														},
														"f:diagnostics": {
															"f:storageAccountType": {}
														},
														"f:encryptionAtHost": {},
														"f:osDisk": {
															"f:diskStorageAccountType": {},
															"f:persistence": {},
															"f:sizeGiB": {}
														},
														"f:vmSize": {}
													}
												},
												"f:release": {
													"f:image": {}
												},
												"f:clusterName": {},
												"f:management": {
													"f:autoRepair": {},
													"f:replace": {
														"f:rollingUpdate": {
															"f:maxSurge": {},
															"f:maxUnavailable": {}
														},
														"f:strategy": {}
													},
													"f:upgradeType": {}
												},
												"f:replicas": {},
												"f:arch": {}
											}
										},
										"manager": "work-agent",
										"operation": "Apply"
									},
									{
										"time": "2026-03-30T13:35:43.0000000Z",
										"apiVersion": "hypershift.openshift.io/v1beta1",
										"fieldsType": "FieldsV1",
										"fieldsV1": {
											"f:metadata": {
												"f:ownerReferences": {
													".": {},
													"k:{\"uid\":\"5ccc687f-b3c3-4100-b9de-12c57b9afc75\"}": {}
												}
											}
										},
										"manager": "work-agent",
										"operation": "Update"
									},
									{
										"time": "2026-03-30T13:35:44.0000000Z",
										"apiVersion": "hypershift.openshift.io/v1beta1",
										"fieldsType": "FieldsV1",
										"fieldsV1": {
											"f:metadata": {
												"f:ownerReferences": {
													"k:{\"uid\":\"4cbad968-cdd9-499b-a2e2-bdc0d4358412\"}": {}
												},
												"f:finalizers": {
													".": {},
													"v:\"hypershift.openshift.io/finalizer\"": {}
												}
											}
										},
										"manager": "hypershift-operator-manager",
										"operation": "Update"
									},
									{
										"time": "2026-03-30T13:36:53.0000000Z",
										"apiVersion": "hypershift.openshift.io/v1beta1",
										"fieldsType": "FieldsV1",
										"fieldsV1": {
											"f:status": {
												".": {},
												"f:conditions": {}
											}
										},
										"manager": "hypershift-operator-manager",
										"operation": "Update",
										"subresource": "status"
									}
								],
								"namespace": "ocm-arohcpprow-2pc2p4sbgf0l2k3pp45jh8desquimj0f",
								"uid": "4af73d31-307a-40cf-9894-7a10e6e9d64e",
								"ownerReferences": [
									{
										"name": "1710ce7a0d4dfc0db490b6b6747644ffbb86e00748dd9562d4bee835e077b083-2pc2p4sbgf0l2k3pp45jh8desquimj0f-arm64-vm-np-1",
										"apiVersion": "work.open-cluster-management.io/v1",
										"uid": "5ccc687f-b3c3-4100-b9de-12c57b9afc75",
										"kind": "AppliedManifestWork"
									},
									{
										"name": "g9s1j1d6a6y5b9t",
										"apiVersion": "hypershift.openshift.io/v1beta1",
										"uid": "4cbad968-cdd9-499b-a2e2-bdc0d4358412",
										"kind": "HostedCluster"
									}
								],
								"resourceVersion": "58045"
							},
							"apiVersion": "hypershift.openshift.io/v1beta1",
							"kind": "NodePool"
						}
					]
				}
			},
			"resourceId": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/nodePools/arm64-vm-np-1/managementClusterContents/readonlyHypershiftNodePool",
			"cosmosMetadata": {
				"resourceID": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster/nodePools/arm64-vm-np-1/managementClusterContents/readonlyHypershiftNodePool"
			}
		}
	}
},
"cluster": prow-j7474816-svc,
"namespace_name": aro-hcp,
"container_name": aro-hcp-backend
serviceprovidercluster 
"timestamp": 2026-03-30T13:32:16.107Z,
"log": {
	"time": "2026-03-30T13:32:16.1076363Z",
	"level": "INFO",
	"source": {
		"function": "github.com/Azure/ARO-HCP/internal/serverutils.DumpDataToLogger-range1",
		"file": "/app/internal/serverutils/dump_data.go",
		"line": 53
	},
	"msg": "dumping resourceID /subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/microsoft.redhatopenshift/hcpopenshiftclusters/arm64-vm-hcp-cluster/serviceProviderClusters/default",
	"controller_name": "datadump",
	"subscription_id": "974ebd46-8ad3-41e3-afef-7ef25fd5c371",
	"resource_group": "arm64-vm-cluster-n2ktdt",
	"resourceType": "microsoft.redhatopenshift/hcpopenshiftclusters",
	"resource_name": "arm64-vm-hcp-cluster",
	"resource_id": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourcegroups/arm64-vm-cluster-n2ktdt/providers/microsoft.redhatopenshift/hcpopenshiftclusters/arm64-vm-hcp-cluster",
	"hcp_cluster_name": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourcegroups/arm64-vm-cluster-n2ktdt/providers/microsoft.redhatopenshift/hcpopenshiftclusters/arm64-vm-hcp-cluster",
	"currentResourceID": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/microsoft.redhatopenshift/hcpopenshiftclusters/arm64-vm-hcp-cluster/serviceProviderClusters/default",
	"content": {
		"resourceType": "microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters",
		"id": "19197cdb-fe62-5550-9604-2738757d21ae",
		"_rid": "425DAJSnY-BSAAAAAAAAAA==",
		"_self": "dbs/425DAA==/colls/425DAJSnY-A=/docs/425DAJSnY-BSAAAAAAAAAA==/",
		"_etag": "\"2600db81-0000-4d00-0000-69ca7ae80000\"",
		"_attachments": "attachments/",
		"_ts": 1774877416,
		"partitionKey": "974ebd46-8ad3-41e3-afef-7ef25fd5c371",
		"resourceID": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/microsoft.redhatopenshift/hcpopenshiftclusters/arm64-vm-hcp-cluster/serviceProviderClusters/default",
		"properties": {
			"status": {
				"control_plane_version": {},
				"validations": [
					{
						"type": "AlwaysSuccessValidation",
						"status": "True",
						"lastTransitionTime": "2026-03-30T13:30:07.9419642Z",
						"reason": "Succeeded",
						"message": "Validation succeeded"
					},
					{
						"type": "AzureClusterResourceGroupExistenceValidation",
						"status": "True",
						"lastTransitionTime": "2026-03-30T13:30:08.6895565Z",
						"reason": "Succeeded",
						"message": "Validation succeeded"
					},
					{
						"type": "AzureClusterManagedIdentitiesExistenceValidation",
						"status": "True",
						"lastTransitionTime": "2026-03-30T13:30:12.4020974Z",
						"reason": "Succeeded",
						"message": "Validation succeeded"
					},
					{
						"type": "AzureResourceProvidersRegistrationValidation",
						"status": "True",
						"lastTransitionTime": "2026-03-30T13:30:16.1079163Z",
						"reason": "Succeeded",
						"message": "Validation succeeded"
					}
				],
				"maestroReadonlyBundles": [
					{
						"name": "readonlyHypershiftHostedCluster",
						"maestroAPIMaestroBundleName": "b38a5d35-b9aa-4b39-8d3b-6eb3e799cc8e",
						"maestroAPIMaestroBundleID": "a39d3886-19ac-5f8f-b836-aba08e471b7a"
					}
				]
			},
			"resourceId": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/microsoft.redhatopenshift/hcpopenshiftclusters/arm64-vm-hcp-cluster/serviceProviderClusters/default",
			"cosmosMetadata": {
				"resourceID": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/microsoft.redhatopenshift/hcpopenshiftclusters/arm64-vm-hcp-cluster/serviceProviderClusters/default",
				"etag": "\"2600b381-0000-4d00-0000-69ca7ae40000\""
			},
			"spec": {
				"control_plane_version": {
					"desired_version": "4.20.16"
				}
			}
		}
	}
},
"cluster": prow-j7474816-svc,
"namespace_name": aro-hcp,
"container_name": aro-hcp-backend
serviceprovidernodepool 
"timestamp": 2026-03-30T13:37:08.203Z,
"log": {
	"time": "2026-03-30T13:37:08.2036460Z",
	"level": "INFO",
	"source": {
		"function": "github.com/Azure/ARO-HCP/internal/serverutils.DumpDataToLogger-range1",
		"file": "/app/internal/serverutils/dump_data.go",
		"line": 53
	},
	"msg": "dumping resourceID /subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/microsoft.redhatopenshift/hcpopenshiftclusters/arm64-vm-hcp-cluster/nodepools/arm64-vm-np-1/serviceProviderNodePools/default",
	"controller_name": "datadump",
	"subscription_id": "974ebd46-8ad3-41e3-afef-7ef25fd5c371",
	"resource_group": "arm64-vm-cluster-n2ktdt",
	"resourceType": "microsoft.redhatopenshift/hcpopenshiftclusters",
	"resource_name": "arm64-vm-hcp-cluster",
	"resource_id": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourcegroups/arm64-vm-cluster-n2ktdt/providers/microsoft.redhatopenshift/hcpopenshiftclusters/arm64-vm-hcp-cluster",
	"hcp_cluster_name": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourcegroups/arm64-vm-cluster-n2ktdt/providers/microsoft.redhatopenshift/hcpopenshiftclusters/arm64-vm-hcp-cluster",
	"currentResourceID": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/microsoft.redhatopenshift/hcpopenshiftclusters/arm64-vm-hcp-cluster/nodepools/arm64-vm-np-1/serviceProviderNodePools/default",
	"content": {
		"resourceType": "microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools",
		"id": "2d8c1336-ea4e-52ae-b5f8-1d7273c428e4",
		"_rid": "425DAJSnY-CuAQAAAAAAAA==",
		"_self": "dbs/425DAA==/colls/425DAJSnY-A=/docs/425DAJSnY-CuAQAAAAAAAA==/",
		"_etag": "\"26008483-0000-4d00-0000-69ca7c460000\"",
		"_attachments": "attachments/",
		"_ts": 1774877766,
		"partitionKey": "974ebd46-8ad3-41e3-afef-7ef25fd5c371",
		"resourceID": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/microsoft.redhatopenshift/hcpopenshiftclusters/arm64-vm-hcp-cluster/nodepools/arm64-vm-np-1/serviceProviderNodePools/default",
		"properties": {
			"status": {
				"maestroReadonlyBundles": [
					{
						"name": "readonlyHypershiftNodePool",
						"maestroAPIMaestroBundleName": "cff3c64f-cf0e-4eca-bb1e-3b8f322bf935",
						"maestroAPIMaestroBundleID": "d0526415-6691-5039-a153-53e4920686a0"
					}
				],
				"nodePoolVersion": {
					"activeVersions": [
						{
							"version": "4.20.15"
						}
					]
				}
			},
			"resourceId": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/microsoft.redhatopenshift/hcpopenshiftclusters/arm64-vm-hcp-cluster/nodepools/arm64-vm-np-1/serviceProviderNodePools/default",
			"cosmosMetadata": {
				"resourceID": "/subscriptions/974ebd46-8ad3-41e3-afef-7ef25fd5c371/resourceGroups/arm64-vm-cluster-n2ktdt/providers/microsoft.redhatopenshift/hcpopenshiftclusters/arm64-vm-hcp-cluster/nodepools/arm64-vm-np-1/serviceProviderNodePools/default",
				"etag": "\"26008383-0000-4d00-0000-69ca7c460000\""
			},
			"spec": {
				"nodePoolVersion": {
					"desiredVersion": "4.20.15"
				}
			}
		}
	}
},
"cluster": prow-j7474816-svc,
"namespace_name": aro-hcp,
"container_name": aro-hcp-backend
maestro readonly orphan deletion logic referencing the maestro bundle IDs shown above 
"timestamp": 2026-03-30T13:57:21.992Z,
"log": {
	"time": "2026-03-30T13:57:21.9923169Z",
	"level": "INFO",
	"source": {
		"function": "github.com/Azure/ARO-HCP/backend/pkg/controllers.deleteOrphanedBundles[...].func1",
		"file": "/app/backend/pkg/controllers/delete_orphaned_maestro_readonly_bundles_controller.go",
		"line": 290
	},
	"msg": "Deleting orphaned Maestro readonly Bundle",
	"controller_name": "deleteorphanedmaestroreadonlybundles",
	"maestroBundleMetadataName": "b38a5d35-b9aa-4b39-8d3b-6eb3e799cc8e",
	"maestroConsumerName": "hcp-underlay-j7474816-mgmt-1",
	"maestroBundleID": "a39d3886-19ac-5f8f-b836-aba08e471b7a"
},
"cluster": prow-j7474816-svc,
"namespace_name": aro-hcp,
"container_name": aro-hcp-backend

...

"timestamp": 2026-03-30T13:57:22.582Z,
"log": {
	"time": "2026-03-30T13:57:22.5821791Z",
	"level": "INFO",
	"source": {
		"function": "github.com/Azure/ARO-HCP/backend/pkg/controllers.deleteOrphanedBundles[...].func1",
		"file": "/app/backend/pkg/controllers/delete_orphaned_maestro_readonly_bundles_controller.go",
		"line": 290
	},
	"msg": "Deleting orphaned Maestro readonly Bundle",
	"controller_name": "deleteorphanedmaestroreadonlybundles",
	"maestroBundleMetadataName": "cff3c64f-cf0e-4eca-bb1e-3b8f322bf935",
	"maestroConsumerName": "hcp-underlay-j7474816-mgmt-1",
	"maestroBundleID": "d0526415-6691-5039-a153-53e4920686a0"
},
"cluster": prow-j7474816-svc,
"namespace_name": aro-hcp,
"container_name": aro-hcp-backend
Details

miguelsorianod
Miguel Soriano (miguelsorianod) commented Mar 30, 2026
View reviewed changes
Comment thread internal/database/database.go
Merged
@miguelsorianod Miguel Soriano (miguelsorianod) changed the title [WIP] feat: add nodepool scoped maestro readonly bundles controllers feat: add nodepool scoped maestro readonly bundles controllers Mar 31, 2026
@miguelsorianod
Miguel Soriano (miguelsorianod) force-pushed the msoriano-add-maestro-readonlynodepools-controllers branch from 2cb3525 to c5b9d6c Compare March 31, 2026 14:20
ahitacat
Alba Hita (ahitacat) reviewed Mar 31, 2026
View reviewed changes
Comment thread .../controllers/read_and_persist_nodepool_scoped_maestro_readonly_bundles_content_controller.go Outdated
Comment thread .../controllers/read_and_persist_nodepool_scoped_maestro_readonly_bundles_content_controller.go Outdated
Comment thread .../controllers/read_and_persist_nodepool_scoped_maestro_readonly_bundles_content_controller.go
Comment thread backend/pkg/maestro/maestro_bundle.go Outdated
@miguelsorianod
Miguel Soriano (miguelsorianod) force-pushed the msoriano-add-maestro-readonlynodepools-controllers branch from c5b9d6c to 59d7cb2 Compare March 31, 2026 17:02
deads2k
David Eads (deads2k) reviewed Mar 31, 2026
View reviewed changes
Comment thread backend/pkg/maestro/maestro_api_maestro_bundle_name_generator.go
@deads2k

David Eads (deads2k) commented Mar 31, 2026

Copy link
Copy Markdown
Collaborator

type and database as expected. Some structural comments on exposure: remember less exposed is better, fewer points of configurability or injection is better. We may not be able to rely on the condition (OCP bug fixe needs backporting), but we should adjust #4695 to ensure we get something in the hostedcluster prior to merging this so we have e2e assurance it continues to function.

@miguelsorianod

Miguel Soriano (miguelsorianod) commented Apr 7, 2026

Copy link
Copy Markdown
Collaborator Author

Two new changes have been performed:

  • We identified that the ManagementClusterContent conditions were always being updated in Cosmos on each reconcile because the calculation of the ManagementClusterContent always sets the LastTransitionTime with the current time. I added a commit that ensures that if the condition exists on Cosmos beforehand, the LastTransitionTime is preserved if there are no changes compared to the new calculation.
  • With the merging of fire controllers more quickly when content has changed #4485, the cluster and nodepool watching controllers reuse a new type genericWatchingController. I refactored this MR to ensure that's leveraged. I added the managementclustercontent informer to enqueue events of that type for both cluster watching and nodepoolwatching controller. However, with the refactor it doesn't seem possible to do further filtering of received events anymore, which means that now the controllers receive managementclustercontent events but they cannot filter further to only process cluster-scoped or nodepool-scoped ones. We need to decide if there are more changes needed around this

@miguelsorianod
Miguel Soriano (miguelsorianod) force-pushed the msoriano-add-maestro-readonlynodepools-controllers branch from 87d20be to 3e1cf83 Compare April 7, 2026 09:14
@miguelsorianod
Miguel Soriano (miguelsorianod) force-pushed the msoriano-add-maestro-readonlynodepools-controllers branch 2 times, most recently from b7059af to 177cea9 Compare April 7, 2026 15:28
@deads2k

David Eads (deads2k) commented Apr 7, 2026

Copy link
Copy Markdown
Collaborator

/approve

leaving lgtm to reviewer of your choice.

@miguelsorianod

Miguel Soriano (miguelsorianod) commented Apr 17, 2026

Copy link
Copy Markdown
Collaborator Author

I also added another commit that deals with the same scenario of CSID not being set for both the create and read maestro readonly bundles.
For the cluster-scoped maestro controllers, that is being added in #4752

Merged
@miguelsorianod
feat: create nodepool scoped maestro readonly bundles controllers
c1f3b39
@miguelsorianod
refactor: move GetOrCreateMaestroBundle into maestro package
25ce6b8
@miguelsorianod
refactor: move maestro api maestro bundle name generator into maestro…
ac05a2a 
… package
@miguelsorianod
refactor: move management cluster content utils into controllerutils …
3848c85 
…package
@miguelsorianod
refactor: move createMaestroClientFromCSProvisionShard into package l…
93a4ebd 
…evel function
@miguelsorianod
feat: add ListAll method that iterates over all internal api types us…
4ab9ade 
…ing a cosmos lister
@miguelsorianod
feat: add orphan cleanup for nodepools and reuse parts of maestro rea…
67ae777 
…donly bundles logic

Extract shared readonly-bundle helpers (build bundle, references, kube
content) into maestro_readonly_bundle_helpers.go and use them from cluster-
and nodepool-scoped create/persist controllers.

Refactor DeleteOrphanedMaestroReadonlyBundles:
- List SPC/SPNP with database.ListAll
- Add maestro.ForEachMaestroBundle for paginated List and unit tests.
- Reuse code in some places
- Update/add controller tests for the new paths.
@miguelsorianod
fix: correctly calculate cs cluster href from nodepool href in maestr…
5a97c2a 
…o np controllers
@miguelsorianod
fix: maestro client leaks on read and persist maestro readonly bundle…
5660595 
…s controllers
@miguelsorianod
Miguel Soriano (miguelsorianod) force-pushed the msoriano-add-maestro-readonlynodepools-controllers branch from 4df1e4d to 43e464e Compare April 20, 2026 07:55
@miguelsorianod
feat: support cluster and nodepool scoped ManagementClusterContent co…
4cf4a03 
…smos resources

Introduce distinct Cosmos resource types for managementClusterContents nested
under an hcpOpenShiftCluster vs under a nodePool, and wire them through
Cosmos CRUD, global listing (union query), and databasetesting mocks.

We remove DBClient.ManagementClusterContents and we instead now provide its
access through HCPClusters(...).ManagementClusterContents(...) or
NodePools(...).ManagementClusterContents(...) instead.

We now have a Cosmos global lister that retrieves all ManagementClusterContents
at both cluster scope and nodepool scope. The same occurs for the corresponding
informer. The cluster watching controller uses the ManagementclusterContent informer
but it only ends up enqueuing cluster-scoped ManagementClusterContent documents.

Now that NodePools have their own ManagementClusterContent subresource we update
the maestro bundle internal name for node pools to have a fixed name
of `readonlyHypershiftNodePool` instead of a prefix, as now there are no potential
naming collisions anymore between nodepools of the same cluster or cross cluster.
@miguelsorianod
refactor: make maestro bundle name generator fully unexported in maes…
6ba9ddd 
…tro controllers
@miguelsorianod
refactor: move maestro readonly bundles resource specific tests to pa…
d87619e 
…ckage level ones
@miguelsorianod
fix: preserve lastTransitionTime in ManagementClusterContents conditions
e3949cd 
The existing ManagementClusterContent in Cosmos might include conditions that
already exist beforehand and that have been calculated in the new desired
content. This commit ensures that LastTransitionTime of those conditions in the case
where the status of them hasn't changed is preserved.
@miguelsorianod
feat: scope ManagementClusterContent informer events to immediate parent
4ec8a5a
@miguelsorianod
feat: update maestro delete orphan bundle to skip cosmos resources th…
c158261 
…at don't have CS ID

We do not want to make the whole sync fail when there are some cosmos
resources that do not have the ClusterServiceID attribute set yet. This
could be a common occurrence when there's enough amount of creation
activity of resources.

It should be safe to skip those because if a maestro bundle in the
maestro API exists it means that there should be a corresponding Cosmos resource
with a maestro bundle reference. If for some reason during the orphan calculation inbetween the first read of cosmos
resources and the read in the Maestro api there's a new bundle in maestro, the
second read of cosmos resources will catch that and prevent accidental deletion.
@miguelsorianod
feat: skip processing when no CS ID is set for create and read nodepo…
f6ed2d9 
…ol readonly maestro bundles

When we move CS creation of resources to the backend the CS ID might not
be set yet once the controllers run so we skip their processing.
@miguelsorianod
Miguel Soriano (miguelsorianod) force-pushed the msoriano-add-maestro-readonlynodepools-controllers branch from 43e464e to f6ed2d9 Compare April 20, 2026 08:07
@ahitacat

Alba Hita (ahitacat) commented Apr 20, 2026

Copy link
Copy Markdown
Collaborator

/lgtm

@openshift-ci openshift-ci Bot added the lgtm label Apr 20, 2026
@openshift-ci

openshift-ci Bot commented Apr 20, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahitacat, deads2k, miguelsorianod

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-bot

openshift-merge-bot Bot commented Apr 20, 2026

Copy link
Copy Markdown
Contributor

/retest-required

Remaining retests: 0 against base HEAD db12f74 and 2 for PR HEAD f6ed2d9 in total

@openshift-ci

openshift-ci Bot commented Apr 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Miguel Soriano (@miguelsorianod): The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/cspr 58d409d link true /test cspr

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@miguelsorianod

Miguel Soriano (miguelsorianod) commented Apr 20, 2026

Copy link
Copy Markdown
Collaborator Author

/retest

@openshift-merge-bot
openshift-merge-bot Bot merged commit 490c96e into main Apr 20, 2026
21 checks passed
@openshift-merge-bot
openshift-merge-bot Bot deleted the msoriano-add-maestro-readonlynodepools-controllers branch April 20, 2026 19:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@deads2k David Eads (deads2k) deads2k left review comments
@geoberle Gerd Oberlechner (geoberle) Awaiting requested review from geoberle
+1 more reviewer
@ahitacat Alba Hita (ahitacat) ahitacat left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@ahitacat Alba Hita (ahitacat)

Labels

approved lgtm

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@miguelsorianod @machi1990 @deads2k @ahitacat