Skip to content

unrevert of read from cosmos - #4610

Merged
openshift-merge-bot[bot] merged 10 commits into
mainfrom
cs-147-read-from-cosmos
May 6, 2026
Merged

unrevert of read from cosmos#4610
openshift-merge-bot[bot] merged 10 commits into
mainfrom
cs-147-read-from-cosmos

Conversation

@deads2k

Copy link
Copy Markdown
Collaborator

/hold

need to add fixes from #4412

@machi1990

Copy link
Copy Markdown
Collaborator

David Eads (@deads2k) The way

func GetClusterServiceUserAssignedIdentities(clusterServiceCluster *arohcpv1alpha1.Cluster) map[string]*arm.UserAssignedIdentity {
is written will get us in a spot where we still can set identity with empty clientId and principalId

This

clientID, _ := operatorIdentity.GetClientID()
needs to
be something like

clientID, ok := operatorIdentity.GetClientID() and we only set the cliendID to the returned response when ok is true.

We also need to ensure that the migration identity controller can handle the case of only some values being partially set

func (c *identityMigrationSyncer) NeedsWork(ctx context.Context, existingCluster *api.HCPOpenShiftCluster) bool {
and resync anything.

i.e NeedsWork re-written in a way that it says work is needed if any of the identity elements have a nil clientId or principalId

Comment thread frontend/pkg/frontend/cluster.go Outdated
}

// Clear the user-assigned identities map since that is built by a controller. The defaults will be set next and valid until we know the actual values.
newInternalCluster.Identity.UserAssignedIdentities = nil

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Behavior question I don't know the answer to. Do we need to validate the user provided .Identity.UserAssignedIdentities for some reason for ARM even though our long-held authoritative source is actually in .CustomerProperties.Platform.OperatorsAuthentication.UserAssignedIdentities.DataPlaneOperators (etc).

If we need to validate that every entry the user provided is present, we need to remove the defaulting I added in this PR and add the information only during reads.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ben Vesel (@bennerv) Matthew Barnes (@mbarnes) will know more;
I believe we could also get away by not overwriting the values and the EnsureDefaults() method.

By that I mean persist whatever we received from the user for the .Identity property as ARM would've validate that prior to the frontend processing the request. We remove the EnsureDefaults().

Then in the controller we sync the properties for the .Identity only if the clientId or principalId are empty for a given entry. When syncing the properties, if CS hasn't set it i.e ok = false then we make sure to live that as nil so that it'll be omitted from the returned json.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand the scope of your question wrt to validation so forgive me if I overanswer.

The .Identity.UserAssignedIdentities part of the API is the means by which user-assigned managed identities actually get assigned to the cluster resource by ARM.

I believe ARM does some validation here before the request reaches the RP, insofar as it will catch references to identities that don't exist. That would be worth verifying though.

The ARM requirement that makes this section tricky is that we have to provide the client and principal ID for each key in .Identity.UserAssignedIdentities whenever we return the cluster resource in a response (aside from the initial PUT response). Those values must be obtained from the Managed Identity Azure service.

Currently Cluster Service obtains those values and supplies them to the RP, which means that in order to satisfy the ARM requirement there has to be a strict 1:1 mapping of keys in the .Identity.UserAssignedIdentities map to a key in either the ...OperatorsAuthentication.UserAssignedIdentities.ControlPlaneOperators map or to the ...OperatorsAuthentication.UserAssignedIdentities.ServiceManagedIdentity field.

Once the backend is obtaining the client and principal IDs itself, we could relax this constraint slightly by tolerating extraneous keys in .Identity.UserAssignedIdentities that don't appear under ...OperatorsAuthentication.UserAssignedIdentities. Vice versa, however, won't work. If a key in ...OperatorsAuthentication.UserAssignedIdentities.ControlPlaneOperators does not appear in .Identity.UserAssignedIdenties, then that would mean the identity is not actually assigned to the cluster resource and won't be any good to the control plane operator.

In terms of the content of the .Identity.UserAssignedIdentities map, the value for each key must be an empty JSON object or -- in the case of a cluster update -- I believe it can be a JSON object that matches exactly what would be returned in a GET request (i.e. a JSON object with clientId and principalId keys and values that match what Cluster Service provides).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I think the summary of all that is, "we must validate the user input as provided. we must return frontend values with all the keys and either accurate or empty values." Doable. Will update.

Comment thread internal/api/types_cluster.go Outdated
Comment on lines +324 to +335
for _, operatorIdentityResourceID := range cluster.CustomerProperties.Platform.OperatorsAuthentication.UserAssignedIdentities.DataPlaneOperators {
if cluster.Identity == nil {
cluster.Identity = &arm.ManagedServiceIdentity{}
}
if cluster.Identity.UserAssignedIdentities == nil {
cluster.Identity.UserAssignedIdentities = make(map[string]*arm.UserAssignedIdentity)
}

if _, ok := cluster.Identity.UserAssignedIdentities[operatorIdentityResourceID.String()]; !ok {
cluster.Identity.UserAssignedIdentities[operatorIdentityResourceID.String()] = &arm.UserAssignedIdentity{}
}
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not needed for data plane; only ControlPlane + SMI identities should be in the cluster.Identity. S
Something like

identity: {
is what the user will provide which contains only the SMI + CP ones as those are treated differently from the DP ones.
For DP we don't request credentials directly from the MI RP but rather perform federations on them so they don't need to be in this list. However for the CP + SMI, do interact with the MI RP and hence the need for the .Identity stanza which makes ARM give the FPA credential ability to retrieve credentials for these identities from the MI RP

return true
}
}
for _, operatorIdentityResourceID := range existingCluster.CustomerProperties.Platform.OperatorsAuthentication.UserAssignedIdentities.DataPlaneOperators {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as #4610 (comment) we don't need to do this for DP identities

Comment thread internal/ocm/convert.go
Comment on lines 318 to 323
clientID, _ := operatorIdentity.GetClientID()
principalID, _ := operatorIdentity.GetPrincipalID()
internalCluster.Identity.UserAssignedIdentities[operatorIdentity.ResourceID()] = &arm.UserAssignedIdentity{
ret[operatorIdentity.ResourceID()] = &arm.UserAssignedIdentity{
ClientID: &clientID,
PrincipalID: &principalID,
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be written as so that we don't set the value to "" and potentially returning it to the end user when CS hasn't set it yet.

uai := &arm.UserAssignedIdentity{}
if clientID, ok := operatorIdentity.GetClientID(); ok {
		uai.ClientID = &clientID
}
if principalID, ok := operatorIdentity.GetPrincipalID(); ok {
	uai.PrincipalID = &principalID
}
ret[operatorIdentity.ResourceID()] = uai

CS only sets this value async (as part of the cluster provisioning process and not during cluster creation).

This will avoid putting us in a situation where we return something like this

..
identity: {
  "..../kms": {"clientID": "", "principalID": ""}
}

back to the end user when they do a GET and once they attempt a PUT using the gotten response it'll fail.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

David Eads (@deads2k) bumping this ^

Comment thread internal/ocm/convert.go
Comment on lines 326 to 331
clientID, _ := mi.ServiceManagedIdentity().GetClientID()
principalID, _ := mi.ServiceManagedIdentity().GetPrincipalID()
internalCluster.Identity.UserAssignedIdentities[mi.ServiceManagedIdentity().ResourceID()] = &arm.UserAssignedIdentity{
ret[mi.ServiceManagedIdentity().ResourceID()] = &arm.UserAssignedIdentity{
ClientID: &clientID,
PrincipalID: &principalID,
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as #4610

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@deads2k

Copy link
Copy Markdown
Collaborator Author

/retest

2 similar comments
@deads2k

Copy link
Copy Markdown
Collaborator Author

/retest

@deads2k

Copy link
Copy Markdown
Collaborator Author

/retest

@deads2k

Copy link
Copy Markdown
Collaborator Author

/hold

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took another round of reviews.
Just 1 one comment and bumping 2 old related comments that I think will be critical to address


controlPlaneExists := false
for _, resourceID := range existingCluster.CustomerProperties.Platform.OperatorsAuthentication.UserAssignedIdentities.ControlPlaneOperators {
if resourceID != nil && resourceID.String() == operatorIdentityResourceIDString {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if resourceID != nil && resourceID.String() == operatorIdentityResourceIDString {
if resourceID.String() == operatorIdentityResourceIDString {

is the nil check needed?

Comment thread internal/ocm/convert.go
Comment on lines 318 to 323
clientID, _ := operatorIdentity.GetClientID()
principalID, _ := operatorIdentity.GetPrincipalID()
internalCluster.Identity.UserAssignedIdentities[operatorIdentity.ResourceID()] = &arm.UserAssignedIdentity{
ret[operatorIdentity.ResourceID()] = &arm.UserAssignedIdentity{
ClientID: &clientID,
PrincipalID: &principalID,
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

David Eads (@deads2k) bumping this ^

Comment thread internal/ocm/convert.go
Comment on lines 326 to 331
clientID, _ := mi.ServiceManagedIdentity().GetClientID()
principalID, _ := mi.ServiceManagedIdentity().GetPrincipalID()
internalCluster.Identity.UserAssignedIdentities[mi.ServiceManagedIdentity().ResourceID()] = &arm.UserAssignedIdentity{
ret[mi.ServiceManagedIdentity().ResourceID()] = &arm.UserAssignedIdentity{
ClientID: &clientID,
PrincipalID: &principalID,
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@deads2k

Copy link
Copy Markdown
Collaborator Author

/retest

@deads2k

Copy link
Copy Markdown
Collaborator Author

/hold cancel

Comment thread internal/ocm/convert.go
Comment on lines -598 to -611
ServiceProviderProperties: api.HCPOpenShiftClusterServiceProviderProperties{
DNS: api.ServiceProviderDNSProfile{
BaseDomain: cluster.DNS().BaseDomain(),
},
Console: api.ServiceProviderConsoleProfile{
URL: cluster.Console().URL(),
},
API: api.ServiceProviderAPIProfile{
URL: cluster.API().URL(),
},
Platform: api.ServiceProviderPlatformProfile{
IssuerURL: cluster.Azure().OidcIssuerUrl(),
},
},

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this removal is problematic

{fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:145]: cluster Properties.API.URL was nil
Expected
    <*string | 0x0>: nil
not to be nil  fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:145]: cluster Properties.API.URL was nil
Expected
    <*string | 0x0>: nil
not to be nil}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it passed on a retest. Looks like it was just a flake

… mutation

Rather than clearing entirely, this change has the frontend create a
valid default. We also default on reading from storage so the return
value is always valid for the RP.
If the strings aren't longer than zero, they aren't valid and confuse
clients.
Copilot AI review requested due to automatic review settings April 30, 2026 13:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR shifts cluster reads/responses further toward Cosmos-only state (removing several legacy Cluster Service → RP conversion/merge paths) and updates integration fixtures accordingly, while adding logic to “complete” identity.userAssignedIdentities based on operator identity references.

Changes:

  • Remove legacy Cluster Service → RP cluster conversion and related tests/controllers; reduce frontend reliance on Cluster Service for reads/listing.
  • Introduce frontend identity “completion” to reconcile/prune identity.userAssignedIdentities from operator identity references.
  • Update integration test artifacts/fixtures to reflect the new identity payload shape and Cosmos document shape.

Reviewed changes

Copilot reviewed 20 out of 23 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
frontend/pkg/frontend/cluster.go Removes CS merge on list/read; adds completeClusterIdentity and calls it during create/update/read.
internal/ocm/convert.go Deletes legacy CS→RP conversion helpers; adds GetClusterServiceUserAssignedIdentities helper.
internal/ocm/convert_test.go Removes tests tied to deleted legacy conversion path.
internal/database/convert_defaults_consistency_test.go Removes defaults-consistency test that depended on legacy CS→RP conversion.
internal/database/convert_cluster.go Adjusts Cosmos identity conversion to deep-copy values when present.
backend/pkg/controllers/clusterpropertiescontroller/identity_migration.go Updates identity migration to use GetClusterServiceUserAssignedIdentities; expands NeedsWork logic.
backend/pkg/controllers/clusterpropertiescontroller/identity_migration_test.go Updates identity migration tests for operator identity references.
backend/pkg/controllers/clusterpropertiescontroller/cluster_customer_properties_migration*.go Removes the customer properties migration controller and its tests.
backend/pkg/app/backend.go Stops registering/running the removed customer properties migration controller.
test-integration/frontend/artifacts/** Updates fixtures for Cosmos cluster docs, identity payload shape, and expected validation errors.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread frontend/pkg/frontend/cluster.go Outdated
Comment thread frontend/pkg/frontend/cluster.go Outdated
Comment thread frontend/pkg/frontend/cluster.go
Comment thread frontend/pkg/frontend/cluster.go Outdated
Comment thread backend/pkg/controllers/clusterpropertiescontroller/identity_migration.go Outdated
Comment thread internal/ocm/convert.go Outdated
Comment thread internal/ocm/convert.go
Comment thread internal/ocm/convert.go
// TODO this is bad, see above TODOs. We want to validate what we store.
newInternalCluster.Identity.UserAssignedIdentities = nil
// we must validate using user provided .Identity.UserAssignedIdentities because that is the intent expressed by the user to allow
// us to use these identities. The information contained in those key is not trusted to be accurate, so we clear this field and set to

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you expand on this? I thought the end-user needs to provide both .Identity.UserAssignedIdentities (as it's a requirement by ARM) as well as the .platform.operatorsAuthentication.* data and we would just limit what we do to validate all the expected data is there and consistent between them instead of modifying .Identity with something different than they have provided.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We validate the identity in the call above. We clear the input before storage and set it to an empty valid value.

Copilot AI review requested due to automatic review settings May 4, 2026 13:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 20 out of 23 changed files in this pull request and generated 5 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 98 to +105
if err != nil {
return utils.TrackError(err)
}
clustersByClusterServiceID := make(map[string]*api.HCPOpenShiftCluster)
for _, internalCluster := range internalClusterIterator.Items(ctx) {
if internalCluster.ServiceProviderProperties.ClusterServiceID == nil {
// TODO this will be removed during our switch to read only from cosmos.
// we can still merge now since the value will never be nil until both the read path is fixed and this PR makes it to prod.
continue
resultingExternalCluster := versionedInterface.NewHCPOpenShiftCluster(internalCluster)
jsonBytes, err := arm.MarshalJSON(resultingExternalCluster)
if err != nil {
return utils.TrackError(err)
Comment on lines +915 to +920
if val, ok := cluster.Identity.UserAssignedIdentities[operatorIdentityResourceID.String()]; !ok || val == nil {
if existingValue, hasExisting := existingUserAssignedIdentity[operatorIdentityResourceID.String()]; hasExisting {
cluster.Identity.UserAssignedIdentities[operatorIdentityResourceID.String()] = existingValue.DeepCopy()
} else {
cluster.Identity.UserAssignedIdentities[operatorIdentityResourceID.String()] = &arm.UserAssignedIdentity{}
}
Comment on lines +933 to +938
if val, ok := cluster.Identity.UserAssignedIdentities[serviceManagedIdentity.String()]; !ok || val == nil {
if existingValue, hasExisting := existingUserAssignedIdentity[serviceManagedIdentity.String()]; hasExisting {
cluster.Identity.UserAssignedIdentities[serviceManagedIdentity.String()] = existingValue.DeepCopy()
} else {
cluster.Identity.UserAssignedIdentities[serviceManagedIdentity.String()] = &arm.UserAssignedIdentity{}
}
Comment on lines +103 to +105
if serviceManagedIdentity := existingCluster.CustomerProperties.Platform.OperatorsAuthentication.UserAssignedIdentities.ServiceManagedIdentity; serviceManagedIdentity != nil {
expectedIdentityResourceIDs[serviceManagedIdentity.String()] = struct{}{}
}
Comment on lines 23 to 27
"message": "Invalid value: \"/subscriptions/different-sub/resourceGroups/some-resource-group/providers/Microsoft.Network/virtualNetworks/test-vnet/subnets/test-subnet\": must not be the same resource group name: \"some-resource-group\"",
"target": "properties.platform.subnetId"
}
{
"code": "InvalidRequestContent",
Comment thread internal/ocm/convert.go
@miguelsorianod

Copy link
Copy Markdown
Collaborator

/lgtm

@openshift-ci

openshift-ci Bot commented May 5, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, machi1990, miguelsorianod

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

/retest-required

Remaining retests: 0 against base HEAD 2c8efd8 and 2 for PR HEAD daf3b71 in total

@openshift-ci

openshift-ci Bot commented May 5, 2026

Copy link
Copy Markdown
Contributor

David Eads (@deads2k): The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/cspr 4252e83 link true /test cspr
ci/prow/images-push 4252e83 link true /test images-push

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

/retest-required

Remaining retests: 0 against base HEAD c2abba6 and 1 for PR HEAD daf3b71 in total

@machi1990

Copy link
Copy Markdown
Collaborator

/retest-required

@openshift-merge-bot
openshift-merge-bot Bot merged commit 737b3da into main May 6, 2026
14 of 15 checks passed
@openshift-merge-bot
openshift-merge-bot Bot deleted the cs-147-read-from-cosmos branch May 6, 2026 11:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants