Replication delays: Not a valid reference update error UserAssignedIdentity and Microsoft.Graph/[email protected]

[Anttarax]

I want to create UserAssignedIdentity, After that I want to Assign an Appreg role to the newly created (service principal of) UserAssignedIdentity.
The UserAssignedIdentity created successfully, but "Microsoft.Graph/[email protected]" thinks that userAssignedIdentity.properties.principalId is invalid, but not.

If I run second time, The deployment success, but first time, when the UserAssignedIdentity creating, I get this error message.

Error message: Not a valid reference update. Graph client request id: 370e13eb-3387-4e5a-b2b6-5f77c357b5dc. Graph request timestamp: 2024-11-18T00:13:49Z.
(Code: BadRequest)

Calling userAssignedIdentities.bicep:

module userAssignedIdentity 'userAssignedIdentities.bicep' = if (userAssignIdentityrequired == true) {
  name: guid(appname, appRoleId)
  scope: resourceGroup(SubscriptionId, ResourceGroupName)
  params: {
    tags: tags
    appname: appname
    appRoleId: appRoleId
    AppServicePrincipal: AppServicePrincipal
  }
}

userAssignedIdentities.bicep:

param appname string
param tags object
extension microsoftGraph
param appRoleId string
param AppServicePrincipal string

resource userAssignedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' =  {
  name: 'id-${appname}'
  location: 'westeurope'
  tags: tags
}

@description('Grant role to the UserAssignedIdentity')
resource appRoleAssignedToUserAssignedIdentity 'Microsoft.Graph/[email protected]' = {
  appRoleId: appRoleId
  principalId: userAssignedIdentity.properties.principalId
  resourceId: AppServicePrincipal
}

[dkershaw10]

@Anttarax - during deployment, what are you setting the resourceId and appRoleId to? I don't see anywhere that defines what AppServicePrincipal or what the AppRoleId values are. Are you providing these on the cmd line?

I'll have to check, but the "Not a valid reference" message can apply to any of the properties of the Microsoft.Graph/appRoleAssignedTo resource properties.

I'm not sure I follow a lot of the information, especially:
If I run second time, The deployment success, but first time, when the UserAssignedIdentity creating, I get this error message.
DONT USE THE SAME SCOPE FOR USERASSIGNEDIDENTITY, BECAUSE in that case there are no error message. But We dont want to create all UserAssignedIdentity in the same ResourceGroup, but the same Appreg Bicep create all of them, each to another ResourceGroup

Can you shed some more light on this please?

[Anttarax]

@dkershaw10 No, I dont providing these on the cmd line. But you should because its a simple way to test it.

I completely automated the creation of Appregs. I use all the Bicep Graph resources, a bicepparam contains all Appregs in an array and everything is generated.

I can share the complete code, but it's very long and it would take you several hours/days to analyze it. Just use a valid AppServicePrincipal and ApproleId, believe me the values ​​are good, as I generate more than 100 Appregs with this code every day. Now the code has only been extended by creating the UserAssignedIdentity (and role assign to the UserAssignedIdentity). System Assigned Managed Identity and Appreg role to Appreg work correctly.
Please unit test it.

[Anttarax]

@dkershaw10 Here you are. A very simple code. Just replace the values of appRoleId and AppServicePrincipal to your own Appreg. You will definitely get an error. The scope and ResourceGroup that I mentioned earlier are not interesting, you will get an error anyway with or without any module scope (at first run, when userAssignedIdentity is not exist and creating.

extension microsoftGraph
param appname string = 'example-app' //Any string
param appRoleId string  = '91804f26-164c-53ec-b602-5f5e895fe73c' //ID of any Role in Appreg (same Appreg as in AppServicePrincipal)
param AppServicePrincipal string = '3135ce86-5184-465a-b1e9-1379e8b7df3d' //Enterprise Application Object ID of any Appreg (same Appreg as in appRoleId)


resource userAssignedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-07-31-preview' =  {
  name: 'id-${appname}'
  location: 'westeurope'
}

@description('Grant role to the UserAssignedIdentity')
resource appRoleAssignedToUserAssignedIdentity 'Microsoft.Graph/[email protected]' = {
  appRoleId: appRoleId
  principalId: userAssignedIdentity.properties.principalId
  resourceId: AppServicePrincipal
}

Output: Not a valid reference update. Graph client request id: f26c6d30-fecd-46c0-9b50-c66ce2571bdd. Graph request timestamp: 2024-11-19T04:07:25Z."}}"}]}}

[dkershaw10]

Interesting. I have a similar script that I tried (below). It completes successfully when I deploy.
I will also try referencing the created objects in a separate script, similar to above.
I will likely need one of our engineers to look at our trace telemetry to see exactly what the issue might be.

extension graphv1_0

// TEMPLATE OVERVIEW: Creates a MI as a client service, registers a resource app which
// defines an app role, and then creates a resource app SP.
// Finally assign the app role (on the resource app SP) to the MI

param location string = resourceGroup().location

param deployDate string = '2024-11-19'
param clientAppName string = 'miService-${deployDate}'
param resourceName string = 'resourceServiceName-${deployDate}'

@description('Id of the application role to add to the resource app')
var appRoleName = 'ResourceAppData.Read.All'
var appRoleId = guid(resourceName, appRoleName)

// create a managed identity for the client service
resource managedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
  name: 'mi-${uniqueString(clientAppName,location)}'
  location: location
}

// create resource app
resource resourceApp 'Microsoft.Graph/[email protected]' = {
  uniqueName: resourceName
  displayName: resourceName
  appRoles: [
    {
      id: appRoleId
      allowedMemberTypes: [ 'User', 'Application' ]
      description: 'Read access to resource app data'
      displayName: appRoleName
      value: appRoleName
      isEnabled: true
    }
  ]
}

// Create resource SP
resource resourceSp 'Microsoft.Graph/[email protected]' = {
  appId: resourceApp.appId
}

// Assign app role (from resource app) to MI
resource appRoleAssignment 'Microsoft.Graph/[email protected]' = {
  principalId: managedIdentity.properties.principalId
  resourceId: resourceSp.id
  appRoleId: appRoleId
}

output clientSpId string = managedIdentity.properties.principalId
output resourceSpId string = resourceSp.id
output appRoleId string = appRoleId

[dkershaw10]

I managed to reproduce your error, but only once, having tried > 10 times with the following steps.

Repro steps

1. Create a Bicep template that creates a user assigned managed identity, that also assigns an app role to the created managed identity - like your template, the resource app and role are pre-created and referenced by their IDs (although you could read them using the existing keyword too).
2. Deploy the template.
3. Wait a bit and redeploy the template.

Expected
Both deployments are successful.

Actual (when it failed)
First deployment fails with the "Not a valid reference update." error
Second deployment succeeds.

Likely cause
When creating a managed identity, this is created via Azure/ARM APIs (which under the hood create a service principal in Entra ID). When creating the app role assignment, this request is coming via the Graph Bicep extension calling Microsoft Graph and then on to Entra ID. It is possible that this second request hits a different read replica from the first request, and that the created service principal is not (yet) present in the read replica. I'll need to ask engineering to confirm if this might be the case.

@Anttarax - if you redeploy the same template file a little later, is the redeployment successful?

cc: @eketo-msft

[Anttarax]

@dkershaw10
Yes. As I mentioned at the very beginning, running it a second time, it succeeds when the userAssignedIdentity already exists.
First step of Repro steps: Delete userAssignedIdentity if already exist. After that run the template.

[slavizh]

@dkershaw10 most likely it is the read replica thing. There is the same issue when creating system/user assigned identities and assigning RBAC roles (role assignments) to them. To overcome this role Assignments API https://learn.microsoft.com/en-us/azure/templates/microsoft.authorization/roleassignments?pivots=deployment-language-bicep has the parameter principalType which when provided value ServicePrincipal it does not check if the identity exists, it just assigns the permissions. Most likely the team behind role assignments knows the inner works details. Of course not suggesting that the same thing for role assignments will work or you have to implement it just mentioning that similar issue exists with other RPs.

[dkershaw10]

@Anttarax - All my "re-runs" were with a new user-assigned MI creation.

@slavizh - yeah maybe the Azure role assignment has some special treatment here (could be because it doesn't want to call out to Entra to check, for perf reasons - who knows :)). Anyways, little we can do about the inner workings of the Entra system for app role assignment.
One thing we have talked about internally (and I think we have a backlog item for it) is to have some retry logic in the Graph Bicep extension. I don't know that we want to do that for every 400 error, but maybe for some specific error conditions, like 400s from this appRoleAssignedTo API, as this seems like a common Azure -> Entra crossover case.
Another one might be when creating a federated identity credential using a newly created managed identity as the subject.

@eketo-msft what do you think?

[Anttarax]

Any news? @dkershaw10

[eketo-msft]

I confirmed the issue was replication latency. The referenced object (MSI backing service principal in Entra) was not available yet.

For appRoleAssignedTo we can consider adding a longer timeout/retry policy since we know there is likely some cross-over from Azure provisioning the resource. From what I can tell based on the few repro's here the SP is being created in Entra within a minute of the failed request, so if we consider replication time it should still only be a few minutes to retry. Let's discuss with the engineering team and see what we can come up with to alleviate the issue.

[dkershaw10]

@eketo-msft I think we need to consider all cross-over scenarios where MSIs might be used, for longer timeout/retry policy:

1. MSI is used in an app role assignment
2. MSI is added as a group member/owner
3. MSI is added as the subject in a Federated Identity Credential (this is coming very soon)