MSI support for ARC on Linux

[SCOMnewbie]

Library version used

4.56.0

.NET version

net6.0

Scenario

ManagedIdentityClient - managed identity

Is this a new or an existing app?

This is a new app or experiment

Issue description and reproduction steps

I'm developing a new Powershell module based on the MSAL.net library and I'm testing several scenarios. One of them is being able to generate an access token from an enrolled ARC Linux (tested on several Ubuntu 22.04). I don't think the problem comes from the Powershell module itself because thanks to your nice work, the code is pretty obvious.

On Windows ARC server it's working well.

The problem is not on the ARC side because I was able to generate tokens using "other methods". Here the error message I received from the prompt:

Relevant code snippets

$ClientApplicationBuilder = [Microsoft.Identity.Client.ManagedIdentityApplicationBuilder]::Create([Microsoft.Identity.Client.AppConfig.ManagedIdentityId]::SystemAssigned)
$ClientApplication = $ClientApplicationBuilder.Build()
$AquireTokenParameters = $ClientApplication.AcquireTokenForManagedIdentity($Scopes)
and then generate the call to get the token

Expected behavior

Should return an access token for the KeyVault scope in this case.

Identity provider

Microsoft Entra ID (Work and School accounts and Personal Microsoft accounts)

Regression

No response

Solution and workarounds

No response

[bgavrilMS]

Thanks for reporting @SCOMnewbie, this will be fun to repro :)

[bgavrilMS]

As a workaround, this may work with Azure Identity library, which also supports Managed Identity, but I am not sure. We practically copied most of the code from there.

[bgavrilMS]

A few questions.

1. What are the "other methods" you used to validate that MSI works?
2. Where is the VM located ? Is it a local VM or is something like an AWS VM?
3. What distro of Linux are you using? I guess Ubuntu 22.04

[SCOMnewbie]

I'm using the local MSI endpoint directly to generate the access token. Here an example, but long story short:

1. This is the local endpoint exposed on the machine: $MSIEndpoint = 'http://localhost:40342/metadata/identity/oauth2/token?api-version=2020-06-01'
2. You define the resource you want to hit and you url encode it: $EncodedURI = [System.Web.HttpUtility]::UrlEncode('https://vault.azure.net')
3. you merge the two: $AudienceURI = '{0}&resource={1}' -f $MSIEndpoint,$EncodedURI
4. Then you call the local endpoint once which will generate a key file in a specific place: Invoke-RestMethod -Uri $AudienceURI -Headers $Headers -ErrorAction stop
5. Then you build a header with this key file content: $Headers = @{'Metadata' = 'true'};$Headers.Add('Authorization',$("Basic $(Get-Content -Path $ARCTokensPath -ErrorAction SilentlyContinue)"))
6. Now you can re-do the same call as before but this time with a proper header in place: Invoke-RestMethod -Uri $AudienceURI -Headers $Headers

Regarding the VM, I've tested from GCP and a hyperV local to my machine lab with a Ubuntu 22.04.

[neha-bhargava]

@SCOMnewbie the way managed identity works is it tries to find a combination of environment variable to build the endpoint to use to get the token. I believe the endpoint here is not correctly discovered by MSAL to use to get the token from. These environment variables are set by default on the azure resources. I don't think the endpoint you mentioned in the comment above is being used.

[bgavrilMS]

@neha-bhargava - didn't we test with Azure Arc? It is a supported sceanrio. I assume we tested with a Windows VM. The repro here is a Linux VM.

I would expect that when installing the Azure Arc agent, the agent will add the correct env variables. If this is not happening, we need to raise this with the Azure Arc folks.

[neha-bhargava]

Azure arc is a supported scenario and should work. But the endpoint mentioned above is confusing to me. For Azure arc, we do get the endpoint from the environment variables but the API version we use is 2019-11-01. So, the endpoint shared above is not something that MSAL will frame. Having logs will help I think.

[SCOMnewbie]

Hi @neha-bhargava @bgavrilMS ,
Thank you for your answers. For the fun this morning, I've decided to see what is created when you enroll a new machine in ARC for both a Windows and a Linux.

For Windows:

As you can see, there is no MSI Endpoint declared in the environment variables:

But the agent is working well as you can see:

Now if I install the Powershell module and run the command, the ARC agent will return a token:

For Linux:

Once enrolled in ARC:

If you check the environment variable, we still don't have any MSI endpoint declared:

but this time, it doesn't work.

I assume that looking at the environment variable is not the only option because again, it's working on Windows.

Cheers

[neha-bhargava]

@SCOMnewbie the environment variable we are looking for is IDENTITY_ENDPOINT. Are you able to get this environment variable on either?

[SCOMnewbie]

Maybe I've misunderstood the question but the screenshots above are all env variables post ARC enrollment. As you can see there is no IDENTITY_ENDPOINT variable.
This is why I'm surprised that it's working on Windows without this environment variable as well.

[bgavrilMS]

@SCOMnewbie - is it possible to get MSAL logs from the Windows env (see https://github.com/AzureAD/microsoft-authentication-library-for-dotnet/wiki/logging)

We are reaching out to Azure Arc folks for an opinion here. My understanding is that Azure Arc + Linux is relatively new and it's possible that the team didn't get their ENV var to be in sync with Windows.

[bgavrilMS]

This seems to be an issue on how the env variables are set.

There is no ‘global’ set of environment variables on Linux, every process hands them on to its children. Azure ARC on Linux uses systemd to make the variable available to all systemd units but haven’t worked out a reliable way to publish it so every process will see it.

[SCOMnewbie]

@bgavrilMS Do you still need the logs in this case?

[bgavrilMS]

No, I think we're good. The logs would only show that the 2 env variables are not there and MSAL thinks it's on a Azure VM instead of Arc. And then it fails with the exception above.

I'm reaching out to Azure SDK folks to see how they overcome this problem on App Services and Service Fabric - that logic also relies on env variables.

[bgavrilMS]

Looks like this never really worked and it's not easy to get right because of how env variables work on Linux.