[BUG] token retrieval intermittently stuck in BearerTokenAuthenticationPolicy in Azure.Core 1.40.0

[v-mykhalchuk]

Library name and version

Azure.Core 1.40.0

Describe the bug

Hello,
After update from version 1.38.0 to 1.40.0 of Azure.Core we have started to experience intermittent issues of Requests being stuck in BearerTokenAuthenticationPolicy.
Specifically, we are trying to get a secret from key vault under user-assigned managed indentity, and we see that the SecretClient.GetSecret dependency is being stuck for a duration of our request timeout, and then fails with OperationCancelledException. Same code seem to work well after downgrade back to 1.38.0 without any other changes.

When the issue occurs, it reproduces pretty reliably in the same place for the same secret until application restart.

There is no high concurrency involved in this code path.

Stack Trace:

System.OperationCanceledException:
at System.Threading.CancellationToken.ThrowOperationCanceledException (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Core.Pipeline.TaskExtensions+WithCancellationTaskAwaiter`1.GetResult (Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy+AccessTokenCache+TokenRequestState+d__19.MoveNext (Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy+AccessTokenCache+d__6.MoveNext (Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy+d__12.MoveNext (Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy+d__7.MoveNext (Azure.Security.KeyVault.Secrets, Version=4.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy+d__11.MoveNext (Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Core.Pipeline.RedirectPolicy+d__7.MoveNext (Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Core.Pipeline.RetryPolicy+d__5.MoveNext (Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Core.Pipeline.RetryPolicy+d__5.MoveNext (Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Core.Pipeline.HttpPipeline+d__18.MoveNext (Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Security.KeyVault.KeyVaultPipeline+d__29.MoveNext (Azure.Security.KeyVault.Secrets, Version=4.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Security.KeyVault.KeyVaultPipeline+d__21'1.MoveNext (Azure.Security.KeyVault.Secrets, Version=4.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.Security.KeyVault.Secrets.SecretClient+d__10.MoveNext (Azure.Security.KeyVault.Secrets, Version=4.6.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)

Expected behavior

KeyVault call is executed

Actual behavior

The call is stuck and times out

Reproduction Steps

Unfortunately, we don't have a repro yet.

Environment

Azure App Services, Windows, .NET 8.0

[jsquire]

Hi @v-mykhalchuk. Thanks for reaching out and we regret that you're experiencing difficulties. Unfortunately, there's not enough information available to assist. Please collect a 5-minute slice of verbose Azure SDK logs around the behavior for analysis. Examples for gathering them can be found in this sample and further context is discussed in [Logging with the Azure SDK for .NET)(https://learn.microsoft.com/dotnet/azure/sdk/logging).

[github-actions]

Hi @v-mykhalchuk. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[PixelByProxy]

I also experienced this issue after upgrading to Azure.Identity 1.12.0 which updated the Azure.Core reference to 1.40.0. Reverting back to 1.11.2 (Azure.Core 1.38.0) resolved the issue.

Stack Trace:
[{"severityLevel":"Warning","outerId":"0","message":"The operation was canceled.","type":"System.OperationCanceledException","id":"54893588","parsedStack":[{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Threading.CancellationToken.ThrowOperationCanceledException","level":0,"line":0},{"assembly":"Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8","method":"Azure.Core.Pipeline.BearerTokenAuthenticationPolicy+AccessTokenCache+TokenRequestState+<GetCurrentHeaderValue>d__19.MoveNext","level":1,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":2,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":3,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":4,"line":0},{"assembly":"Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8","method":"Azure.Core.Pipeline.BearerTokenAuthenticationPolicy+AccessTokenCache+<GetAuthHeaderValueAsync>d__6.MoveNext","level":5,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":6,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":7,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":8,"line":0},{"assembly":"Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8","method":"Azure.Core.Pipeline.BearerTokenAuthenticationPolicy+<AuthenticateAndAuthorizeRequestAsync>d__12.MoveNext","level":9,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":10,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":11,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":12,"line":0},{"assembly":"Azure.Storage.Blobs, Version=12.19.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8","method":"Azure.Storage.StorageBearerTokenChallengeAuthorizationPolicy+<AuthorizeRequestInternal>d__7.MoveNext","level":13,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":14,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":15,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":16,"line":0},{"assembly":"Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8","method":"Azure.Core.Pipeline.BearerTokenAuthenticationPolicy+<ProcessAsync>d__11.MoveNext","level":17,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":18,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":19,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":20,"line":0},{"assembly":"Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8","method":"Azure.Core.Pipeline.HttpPipelineSynchronousPolicy+<InnerProcessAsync>d__5.MoveNext","level":21,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":22,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":23,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":24,"line":0},{"assembly":"Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8","method":"Azure.Core.Pipeline.RedirectPolicy+<ProcessAsync>d__7.MoveNext","level":25,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":26,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":27,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":28,"line":0},{"assembly":"Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8","method":"Azure.Core.Pipeline.RetryPolicy+<ProcessAsync>d__5.MoveNext","level":29,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":30,"line":0},{"assembly":"Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8","method":"Azure.Core.Pipeline.RetryPolicy+<ProcessAsync>d__5.MoveNext","level":31,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":32,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":33,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":34,"line":0},{"assembly":"Azure.Storage.Blobs, Version=12.19.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8","method":"Azure.Storage.Blobs.PageBlobRestClient+<UploadPagesAsync>d__11.MoveNext","level":35,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":36,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":37,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":38,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.ConfiguredTaskAwaitable``1+ConfiguredTaskAwaiter.GetResult","level":39,"line":0},{"assembly":"Azure.Storage.Blobs, Version=12.19.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8","method":"Azure.Storage.Blobs.Specialized.PageBlobClient+<UploadPagesInternal>d__36.MoveNext","level":40,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":41,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":42,"line":0},{"assembly":"System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":43,"line":0},{"assembly":"Azure.Storage.Blobs, Version=12.19.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8","method":"Azure.Storage.Blobs.Specialized.PageBlobClient+<UploadPagesAsync>d__33.MoveNext","level":44,"line":0}

[v-mykhalchuk]

@jsquire we are working on some sort of repro for this. We had to revert back the version in our main code as this was blocking our testing. I will reply back when we are able to collect any additional info on this.