[BUG] token retrieval intermittently stuck in BearerTokenAuthenticationPolicy in Azure.Core 1.40.0

[MarcWils]

Library name and version

Azure.Core 1.40.0

Describe the bug

Same as #44817. However, the "fix" in #44882 only works if a CancellationToken is passed on and cancellation is requested.

We're using ASP.NET Core Data Protection with Azure.Extensions.AspNetCore.DataProtection.Blobs. This is building an Azure.Core.HttpMessage with default cancellation token. So, it is never cancelled and requests hang indefinitely.

I know Azure.Core 1.42.0 is the latest version. But it seems the fix in #44882 would not work in this scenario. The workaround using an CancellationToken does not solve the real issue. There seems to be a specific scenario in which the CurrentTokenTcs never gets a result.

Multiple of our applications deadlock in this situation:

[Managed to Native Transition]
System.Private.CoreLib.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout = -1, System.Threading.CancellationToken cancellationToken) Line 264 C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 2386 C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.InternalWaitCore(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 2354 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task = Id = 50, Status = WaitingForActivation, Method = "{null}", Result = "{Not yet computed}") Line 51 C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter<Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.AuthHeaderValueInfo>.GetResult() Line 173 C#
[Waiting on Async Operation, double-click or press enter to view Async Call Stacks]
Azure.Core.dll!Azure.Core.Pipeline.TaskExtensions.EnsureCompleted<Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.AuthHeaderValueInfo>(System.Threading.Tasks.Task<Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.AuthHeaderValueInfo> task = Id = 50, Status = WaitingForActivation, Method = "{null}", Result = "{Not yet computed}") Line 33 C#
Azure.Core.dll!Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.TokenRequestState.GetCurrentHeaderValue(bool async = false, bool checkForCompletion = false, System.Threading.CancellationToken cancellationToken = IsCancellationRequested = false) Line 438 C#
[Resuming Async Method]
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start<Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.TokenRequestState.d__19>(ref Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.TokenRequestState.d__19 stateMachine = {Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.TokenRequestState.d__19}) Line 55 C#
Azure.Core.dll!Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetAuthHeaderValueAsync(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, Azure.Core.TokenRequestContext context = {Azure.Core.TokenRequestContext}, bool async = false) Line 207 C#
Azure.Core.dll!Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AuthenticateAndAuthorizeRequest(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, Azure.Core.TokenRequestContext context = {Azure.Core.TokenRequestContext}) Line 172 C#
Azure.Storage.Blobs.dll!Azure.Storage.StorageBearerTokenChallengeAuthorizationPolicy.AuthorizeRequestInternal(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, bool async = false) Line 69 C#
Azure.Storage.Blobs.dll!Azure.Storage.StorageBearerTokenChallengeAuthorizationPolicy.AuthorizeRequest(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}) Line 52 C#
Azure.Core.dll!Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline = "System.ReadOnlyMemory[4]", bool async = false) Line 127 C#
Azure.Core.dll!Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.Process(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline = "System.ReadOnlyMemory[4]") Line 61 C#
Azure.Core.dll!Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(Azure.Core.HttpMessage message, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline) Line 47 C#
Azure.Core.dll!Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline = "System.ReadOnlyMemory[5]") Line 40 C#
Azure.Core.dll!Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(Azure.Core.HttpMessage message, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline) Line 47 C#
Azure.Core.dll!Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline = "System.ReadOnlyMemory[6]", bool async = false) Line 50 C#
Azure.Core.dll!Azure.Core.Pipeline.RedirectPolicy.Process(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline = "System.ReadOnlyMemory[6]") Line 198 C#
Azure.Core.dll!Azure.Core.Pipeline.RetryPolicy.ProcessAsync(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline = "System.ReadOnlyMemory[7]", bool async = false) Line 85 C#
Azure.Core.dll!Azure.Core.Pipeline.RetryPolicy.Process(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline = "System.ReadOnlyMemory[7]") Line 59 C#
Azure.Core.dll!Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(Azure.Core.HttpMessage message, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline) Line 47 C#
Azure.Core.dll!Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline = "System.ReadOnlyMemory[8]") Line 40 C#
Azure.Core.dll!Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(Azure.Core.HttpMessage message, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline) Line 47 C#
Azure.Core.dll!Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline = "System.ReadOnlyMemory[9]") Line 40 C#
Azure.Core.dll!Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(Azure.Core.HttpMessage message, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline) Line 47 C#
Azure.Core.dll!Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline = "System.ReadOnlyMemory[10]") Line 40 C#
Azure.Core.dll!Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(Azure.Core.HttpMessage message, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline) Line 47 C#
Azure.Core.dll!Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline = "System.ReadOnlyMemory[11]") Line 40 C#
Azure.Core.dll!Azure.Core.Pipeline.HttpPipeline.Send(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, System.Threading.CancellationToken cancellationToken = IsCancellationRequested = false) Line 174 C#
Azure.Storage.Blobs.dll!Azure.Storage.Blobs.BlobRestClient.Download(string snapshot = null, string versionId = null, int? timeout = null, string range = "bytes=0-268435455", string leaseId = null, bool? rangeGetContentMD5 = null, bool? rangeGetContentCRC64 = null, string encryptionKey = null, string encryptionKeySha256 = null, Azure.Storage.Blobs.Models.EncryptionAlgorithmTypeInternal? encryptionAlgorithm = null, System.DateTimeOffset? ifModifiedSince = null, System.DateTimeOffset? ifUnmodifiedSince = null, string ifMatch = null, string ifNoneMatch = "0x8DC8A29E41EA7D4", string ifTags = null, System.Threading.CancellationToken cancellationToken = IsCancellationRequested = false) Line 175 C#
Azure.Storage.Blobs.dll!Azure.Storage.Blobs.Specialized.BlobBaseClient.StartDownloadAsync(Azure.HttpRange range = {Azure.HttpRange}, Azure.Storage.Blobs.Models.BlobRequestConditions conditions = {Azure.Storage.Blobs.Models.BlobRequestConditions}, Azure.Storage.DownloadTransferValidationOptions validationOptions = {Azure.Storage.DownloadTransferValidationOptions}, long startOffset = 0, bool async = false, System.Threading.CancellationToken cancellationToken = IsCancellationRequested = false) Line 1737 C#
Azure.Storage.Blobs.dll!Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadStreamingInternal(Azure.HttpRange range = {Azure.HttpRange}, Azure.Storage.Blobs.Models.BlobRequestConditions conditions = {Azure.Storage.Blobs.Models.BlobRequestConditions}, Azure.Storage.DownloadTransferValidationOptions transferValidationOverride = {Azure.Storage.DownloadTransferValidationOptions}, System.IProgress progressHandler = null, string operationName = "BlobBaseClient.DownloadStreaming", bool async = false, System.Threading.CancellationToken cancellationToken = IsCancellationRequested = false) Line 1561 C#
Azure.Storage.Blobs.dll!Azure.Storage.Blobs.PartitionedDownloader.DownloadTo(System.IO.Stream destination = {System.IO.MemoryStream}, Azure.Storage.Blobs.Models.BlobRequestConditions conditions = {Azure.Storage.Blobs.Models.BlobRequestConditions}, System.Threading.CancellationToken cancellationToken) Line 307 C#
Azure.Storage.Blobs.dll!Azure.Storage.Blobs.Specialized.BlobBaseClient.StagedDownloadAsync(System.IO.Stream destination = {System.IO.MemoryStream}, Azure.Storage.Blobs.Models.BlobRequestConditions conditions = {Azure.Storage.Blobs.Models.BlobRequestConditions}, System.IProgress progressHandler = null, Azure.Storage.StorageTransferOptions transferOptions = {Azure.Storage.StorageTransferOptions}, Azure.Storage.DownloadTransferValidationOptions transferValidationOverride = null, bool async = false, System.Threading.CancellationToken cancellationToken = IsCancellationRequested = false) Line 2893 C#
Azure.Storage.Blobs.dll!Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadTo(System.IO.Stream destination = {System.IO.MemoryStream}, Azure.Storage.Blobs.Models.BlobRequestConditions conditions = {Azure.Storage.Blobs.Models.BlobRequestConditions}, Azure.Storage.StorageTransferOptions transferOptions = {Azure.Storage.StorageTransferOptions}, System.Threading.CancellationToken cancellationToken = IsCancellationRequested = false) Line 2677 C#
Azure.Extensions.AspNetCore.DataProtection.Blobs.dll!Azure.Extensions.AspNetCore.DataProtection.Blobs.AzureBlobXmlRepository.GetLatestData() Line 199 C#
Azure.Extensions.AspNetCore.DataProtection.Blobs.dll!Azure.Extensions.AspNetCore.DataProtection.Blobs.AzureBlobXmlRepository.GetAllElements() Line 57 C#
Microsoft.AspNetCore.DataProtection.dll!Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.GetAllKeys() Unknown
Microsoft.AspNetCore.DataProtection.dll!Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.CreateCacheableKeyRingCore(System.DateTimeOffset now = {System.DateTimeOffset}, Microsoft.AspNetCore.DataProtection.KeyManagement.IKey keyJustAdded = null) Unknown
Microsoft.AspNetCore.DataProtection.dll!Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.ICacheableKeyRingProvider.GetCacheableKeyRing(System.DateTimeOffset now = {System.DateTimeOffset}) Unknown
Microsoft.AspNetCore.DataProtection.dll!Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.GetCurrentKeyRingCore(System.DateTime utcNow, bool forceRefresh) Unknown
Microsoft.AspNetCore.DataProtection.dll!Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(byte[] plaintext = {byte[949538]}) Unknown

Expected behavior

Blob storage call is executed

Actual behavior

The call is stuck and never completes

Reproduction Steps

Hard to reproduce

Environment

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[jsquire]

@christothes: Would you please take a look and offer your thoughts?

[MarcWils]

@christothes : Maybe it's due to the fact there's a single BlobRestClient' with a single HttpPipeline containing the same BearerTokenAuthenticationPolicy instance? I can see 2 concurrent Download calls on the same BlobRestClient instance.

I'm not too familiar with this code, but is thread safe? Concurrent calls on https://github.com/christothes/azure-sdk-for-net/blob/6c76e7d2374473c7be189f7cf35d641024a0d164/sdk/core/Azure.Core/src/Pipeline/BearerTokenAuthenticationPolicy.cs#L256 could modify the same state. Although with the lock, it would be one after the other. However, does https://github.com/christothes/azure-sdk-for-net/blob/6c76e7d2374473c7be189f7cf35d641024a0d164/sdk/core/Azure.Core/src/Pipeline/BearerTokenAuthenticationPolicy.cs#L203-L205 still work if the 2nd calll also changed the state before this code is executed?

[timaiv]

#45223

[MarcWils]

My bad, i missed this subtle change: 93512b1?diff=split&w=0#diff-f6d09d34c9aed3acf5957c27473cfdc4a1d52f0219b8a25339e0abb206a89209R412.

This issue can be closed as I've only seen it with Azure.Core 1.40.0. So it might be already be fixed.