Intermittent Error with Remote Session State Sharing in .NET Core 6.0 Application Using SystemWebAdapters

[darshan-edspl]

I am reaching out regarding an intermittent issue we are experiencing with session state sharing between our .NET Framework 4.7.2 and .NET Core 6.0 applications using the side-by-side incremental migration approach and the YARP proxy. I am utilizing the System.WebAdapters package to share sessions between the old and new applications, and while this setup generally works as expected, we encounter a sporadic error when accessing certain upgraded pages in our .NET Core 6.0 application.

This error does not occur every time; it seems to happen randomly when old application page stay ideal for around 3-4 min and trying to access upgraded pages in the .NET Core application. And there is traffic on the site of around 10 user accessing the site.

To provide further context:
We are using CustomSerialization to manage dynamic sessions across both applications, which includes serialization and deserialization of session keys in use.
Session sharing is configured to work between the applications and generally behaves correctly, with data persisting accurately in both contexts.

The specific error we receive is as follows:

Unable to load remote session state for session r5fnl1cnxcv53m3okxorxs5y
System.Net.Http.HttpRequestException: Response status code does not indicate success: 302 (Found).
at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
at Microsoft.AspNetCore.SystemWebAdapters.SessionState.RemoteSession.RemoteAppSessionStateManager.GetSessionDataAsync(String sessionId, Boolean readOnly, HttpContext callingContext, CancellationToken token)
at Microsoft.AspNetCore.SystemWebAdapters.SessionState.RemoteSession.RemoteAppSessionStateManager.CreateAsync(HttpContext context, SessionAttribute metadata)

Please find attached:

Program.cs file from the .NET Core 6.0 application.
global.asax file from the .NET Framework 4.7.2 application.
The CustomSerialization class we are using for session serialization and deserialization in both old and new application.
CustomSerialization_cs.txt
Program_cs.txt
Global_asax_cs.txt

Any insights into the cause of this intermittent error, and guidance on how to ensure consistent session state sharing, would be greatly appreciated.

ASP.NET Framework Application:

• Technologies and versions used (i.e. MVC/WebForms/etc):
• .NET Framework Version: 4.7.2
• IIS Version: 10
• Windows Version: Windows Server 2016

ASP.NET Core Application:

• Targeted .NET version: .Net Core 6.0

Thank you in advance for your assistance.

[twsouthwick]

Can you share the modules/handlers in your web.config as well as the contents of your global.asax.cs? Redact whatever you need there, I'm interested in what else might be hooking into the module pipeline that would be returning a 302 as the session endpoint doesn't do that.

Have you been able to create a standalone repro of it? One that we can run to see it happen?

[darshan-edspl]

Thanks for quick reply.

I am attaching web.config and global.asax.cs
[Global_asax_cs.txt]
(https://github.com/user-attachments/files/17810751/Global_asax_cs.txt)
[Web_config.txt]
(https://github.com/user-attachments/files/17810763/Web_config.txt)

As per the Microsoft.AspNetCore.SystemWebAdapters.SessionState.RemoteSession library, we are encountering an error in the 'CreateAsync' method of the 'RemoteAppSessionStateManager' class.

Upon further investigation, I found that the error occurs in the following statement within the 'GetSessionDataAsync' method.
var response = await _backchannelClient.SendAsync(req, HttpCompletionOption.ResponseHeadersRead, token);

[darshan-edspl]

More detail about the error

While monitoring in detail, I discovered that the new application (.NET Core) is attempting to retrieve session details from the old application while serving web pages for the new application. It appears that the BinarySessionSerializer.Write process gets stuck when many users try to access the new application. I think this detail will be useful to find root cause of the problem. The error details are as follows:

URL: https://abc.com/systemweb-adapters/session | Route URL: {controller}/{action}/{id} | Route Data:
[controller: systemweb-adapters, action: session, id: ] -- Collection was modified after the enumerator was instantiated.
-- Stack Trace: at System.Collections.Specialized.NameObjectCollectionBase.NameObjectKeysEnumerator.MoveNext()
at System.Linq.Enumerable.d__97`1.MoveNext()
at Microsoft.AspNetCore.SystemWebAdapters.SessionState.Serialization.BinarySessionSerializer.Write(ISessionState state, BinaryWriter writer)
at Microsoft.AspNetCore.SystemWebAdapters.SessionState.Serialization.BinarySessionSerializer.SerializeAsync(ISessionState state, Stream stream, CancellationToken token)
at Microsoft.AspNetCore.SystemWebAdapters.SessionState.RemoteSession.GetWriteableSessionHandler.d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNetCore.SystemWebAdapters.SessionState.RemoteSession.GetWriteableSessionHandler.d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.TaskAsyncHelper.EndTask(IAsyncResult ar)
at System.Web.HttpTaskAsyncHandler.System.Web.IHttpAsyncHandler.EndProcessRequest(IAsyncResult result)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

[twsouthwick]

This is odd, but helpful. Can you provide the following:

• What is your session provider (usually configured in web.config)?
• Are you doing any threading (either System.Threading.Thread or Tasks) that may be adding items to the session state?

[darshan-edspl]

What is your session provider (usually configured in web.config)?
Answer : - web.config already attached detail as below. InProd mode is used.

<sessionState customProvider="DefaultSessionProvider" mode="InProc" timeout="45">
  <providers>
    <add name="DefaultSessionProvider" type="System.Web.Providers.DefaultSessionStateProvider, System.Web.Providers, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" connectionStringName="DefaultConnection" />
  </providers>
</sessionState>

Are you doing any threading (either System.Threading.Thread or Tasks) that may be adding items to the session state?
Answer : **No, Threading used. Task is used in core application for async action method call. **

[twsouthwick]

This is weird - I can see where it's throwing the exception but it appears to be guarded against. However, we can copy the keys over and it should fix it. I haven't been able to reproduce it - @darshan-edspl can you test it with a build once it's available?

Also, it appears you're using v1.x of a library that has a v2.0 available. Also, there's a newer set of libraries for session state that make use of async patterns that you may want to try: https://github.com/aspnet/AspNetSessionState

[darshan-edspl]

Thanks for the fix, really helpful.
Yes sure I can test it. Let me know once it is available.
And to test it I need to download the package again? Or What steps I need to follow?

[twsouthwick]

@darshan-edspl the fix is in. I'm probably 80% sure it'll fix your issue. Please give it a try from the CI feed (probably available later today or tomorrow) and let me know.

[darshan-edspl]

Thank you very much for fixing the issue quickly. I will check and provide an update here.