Access Violation Exception after upgrading from .NET 8 to .NET 9

[karmeli87]

Description

We recently upgraded from .NET 8 to .NET 9 and start to experience crashes in our tests

Faulting application name: testhost.exe, version: 17.1200.24.56501, time stamp: 0x63740000
Faulting module name: coreclr.dll, version: 9.0.24.52809, time stamp: 0x672049fc
Exception code: 0xc0000005
Fault offset: 0x00000000002eea8b
Faulting process id: 0x2c8c
Faulting application start time: 0x01db4ea81cad46d8
Faulting application path: c:\Jenkins\workspace\Testsv70_Winx64\s\test\SlowTests\bin\Release\net9.0\testhost.exe
Faulting module path: C:\Program Files\dotnet\shared\Microsoft.NETCore.App\9.0.0\coreclr.dll
Report Id: 3804b1ee-81a8-4f06-bef5-760530d90989
Faulting package full name: 
Faulting package-relative application ID: 

Application: testhost.exe
CoreCLR Version: 9.0.24.52809
.NET Version: 9.0.0
Description: The process was terminated due to an unhandled exception.
Stack:
   at System.Runtime.ExceptionServices.InternalCalls.RhpCallFinallyFunclet(Byte*, Void*, ExInfo*)
   at System.Runtime.ExceptionServices.InternalCalls.RhpCallFinallyFunclet(Byte*, Void*, ExInfo*)
   at System.Runtime.EH.InvokeSecondPass(ExInfo ByRef, UInt32, UInt32)
   at System.Runtime.EH.DispatchEx(System.Runtime.StackFrameIterator ByRef, ExInfo ByRef)
   at System.Runtime.EH.RhRethrow(ExInfo ByRef, ExInfo ByRef)
   at Raven.Server.Documents.Replication.Incoming.AbstractIncomingReplicationHandler`2[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, 
Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, 
PublicKeyToken=7cec85d7bea7798e]].ReceiveReplicationBatches()
   at Raven.Server.Documents.Replication.Incoming.AbstractIncomingReplicationHandler`2[[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, 
Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, 
PublicKeyToken=7cec85d7bea7798e]].DoIncomingReplication()
   at Raven.Server.Utils.PoolOfThreads+PooledThread.DoWork()
   at Raven.Server.Utils.PoolOfThreads+PooledThread.Run()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

Will attach the dump privately.
https://developercommunity.visualstudio.com/t/Access-Violation-Exception-after-upgradi/10814326

Reproduction Steps

Not at the moment.

Expected behavior

not to crash with an AVE

Actual behavior

crashing with an AVE

Regression?

Yes, this was working without crashing on .NET 8

Known Workarounds

No response

Configuration

No response

Other information

No response

[janvorli]

It looks like a GC hole. Could you please share one more dump from a crash so that I can get more confidence on that? It would be great to get a dump with enabled stresslog which would allow me to see a recent history of GC stack walks / relocations. To do that, please set the following env variables:

DOTNET_StressLog=1 
DOTNET_LogLevel=7 
DOTNET_LogFacility=80103 
DOTNET_StressLogSize=2000000 
DOTNET_TotalStressLogSize=40000000

The stresslog is an in-memory log that gets included in the crash dump and can be dumped via the !stresslog command in WinDbg.

[karmeli87]

Hi,

Thanks for looking into that, we will set those env variables and wait for the next dump.

Since the upgrade to .NET 9, we get almost every test run on linux a seg fault, while on windows we get only once a day or two the above AVE.

If you like we can also share the linux dumps, but I don't want to mix several issues in one ticket and prefer to tackle one bug at a time.

But if you believe it can shed more light into current issue we can share those dumps as well.

[janvorli]

@karmeli87 It would be great to get linux dumps as well. Chances are it is the same issue.

[karmeli87]

I have uploaded the linux dump here https://developercommunity.visualstudio.com/t/Access-Violation-Exception-after-upgradi/10814326#T-N10815576

[janvorli]

The Linux crash is a completely different thing. It looks like a JIT bug. Can you please create a separate issue for it? I've shared the call stack and crash reason in this gist: https://gist.github.com/janvorli/e9ce073e24f58ccf1629e3cfa0f6c414, so you can cut and paste it into the issue description.

[jkotas]

https://gist.github.com/janvorli/e9ce073e24f58ccf1629e3cfa0f6c414, so you can cut and paste it into the issue description.

This looks like a duplicate of #109981

[karmeli87]

According to #109981 (comment) setting the following DOTNET_JitEnableInductionVariableOpts=0 should workaround this issue.

We will try that and I report back once we will experience another crash.

[karmeli87]

I have shared another linux dump here https://developercommunity.visualstudio.com/t/Access-Violation-Exception-after-upgradi/10814326#T-N10815576

with those env var set

DOTNET_JitEnableInductionVariableOpts=0

DOTNET_StressLog=1 
DOTNET_LogLevel=7 
DOTNET_LogFacility=80103 
DOTNET_StressLogSize=2000000 
DOTNET_TotalStressLogSize=40000000

Please let me know if should open a separate issue for those crashes.

[janvorli]

@karmeli87 the crash is exactly the same as the one before (linux).