FileSystemWatcher crash UNALIGNED_STACK_POINTER

[nuzolx]

Hi,

I have a dotnet core, 3.1.8, program with five FileWatcher.
They watch folders on a network filesystem.
With some load (200 - 250 files in total), the program crash unexpectedly.
These files arrive at the same time, moved by another process from another server.

In event viewer, I see something but that's not really relevant (for me?).

Faultinq module name: coreclr.dll, version: 470020.41105, time stamp: Ox5f3397ec
Exception code: Oxc0000005
Fault offset: Ox0000000000191dcb
Faultinq process id: Oxf7a0
Faultinq application start time: Ox01d686baa97d91f4

I have a un dump generated when the crash occur.
I'm not very comfortable with that.
So I try some stuff with WinDBG.

*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************

SYMSRV:  BYINDEX: 0x25D
         https://msdl.microsoft.com/download/symbols
         SOS_AMD64_AMD64_4.700.20.41105.dll
         5F3397EC56f000
SYMSRV:  UNC: C:\ProgramData\Dbg\sym\SOS_AMD64_AMD64_4.700.20.41105.dll\5F3397EC56f000\SOS_AMD64_AMD64_4.700.20.41105.dll - path not found
SYMSRV:  UNC: C:\ProgramData\Dbg\sym\SOS_AMD64_AMD64_4.700.20.41105.dll\5F3397EC56f000\SOS_AMD64_AMD64_4.700.20.41105.dl_ - path not found
SYMSRV:  UNC: C:\ProgramData\Dbg\sym\SOS_AMD64_AMD64_4.700.20.41105.dll\5F3397EC56f000\file.ptr - path not found
SYMSRV:  HTTPGET: /download/symbols/SOS_AMD64_AMD64_4.700.20.41105.dll/5F3397EC56f000/SOS_AMD64_AMD64_4.700.20.41105.dll
SYMSRV:  HttpQueryInfo: 80190194 - HTTP_STATUS_NOT_FOUND
SYMSRV:  HTTPGET: /download/symbols/SOS_AMD64_AMD64_4.700.20.41105.dll/5F3397EC56f000/SOS_AMD64_AMD64_4.700.20.41105.dl_
SYMSRV:  HttpQueryInfo: 80190194 - HTTP_STATUS_NOT_FOUND
SYMSRV:  HTTPGET: /download/symbols/SOS_AMD64_AMD64_4.700.20.41105.dll/5F3397EC56f000/file.ptr
SYMSRV:  HttpQueryInfo: 80190194 - HTTP_STATUS_NOT_FOUND
SYMSRV:  RESULT: 0x80190194
SYMSRV:  BYINDEX: 0x25E
         C:\debug\symbols
         SOS_AMD64_AMD64_4.700.20.41105.dll
         5F3397EC56f000
SYMSRV:  UNC: C:\debug\symbols\SOS_AMD64_AMD64_4.700.20.41105.dll\5F3397EC56f000\SOS_AMD64_AMD64_4.700.20.41105.dll - path not found
SYMSRV:  UNC: C:\debug\symbols\SOS_AMD64_AMD64_4.700.20.41105.dll\5F3397EC56f000\SOS_AMD64_AMD64_4.700.20.41105.dl_ - path not found
SYMSRV:  UNC: C:\debug\symbols\SOS_AMD64_AMD64_4.700.20.41105.dll\5F3397EC56f000\file.ptr - path not found
SYMSRV:  RESULT: 0x80070003
SYMSRV:  BYINDEX: 0x25F
         https://msdl.microsoft.com/download/symbols
         coreclr.dll
         5F3397EC56f000
SYMSRV:  PATH: C:\ProgramData\Dbg\sym\coreclr.dll\5F3397EC56f000\coreclr.dll
SYMSRV:  RESULT: 0x00000000
DBGHELP: C:\ProgramData\Dbg\sym\coreclr.dll\5F3397EC56f000\coreclr.dll - OK
SYMSRV:  BYINDEX: 0x260
         https://msdl.microsoft.com/download/symbols
         coreclr.dll
         5F3397EC56f000
SYMSRV:  PATH: C:\ProgramData\Dbg\sym\coreclr.dll\5F3397EC56f000\coreclr.dll
SYMSRV:  RESULT: 0x00000000
DBGHELP: C:\ProgramData\Dbg\sym\coreclr.dll\5F3397EC56f000\coreclr.dll - OK
Failed to request MethodData, not in JIT code range
MethodDesc:   00007ffc6ca07240
Method Name:          System.IO.FileSystemWatcher.ParseEventBufferAndNotifyForEach(Byte[])
Class:                00007ffc6c9f8a38
MethodTable:          00007ffc6ca073e0
mdToken:              000000000600005A
Module:               00007ffc6ca06520
IsJitted:             yes
Current CodeAddr:     00007ffc6d7a4980
Version History:
  ILCodeVersion:      0000000000000000
  ReJIT ID:           0
  IL Addr:            0000000000000000
     CodeAddr:           00007ffc6d7a4980  (OptimizedTier1)
     NativeCodeVersion:  000001E5EB703B40
     CodeAddr:           00007ffce5c3bc10  (ReadyToRun)
     NativeCodeVersion:  0000000000000000
SYMSRV:  BYINDEX: 0x261
         https://msdl.microsoft.com/download/symbols
         System.IO.FileSystem.Watcher.pdb
         FA68AFAB85EA4A57B6E929B2AB30C79D1
SYMSRV:  PATH: C:\ProgramData\Dbg\sym\System.IO.FileSystem.Watcher.pdb\FA68AFAB85EA4A57B6E929B2AB30C79D1\System.IO.FileSystem.Watcher.pdb
SYMSRV:  RESULT: 0x00000000
Source file:  /_/src/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.Win32.cs @ 249
DBGHELP: coreclr is not source indexed

KEY_VALUES_STRING: 1

    Key  : AV.Dereference
    Value: NullPtr

    Key  : AV.Fault
    Value: Read

    Key  : Analysis.CPU.mSec
    Value: 2749

    Key  : Analysis.DebugAnalysisProvider.CPP
    Value: Create: 8007007e on xxx

    Key  : Analysis.DebugData
    Value: CreateObject

    Key  : Analysis.DebugModel
    Value: CreateObject

    Key  : Analysis.Elapsed.mSec
    Value: 3297

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 502

    Key  : Analysis.System
    Value: CreateObject

    Key  : CLR.Engine
    Value: CORECLR

    Key  : CLR.Version
    Value: 4.700.20.41105

    Key  : Timeline.OS.Boot.DeltaSec
    Value: 10334662

    Key  : Timeline.Process.Start.DeltaSec
    Value: 73117

    Key  : WER.OS.Branch
    Value: rs5_release

    Key  : WER.OS.Timestamp
    Value: 2018-09-14T14:34:00Z

    Key  : WER.OS.Version
    Value: 10.0.17763.1

    Key  : WER.Process.Version
    Value: 1.0.0.0


ADDITIONAL_XML: 1

OS_BUILD_LAYERS: 1

NTGLOBALFLAG:  0

PROCESS_BAM_CURRENT_THROTTLED: 0

PROCESS_BAM_PREVIOUS_THROTTLED: 0

APPLICATION_VERIFIER_FLAGS:  0

CONTEXT:  00007ffd20f787d1 -- (.cxr 0x7ffd20f787d1)
rax=480000021e850f0f rbx=8341d23300000086 rcx=894800000098838b
rdx=850f00007d830845 rsi=0327840ff8b54148 rdi=483cfe24068a0000
rip=0ffda817418d0000 rsp=000003e6830f02fd rbp=3e80d68b48ff3345
 r8=0a8a00000341840f  r9=0f583cc52241c18a r10=4cc18a0000052884
r11=0f403cf02401428d r12=f2f9800000022484 r13=418d000a293c840f
r14=000282860f013c3e r15=0565840ff3f98000
iopl=2         nv dn di ng nz na pe cy
cs=2b5b  ss=1065  ds=000a  es=8b44  fs=0870  gs=8b4c             efl=00006583
0ffda817`418d0000 ??              ???
Resetting default scope

EXCEPTION_RECORD:  00000022d55bd850 -- (.exr 0x22d55bd850)
ExceptionAddress: 0000000000000000
   ExceptionCode: 0000000b
  ExceptionFlags: 00000000
NumberParameters: 0

PROCESS_NAME:  xxx.exe

READ_ADDRESS:  0000000000000000 

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.

EXCEPTION_CODE_STR:  c0000005

FAULTING_THREAD:  ffffffff

IP_ON_HEAP:  0ffda817418d0000
The fault address in not in any loaded module, please check your build's rebase
log at <releasedir>\bin\build_logs\timebuild\ntrebase.log for module which may
contain the address if it were loaded.

UNALIGNED_STACK_POINTER:  000003e6830f02fd

STACK_TEXT:  
00000022`d55bc8a0 00007ffc`cc46789f coreclr!EEPolicy::HandleFatalError+0x8b
00000022`d55bceb0 00007ffc`cc381de6 coreclr!ProcessCLRException+0x148c86
00000022`d55bd0b0 00007ffd`2101478f ntdll!RtlpExecuteHandlerForException+0xf
00000022`d55bd0e0 00007ffd`20f74bef ntdll!RtlDispatchException+0x40f
00000022`d55bd810 00007ffd`20f789e6 ntdll!RtlRaiseException+0x316
00000022`d55be010 00007ffd`1d369709 KERNELBASE!RaiseException+0x69
00000022`d55be0f0 00007ffc`cc336619 coreclr!NakedThrowHelper2+0x9
00000022`d55be120 00007ffc`cc33664b coreclr!NakedThrowHelper_RspAligned+0x1e
00000022`d55be648 00007ffc`cc336655 coreclr!NakedThrowHelper_FixRsp+0x5
00000022`d55be650 00007ffc`6d7a49d4 unknown!unknown+0x0
00000022`d55be658 00007ffd`2100f874 ntdll!NtRemoveIoCompletion+0x14
00000022`d55be660 00007ffd`20fc9870 ntdll!RtlSetLastWin32Error+0x40


STACK_COMMAND:  .ecxr ; kb ; ** Pseudo Context ** Pseudo ** Value: 207fc8fb430 ** ; kb

FAULTING_SOURCE_LINE:  f:\workspace\_work\1\s\src\vm\eepolicy.cpp

FAULTING_SOURCE_FILE:  f:\workspace\_work\1\s\src\vm\eepolicy.cpp

FAULTING_SOURCE_LINE_NUMBER:  1177

FAULTING_SOURCE_CODE:  
No source found for 'f:\workspace\_work\1\s\src\vm\eepolicy.cpp'


SYMBOL_NAME:  coreclr!EEPolicy::HandleFatalError+8b

MODULE_NAME: coreclr

IMAGE_NAME:  coreclr.dll

FAILURE_BUCKET_ID:  NULL_POINTER_READ_c0000005_coreclr.dll!EEPolicy::HandleFatalError

OS_VERSION:  10.0.17763.1

BUILDLAB_STR:  rs5_release

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

IMAGE_VERSION:  4.700.20.41105

FAILURE_ID_HASH:  {14f01c38-e318-9c7b-c758-0ddeb36df05a}

Followup:     MachineOwner
---------

DBGHELP: coreclr is not source indexed

And also with dotnet-dump.

> clrstack
OS Thread Id: 0xfd4c (27)
        Child SP               IP Call Site
00000022D55BE150 00007ffccc46789f [FaultingExceptionFrame: 00000022d55be150]
00000022D55BE650 00007FFC6D7A49D4 System.IO.FileSystemWatcher.ParseEventBufferAndNotifyForEach(Byte[]) [/_/src/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.Win32.cs @ 249]
00000022D55BE6F0 00007FFC6D7A48E6 System.IO.FileSystemWatcher.ReadDirectoryChangesCallback(UInt32, UInt32, System.Threading.NativeOverlapped*) [/_/src/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.Win32.cs @ 242]
00000022D55BE750 00007FFC6D6F189C System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs @ 201]
00000022D55BE7C0 00007FFC6D7359B5 System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*) [/_/src/System.Private.CoreLib/src/System/Threading/Overlapped.cs @ 59]
00000022D55BE8F0 00007ffccc336ba3 [GCFrame: 00000022d55be8f0]
00000022D55BEAB0 00007ffccc336ba3 [DebuggerU2MCatchHandlerFrame: 00000022d55beab0]
> pe
Exception object: 000001e580001198
Exception type:   System.ExecutionEngineException
Message:          <none>
InnerException:   <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 80131506

This crash occur in a production server. At this time, I have no smalls repro steps.
If it's revelant, I can share the dumpfile.

[GrabYourPitchforks]

@carlossanlop Heads up - if this ends up being a real issue it might be a candidate for downlevel (2.1, 3.1, 5.x) servicing.

[nuzolx]

The filewatchers initialization:

private void InitializeInnerFilewatcher(List<string> filters)
{
        _watcher = new FileSystemWatcher(WatchPath);
        _watcher.InternalBufferSize = 65536;
        if (filters.Count > 1)
        {
            _watcher.Filter = FILTER_ALL; // *.*
            _customFilters = filters;
        }
        else
            _watcher.Filter = filters.First();
        _watcher.NotifyFilter = NotifyFilters.LastWrite | NotifyFilters.FileName;
        _watcher.Changed += new FileSystemEventHandler(FileCreatedOrChanged);
        _watcher.Created += new FileSystemEventHandler(FileCreatedOrChanged);
        _watcher.Renamed += new RenamedEventHandler(FileRenamed);
        _watcher.Error += Watcher_Error;
        _watcher.EnableRaisingEvents = true;
}

And here we have, the "process" part for each event triggered by the filewatcher:

private void TryHandle(FileSystemEventArgs arg)
{
        if (!File.Exists(arg.FullPath))
            return;

        if (!_customFilters.Any() || _customFilters.Any(x => PatternMatcher.MatchPattern(x, arg.Name)))
            _memoryCache.AddOrGetExisting(arg.FullPath, arg, _cacheItemPolicy);
 }

I tried to avoid any real process on triggered file system events, so I push the file path in the memoryCache and later I send it to a ServiceBus queue for processing the file by any consumer.

[danmoseley]

Similar? #40412 (but that is x64 not ARM64)

[danmoseley]

@nuzolx is it feasible to try with 5.0? We probably didn't change relevant library code, but it's conceivable it's runtime related.

Also, I see this is ARM64. Do you use x64 - is it possible to say whether it happens with that?

Is it possible to run with pageheap enabled (#40412 (comment)) .. this will be very slow, so would have to be in a test instance?

[nuzolx]

@danmosemsft « Also, I see this is ARM64 ». You say that about some stuff in the windbg loading section? I think this is misinterpreted. This program is publish only for win-x64 runtime (and self-contained option).

As you, I see the #40412 and the potential link.
We contacted support as you suggest. With their help we may have a re-production case today.
I this happens, I will try to upgrade, enable pageheap and push results here.

[danmoseley]

I misread AMD64...an old term

[jeffhandley]

Closing this issue as a duplicate of #40412. We are still working to get this fix in place for 5.0 in #42420 and to backport to 3.1 in dotnet/corefx#42989.