Assembly.Load opens /dev/zero in linux and does not close it

[for7raid]

too many files open error in linux

After several hours of work web application on cent os 7 gets error "Too many files open"

General

After each request to my application the system opens and keep open a lot of /dev/zero files.
I run command lsof | grep zero | wc -l and looking out that count of files increase and increase.

System Info

OS LINUX X64 Linux 3.10.0-1062.4.1.el7.x86_64 dotnet/core#1 SMP Fri Oct 18 17:15:30 UTC 2019 Cent OS 7
.NET Core 3.0.0

[scalablecory]

Are you opening /dev/zero, or do you think it is being opened by something within .NET?

Can you share your code?

[for7raid]

We do not open /dev/zero stream in our code.
What I found out.

1. Clear (dotnet new mvc) application do not reproduce this issue (it seems to me)
2. We have custom fileproviders for razor and static files, and autofac with specific lifetime model. File providers use FileInfo, PhysicalFileInfo and FileWatchers.
3. It reproduce only when Razor processes the request. In case of Json result - everything is ok, static file is ok too

And it very strange that Razor asks and opens new stream of Shared files (like layout, viewimport etc) several times due one page.

We can not share the code, because it is very big corporate system. We try to determinate place in code. Will touch you on progress.

[carlossanlop]

We have custom fileproviders for razor and static files, and autofac with specific lifetime model. File providers use FileInfo, PhysicalFileInfo and FileWatchers.

Are you properly closing the file handles (disposing the file handling objects) after using them?

[for7raid]

After the long time of investigation we found out the reason.
We do not precompile views on publish and use runtime compilation, so .net tries find (I guess) *.views.dll, cant find it and redirect stream to /dev/zero and doesn't close it.

Steps to reproduce:

1. Create new clear mvc app dotnet new mvc
2. Add runtime view compilation nuget package Microsoft.AspNetCore.Mvc.Razor.RuntimeCompilation
3. Edit code in Startup.cs file

 services.AddControllersWithViews()
            .AddRazorRuntimeCompilation();

4. Edit project file to stop compile views on publish

  <PropertyGroup>
      <MvcRazorExcludeRefAssembliesFromPublish>false</MvcRazorExcludeRefAssembliesFromPublish>
      <PreserveCompilationContext>true</PreserveCompilationContext>
      <MvcRazorCompileOnPublish>false</MvcRazorCompileOnPublish>
  </PropertyGroup>

5. Publish app to folder with following parameters

Configuration: Release
Target Framework: netcoreapp3.0
Deployment mode: Framework-Dependent
Target Runtime: Portable

6. If publish still reproduces *.view.dll file, remove it, copy Views folder to publish folder
7. Copy publish folder to linux machine
8. Run in standart way dotnet {fileName}.dll
9. Open url in browser, wait while app compile views
10. Check opened /dev/zero files via lsof app lsof | grep zero | wc -l
11. Count of /dev/zero over the 0 items
12. Open another page
13. Count of /dev/zero increased

P.S. May be we should move this issue to asp repo?

[scalablecory]

Thanks, that's helpful.

@Pilchie can you help route

[Pilchie]

This issue was moved to dotnet/aspnetcore#17395

[for7raid]

Hi @Pilchie @carlossanlop
We think this is not asp issue, but core.
Assemly.Load opens /dev/zero stream

Here code to reproduce

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Reflection;
using System.Text;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;
using Microsoft.CodeAnalysis.Emit;
using Microsoft.CodeAnalysis.Text;

namespace TestAssemblyLoad
{
    class Program
    {
        private static readonly Encoding Encoding = Encoding.UTF8;
        private static readonly CSharpParseOptions ParseOptions = CSharpParseOptions.Default.WithLanguageVersion(LanguageVersion.Latest);
        private static readonly CSharpCompilationOptions CompilationOptions = new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary);
        private static readonly ConcurrentDictionary<string, MetadataReference> References = new ConcurrentDictionary<string, MetadataReference>();

        static Program()
        {
            var referencedAssemblies = Assembly.GetEntryAssembly().GetReferencedAssemblies();
            foreach (var referencedAssembly in referencedAssemblies)
                AddReference(referencedAssembly);
        }

        private static void AddReference(AssemblyName assemblyName)
        {
            var assembly = Assembly.Load(assemblyName);
            References.TryAdd(assembly.Location, MetadataReference.CreateFromFile(assembly.Location));

            var referencedAssemblyNames = assembly.GetReferencedAssemblies();
            foreach (var referencedAssemblyName in referencedAssemblyNames)
            {
                var referencedAssembly = Assembly.Load(referencedAssemblyName);
                References.TryAdd(referencedAssembly.Location, MetadataReference.CreateFromFile(referencedAssembly.Location));
            }
        }

        static void Main(string[] args)
        {
            Console.WriteLine("PID: " + Process.GetCurrentProcess().Id);
            Console.ReadKey();

            Run(@"using System;
public class SubProgram
{
    public static void Main()
    {
        Console.WriteLine(""Hello World!"");
    }
}");

            Console.WriteLine("The end...");
            Console.ReadKey();
        }

        public static void Run(string code)
        {
            var assemblyName = nameof(Assembly) + "_" + Guid.NewGuid().ToString("N");

            var syntaxTrees = new List<SyntaxTree>();
            var embeddedTexts = new List<EmbeddedText>();

            var filePath = $"{nameof(Assembly)}_file_{Guid.NewGuid():N}.cs";

            var buffer = Encoding.GetBytes(code);

            var sourceText = SourceText.From(buffer, buffer.Length, Encoding, canBeEmbedded: true);
            var syntaxTree = CSharpSyntaxTree.ParseText(sourceText, ParseOptions, filePath);
            var embeddedText = EmbeddedText.FromSource(filePath, sourceText);

            syntaxTrees.Add(syntaxTree);
            embeddedTexts.Add(embeddedText);

            var compilation = CSharpCompilation.Create(
                assemblyName,
                syntaxTrees: syntaxTrees,
                references: References.Values,
                options: CompilationOptions);

            var emitOptions = new EmitOptions(
                debugInformationFormat: DebugInformationFormat.PortablePdb,
                pdbFilePath: Path.ChangeExtension(assemblyName, "pdb"));

            using var stream = new MemoryStream();
            using var symbolsStream = new MemoryStream();

            var emitResult = compilation.Emit(stream, symbolsStream, embeddedTexts: embeddedTexts, options: emitOptions);

            if (emitResult.Success)
            {
                stream.Seek(0, SeekOrigin.Begin);
                symbolsStream.Seek(0, SeekOrigin.Begin);
            }

            var streamArr = stream.ToArray();
            var symbolsStreamArr = symbolsStream.ToArray();

            var assembly = Assembly.Load(streamArr, symbolsStreamArr);
            var type = assembly.GetType("SubProgram");
            var method = type.GetMethod("Main");
            method.Invoke(null, null);

            //stream.Dispose();
            //symbolsStream.Dispose();
        }
    }
}

after line var assembly = Assembly.Load(streamArr, symbolsStreamArr); count of /dev/zero stream increased.

We deepened to code inside the framework, file \src\System.Private.CoreLib\src\System\Runtime\Loader\AssemblyLoadContext.CoreCLR.cs has method

internal unsafe Assembly InternalLoad(ReadOnlySpan<byte> arrAssembly, ReadOnlySpan<byte> arrSymbols){
...
	fixed (byte* ptrAssembly = arrAssembly, ptrSymbols = arrSymbols)
	{
		LoadFromStream(_nativeAssemblyLoadContext, new IntPtr(ptrAssembly), arrAssembly.Length,
			new IntPtr(ptrSymbols), arrSymbols.Length, JitHelpers.GetObjectHandleOnStack(ref loadedAssembly));
	}
...
}

and dll import

[DllImport(JitHelpers.QCall, CharSet = CharSet.Unicode)]
private static extern IntPtr LoadFromStream(IntPtr ptrNativeAssemblyLoadContext, IntPtr ptrAssemblyArray, int iAssemblyArrayLen, IntPtr ptrSymbols, int iSymbolArrayLen, ObjectHandleOnStack retAssembly);

The LoadFromStream opens IntPtr and does not close it.
May be it should be void, like

[DllImport(JitHelpers.QCall, CharSet = CharSet.Unicode)]
private static extern void LoadFromPath(IntPtr ptrNativeAssemblyLoadContext, string? ilPath, string? niPath, ObjectHandleOnStack retAssembly);

AspRuntimeCompilation loads assembly with the same way while processing and compile views.

[BineG]

Another code snippet to observe the issue. Memory consumption just keeps climbing.

Also, when calling context.Unload() I get an ExecutionEngineException?

`

class Program
{
static void Main(string[] args)
{
AssemblyExecutor executor = new AssemblyExecutor();

        for (int i = 0; i < 1000; i++)
        {
            var assemblyPath = Path.Combine(Directory.GetCurrentDirectory(), "..", "..", "..", "..", "Assembly", "DotNetTestImplementation.dll");
            executor.Execute(assemblyPath);

            Console.WriteLine($"Iteration {i} completed");

            if (i > 0 && i % 10 == 0)
            {
                executor.Clear();

                GC.Collect();
                GC.WaitForPendingFinalizers();

                Console.WriteLine("GC called");
            }
        }
    }

   
}

public class AssemblyExecutor
{
    private ConcurrentDictionary<string, WeakReference> references = new ConcurrentDictionary<string, WeakReference>();

    [MethodImpl(MethodImplOptions.NoInlining)]
    public void Clear()
    {
        WeakReference wr = references.FirstOrDefault().Value;
        if (wr == null)
            return;

        //(wr.Target as CollectibleAssemblyLoadContext)?.Unload();
    }

    public void Execute(string assemblyPath)
    {
        var wr = references.AddOrUpdate(
                    assemblyPath,
                    (k) =>
                    {
                        return CreateReference(assemblyPath);
                    },
                    (k, existingReference) =>
                    {
                        if (existingReference?.Target == null || !existingReference.IsAlive)
                        {
                            return CreateReference(assemblyPath);
                        }

                        return existingReference;
                    }
                );



        ExecuteAssembly(wr);
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    private void ExecuteAssembly(WeakReference wr)
    {
        CollectibleAssemblyLoadContext context = wr.Target as CollectibleAssemblyLoadContext;
        if (context == null)
            return;

        var assembly = context.Assemblies.FirstOrDefault(a => a.ExportedTypes.Any(t => t.Name == "TestEntry"));

        var types = assembly.GetTypes().ToList();

        var type = assembly.GetType("TestEntry");

        var greetMethod = type.GetMethod("Execute");

        var instance = Activator.CreateInstance(type);
        var result = greetMethod.Invoke(instance, new object[] { null, null });
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    private WeakReference CreateReference(string assemblyPath)
    {
        Console.WriteLine($"Creating new context");
        var context = new CollectibleAssemblyLoadContext(assemblyPath);
        using (var fs = new FileStream(assemblyPath, FileMode.Open, FileAccess.Read))
        {
            var assembly = context.LoadFromStream(fs);
        }

        return new WeakReference(context);
    }
}

public class CollectibleAssemblyLoadContext : AssemblyLoadContext
{
    private AssemblyDependencyResolver _resolver;

    public CollectibleAssemblyLoadContext(string path)
        : base(isCollectible: true)
    {
        _resolver = new AssemblyDependencyResolver(path);
    }

    protected override Assembly Load(AssemblyName assemblyName)
    {
        var deps = AssemblyLoadContext.Default;
        var res = deps.Assemblies.Where(d => d.FullName.Contains(assemblyName.Name)).ToList();
        if (res.Any())
        {
            return Assembly.Load(new AssemblyName(res.First().FullName));
        }

        string assemblyPath = _resolver.ResolveAssemblyToPath(assemblyName);
        if (assemblyPath == null)
            return null;

        return LoadFromAssemblyPath(assemblyPath);
    }
}

`

[mikhail-barg]

Is there any progress on this issue? We are using Assembly.Load to load results of Roslyn compilation for thousands of small assemblies, and things get really ugly with all FD pool is wasted on /dev/zero (

[mikhail-barg]

Would you mind please reopening this issue for it does not seem to be aspnet-related? We are not using aspnet, but are affected by this problem still

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[Pilchie]

@jeffschwMSFT - can you help route within runtime?

[ghost]

Tagging subscribers to this area: @vitek-karas, @agocke, @CoffeeFlux
See info in area-owners.md if you want to be subscribed.

Issue Details

---

too many files open error in linux

After several hours of work web application on cent os 7 gets error "Too many files open"

General

After each request to my application the system opens and keep open a lot of /dev/zero files.
I run command lsof | grep zero | wc -l and looking out that count of files increase and increase.

System Info

OS LINUX X64 Linux 3.10.0-1062.4.1.el7.x86_64 dotnet/core#1 SMP Fri Oct 18 17:15:30 UTC 2019 Cent OS 7
.NET Core 3.0.0

Author:	for7raid
Assignees:	Pilchie
Labels:	area-AssemblyLoader-coreclr, untriaged
Milestone:	-

[VSadov]

I can reproduce this. Yes LoadFromStream leaks dev\zero handles.
Mismatch in the qcall return type is the likely reason.

[VSadov]

Nope, while the return type seems incorrect, that is not the reason for the leak. Fixing that does no fix the problem.
We are leaking actual handles somewhere.

[VSadov]

Opening \dev\zero is currently by-design. It is used in the pattern that creates anonymous memory mappings without relying on MAP_ANONYMOUS. It is likely something that was done long ago for portability.

I think it is no longer needed and should be removed to avoid running into fd limits.

The main reason for the leak though is that assemblies loaded via Assembly.Load are not unloadable.
If you load a lot you will see resource consumption going up. That part is by-design.

[VSadov]

The following example, which uses unloadable context, does not have the issue.

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Reflection;
using System.Runtime.Loader;
using System.Text;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;
using Microsoft.CodeAnalysis.Emit;
using Microsoft.CodeAnalysis.Text;

namespace TestAssemblyLoad
{
    class Program
    {
        private static readonly Encoding Encoding = Encoding.UTF8;
        private static readonly CSharpParseOptions ParseOptions = CSharpParseOptions.Default.WithLanguageVersion(LanguageVersion.Latest);
        private static readonly CSharpCompilationOptions CompilationOptions = new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary);
        private static readonly ConcurrentDictionary<string, MetadataReference> References = new ConcurrentDictionary<string, MetadataReference>();

        static Program()
        {
            var referencedAssemblies = Assembly.GetEntryAssembly().GetReferencedAssemblies();
            foreach (var referencedAssembly in referencedAssemblies)
                AddReference(referencedAssembly);
        }

        private static void AddReference(AssemblyName assemblyName)
        {
            var assembly = Assembly.Load(assemblyName);
            References.TryAdd(assembly.Location, MetadataReference.CreateFromFile(assembly.Location));

            var referencedAssemblyNames = assembly.GetReferencedAssemblies();
            foreach (var referencedAssemblyName in referencedAssemblyNames)
            {
                var referencedAssembly = Assembly.Load(referencedAssemblyName);
                References.TryAdd(referencedAssembly.Location, MetadataReference.CreateFromFile(referencedAssembly.Location));
            }
        }

        static void Main(string[] args)
        {
            Console.WriteLine("PID: " + Process.GetCurrentProcess().Id);
            Console.ReadKey();

            Run(@"using System;
public class SubProgram
{
    public static void Main()
    {
        Console.Write(""."");
    }
}");

            Console.WriteLine("The end...");
            Console.ReadKey();
        }

        public static void Run(string code)
        {
            var assemblyName = nameof(Assembly) + "_" + Guid.NewGuid().ToString("N");

            var syntaxTrees = new List<SyntaxTree>();
            var embeddedTexts = new List<EmbeddedText>();

            var filePath = $"{nameof(Assembly)}_file_{Guid.NewGuid():N}.cs";

            var buffer = Encoding.GetBytes(code);

            var sourceText = SourceText.From(buffer, buffer.Length, Encoding, canBeEmbedded: true);
            var syntaxTree = CSharpSyntaxTree.ParseText(sourceText, ParseOptions, filePath);
            var embeddedText = EmbeddedText.FromSource(filePath, sourceText);

            syntaxTrees.Add(syntaxTree);
            embeddedTexts.Add(embeddedText);

            var compilation = CSharpCompilation.Create(
                assemblyName,
                syntaxTrees: syntaxTrees,
                references: References.Values,
                options: CompilationOptions);

            var emitOptions = new EmitOptions(
                debugInformationFormat: DebugInformationFormat.PortablePdb,
                pdbFilePath: Path.ChangeExtension(assemblyName, "pdb"));

            using var stream = new MemoryStream();
            using var symbolsStream = new MemoryStream();

            var emitResult = compilation.Emit(stream, symbolsStream, embeddedTexts: embeddedTexts, options: emitOptions);

            if (emitResult.Success)
            {
                stream.Seek(0, SeekOrigin.Begin);
                symbolsStream.Seek(0, SeekOrigin.Begin);
            }

            var streamArr = stream.ToArray();
            var symbolsStreamArr = symbolsStream.ToArray();

            var assembly = Assembly.Load(streamArr, symbolsStreamArr);
            for(int i= 0; i < 100000000; i++)
            {
                // THIS WOULD NOT UNLOAD !!!
                // Assembly.Load(streamArr, symbolsStreamArr);

                stream.Seek(0, SeekOrigin.Begin);
                symbolsStream.Seek(0, SeekOrigin.Begin);

                var alc = new AssemblyLoadContext(null, isCollectible: true);
                Assembly a = alc.LoadFromStream(stream, symbolsStream);

                ProcessStartInfo pi = new ProcessStartInfo();
                pi.UseShellExecute = true;
                pi.FileName = "/usr/bin/lsof";
                pi.Arguments = " /dev/zero";
                Process.Start(pi);

                Console.ReadKey();

                var type = assembly.GetType("SubProgram");
                var method = type.GetMethod("Main");
                method.Invoke(null, null);
                
                alc.Unload();

                GC.Collect();
                GC.WaitForPendingFinalizers();
                GC.Collect();
                GC.WaitForPendingFinalizers();
                GC.Collect();
                GC.WaitForPendingFinalizers();

                Process.Start(pi);
            }

            stream.Dispose();
            symbolsStream.Dispose();
        }
    }
}

[VSadov]

I think we still need to get rid of opening /dev/zero - just to avoid running into fd limits, which could be a relatively small number.

[VSadov]

fixed in #48140