In .NET 7 large number of concurrent requests will memory leak #81056
Comments
|
@Cricle Are you seeing the memory usage go down eventually, or does it stay at 1G on .NET 7? |
It stay above 1G |
|
Are you running the application from Visual Studio? |
No, I compiled with AOT or docker image run, both like that. |
|
I tried to reproduce that situation again today. The computer has been restarted since the last test. |
|
Brief.zip https://hub.docker.com/r/hcricle/brief Here is compiled with AOT result |
|
Hey @Cricle, could you please clarify whether you notice the leak only when compiled as NativeAOT, or without it as well? |
Only AOT |
|
Ok thanks for clarifying. Adding @MichalStrehovsky @VSadov in case they are aware of anything related to this. |
|
I am not aware of any known leaks specific to NativeAOT. From the description it looks like this is on Windows10. |
|
If the managed memory analysis tools work, we could take a look at what it might be. |
Only PublishAOT but it will also. csproj only <PublishRelease>true</PublishRelease>Command dotnet publish -r win-x64 /p:PublishAot=trueRun debug in vs2022, no problem, stay 220mb |
|
@mangod9 Should this be transferred to the runtime repo? |
|
yeah its something in the runtime/nativeaot side I am guessing. |
|
I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label. |
ghost
added
the
|
Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas Issue DetailsIs there an existing issue for this?
Describe the bugWhen TFM=.NET7 When TFM=.NET6 The Code in program Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddSingleton(new HitokotoManager(Init()));
builder.Services.AddResponseCompression();
var app = builder.Build();
app.UseResponseCompression();
app.Use(async (HttpContext builder, RequestDelegate next) =>
{
//Write 1kb data in response
});Expected BehaviorMemory will not be much larger due to long concurrent requests Steps To ReproduceNo response Exceptions (if any)No response .NET Version7.0.100 Anything else?.NET SDK: 运行时环境: Host: .NET SDKs installed: .NET runtimes installed: Other architectures found: Environment variables: global.json file: Learn more: Download .NET:
|
|
I think I am able to reproduce this. At least I do see abnormal memory consumption on 7.0 with NativeAOT. |
|
I am not able to reproduce the problem with 8.0. Possibly the bug was fixed or maybe it is still there, but scenario no longer meets conditions for it to appear. Still need to figure what is going on on 7.0. (and to be sure it is indeed fixed in 8.0) |
|
With 7.0 I see CoreCLR stabilizing aroung 950 Mb commit size. With NativeAOT commit size grows to multiple gigabytes. Sometimes it stabilizes around 2Gb, sometimes it goes to 8Gb and still keeps going. There is something clearly wrong in 7.0 |
|
BTW. The app is using 455 threads total. This is on Windows10 machine with 32 logical cores. I assume 64 threads belong to GC and the rest 391 is the app threads. Same app on CoreCLR uses fewer threads (under 120 total). NativeAOT has a different threadpool on Windows, compared to CoreCLR, so we can't compare directly. The thread count is stable though, and should not cause memory retention. It is likely that the culprit is somewhere else. Edit: yes, the app uses roughly the same number of threads on 8.0 (around 430 threads), yet there is no "leak", so that is not the root cause. |
|
@jkotas - some input on how many threads an app can have. (re: #88772 (comment) ). |
|
One big difference on .net 7 is that native aot doesn't use regions. We enabled regions early in 8 but on 7 jit and aot differ in this. Not sure if that could matter but calling it out |
This seems unlikely. Regions may help with fragmentation in huge heaps, but unlikely to help with leaks. |
|
When comparing stacks that cause allocations the following stands out: The reson for this allocation is that The fat locks can in theory add up, but should not leak. Perhaps finalizer thread cannot keep up with reclaiming so many locks. |
|
A few more bits of info.
Basically - locks are eventually reclaimed, but it takes long enough time that millions of locks may be necessary to satisfy the demand of the app. The fat lock itself and its finalizer tracker are very small objects. Millions of locks by itself would not explain consuming gigabytes of memory. However, since the locks are logically attached to something else, they are an indication that some other data structures are not promptly destroyed. These locks protect instances of: I wonder if the cache holds to some sizeable object graphs for too long? |
|
There are 2 main dictionaries in the DI container implementation, the one that stores registrations (global) and one allocated per request to store scoped instances. The latter is likely what you’re seeing. It stores object graphs for scopes instances instantiated by the container. Interesting to see how expensive that lock is on native AOT |
|
Interestingly, disabling thin locks (so that every lock would allocate a lock object vs. alloc-free use of sync bits), and disabling regions (to rule out regions as a culprit) still does not make the bug to surface on 8.0 NativeAOT. Runtime churns through a lot of lock objects, but everything seems to be reclaimed orderly and promptly (as expected). |
On 8.0 NativeAOT the cost of a fat lock is the You'd typically need to go out of your way to make locks themselves a problem (before what they are protecting is already a problem). Also note that neither CoreCLR nor NativeAOT will create lock objects if they can get away with just using sync bits in the object header. Trivial noncontending or microcontending uses of However not having thin locks does not seem the root cause here. Taking out the feature on 8.0 does not cause the "leak" to appear. |
For the perspective on this - with thin locks enabled this app has fat lock watermark at 8 on NativeAOT 8.0 Compared to 66K of simultaneous lock instances needed otherwise, thin locks is a quite impressive optimization. |
|
Ok, all the stuff about locks can be ignored. It is interesting, but not the root cause here. The actual reason for the "leak" is simply because server GC gets really lazy for this scenario and does not collect very often. I see just one GC happening every few seconds and that sems to be Gen0.
public static Thread gcForcer = new Thread(
() =>
{
for (; ; )
{
// force a GC once a while. It does not need to be blocking.
GC.Collect(2, GCCollectionMode.Forced, blocking: false);
Thread.Sleep(10000);
}
}
);
public static async Task Main(string[] args)
{
gcForcer.Start();In a way GC is not completely wrong here - there is plenty of RAM on the machine and it can afford 8Gb heap, but the expectations are clearly that it collects more often. I am not sure what changed in 8.0 that this scenario no longer reproduces (regardless of regions turned on or off). @mangod9 - I think GC team needs to take a look. It is also possible that something changed in ASP.net/ Either way - something changed that caused GC to be less lazy and fixed this scenario in 8.0. Porting that change to 7.0, if possible, would be the likely fix. |
|
Tagging subscribers to this area: @dotnet/gc Issue DetailsIs there an existing issue for this?
Describe the bugWhen TFM=.NET7 When TFM=.NET6 The Code in program Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddSingleton(new HitokotoManager(Init()));
builder.Services.AddResponseCompression();
var app = builder.Build();
app.UseResponseCompression();
app.Use(async (HttpContext builder, RequestDelegate next) =>
{
//Write 1kb data in response
});Expected BehaviorMemory will not be much larger due to long concurrent requests Steps To ReproduceNo response Exceptions (if any)No response .NET Version7.0.100 Anything else?.NET SDK: 运行时环境: Host: .NET SDKs installed: .NET runtimes installed: Other architectures found: Environment variables: global.json file: Learn more: Download .NET:
|
|
On a 1GB RAM cloud Windows OS, the page file is not set. The available memory is about 400MB, |
|
I think this has already been resolved in net8.0. I try to compile AOT with net8.0, and use ab.exe to benchmark. The memory is normal. |
Thank you for checking! |
Is there an existing issue for this?
Describe the bug
When TFM=.NET7
wrk -t8 -c1000 -d30s --latency http://192.168.2.86:8889many times, memory will grow to 1G.When TFM=.NET6
wrk -t8 -c1000 -d30s --latency http://192.168.2.86:8889many times, memory only 145M.The Code in program
Expected Behavior
Memory will not be much larger due to long concurrent requests
Steps To Reproduce
No response
Exceptions (if any)
No response
.NET Version
7.0.100
Anything else?
.NET SDK:
Version: 7.0.100
Commit: e12b7af219
运行时环境:
OS Name: Windows
OS Version: 10.0.19045
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\7.0.100\
Host:
Version: 7.0.0
Architecture: x64
Commit: d099f07
.NET SDKs installed:
3.1.426 [C:\Program Files\dotnet\sdk]
5.0.303 [C:\Program Files\dotnet\sdk]
5.0.403 [C:\Program Files\dotnet\sdk]
5.0.408 [C:\Program Files\dotnet\sdk]
6.0.112 [C:\Program Files\dotnet\sdk]
6.0.307 [C:\Program Files\dotnet\sdk]
7.0.100 [C:\Program Files\dotnet\sdk]
.NET runtimes installed:
Microsoft.AspNetCore.All 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.19 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.31 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.10 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.0-preview.4.22251.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.19 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.30 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.31 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.0-preview.4.22229.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.1.19 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 3.1.30 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 3.1.31 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.10 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.11 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.12 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 6.0.11 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 6.0.12 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 7.0.0-preview.4.22229.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 7.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Other architectures found:
arm64 [C:\Program Files\dotnet]
registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\arm64\InstallLocation]
x86 [C:\Program Files (x86)\dotnet]
registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]
Environment variables:
Not set
global.json file:
Not found
Learn more:
https://aka.ms/dotnet/info
Download .NET:
https://aka.ms/dotnet/download
The text was updated successfully, but these errors were encountered: