Blazor WASM does not free memory - possibly a memory leak

[jahnotto]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

When allocating large arrays in Blazor wasm 6.0.1, the memory is never freed up when the reference is set to null. Consecutive allocations ends up with an OutOfMemoryException.

Either I'm doing something very wrong, or there is a memory leak.

Expected Behavior

The GC should free the memory that is no longer used, and should not result in an OutOfMemoryException.

Steps To Reproduce

Minimal repro project is here: https://github.com/jahnotto/BlazorWasmMemoryIssue

Try to click multiple times on "Allocate DotNet memory". The memory keeps increasing. Clicking "Free DotNet memory" does not make any difference. Same behavior is seen in debug, release and even AOT.

Allocating JS memory, on the other hand, works as expected.

Exceptions (if any)

OutOfMemoryException

.NET Version

6.0.101

Anything else?

Blazor wasm 6.0.1
Tested on both Edge and Chrome

Note: GCSettings.GCLatencyMode is set to Batch in Blazor wasm and can't be changed. I don't know if this relevant, but I'd expect it to be Interactive for an interactive client side application.

[ghost]

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

When allocating large arrays in Blazor wasm 6.0.1, the memory is never freed up when the reference is set to null. Consecutive allocations ends up with an OutOfMemoryException.

Either I'm doing something very wrong, or there is a memory leak.

Expected Behavior

The GC should free the memory that is no longer used, and should not result in an OutOfMemoryException.

Steps To Reproduce

Minimal repro project is here: https://github.com/jahnotto/BlazorWasmMemoryIssue

Try to click multiple times on "Allocate DotNet memory". The memory keeps increasing. Clicking "Free DotNet memory" does not make any difference. Same behavior is seen in debug, release and even AOT.

Allocating JS memory, on the other hand, works as expected.

Exceptions (if any)

OutOfMemoryException

.NET Version

6.0.101

Anything else?

Blazor wasm 6.0.1
Tested on both Edge and Chrome

Note: GCSettings.GCLatencyMode is set to Batch in Blazor wasm and can't be changed. I don't know if this relevant, but I'd expect it to be Interactive for an interactive client side application.

Author:	jahnotto
Assignees:	-
Labels:	area-GC-coreclr, untriaged
Milestone:	-

[mangod9]

hi @SamMonoRT could you please suggest what area this could be moved under?

[ghost]

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

When allocating large arrays in Blazor wasm 6.0.1, the memory is never freed up when the reference is set to null. Consecutive allocations ends up with an OutOfMemoryException.

Either I'm doing something very wrong, or there is a memory leak.

Expected Behavior

The GC should free the memory that is no longer used, and should not result in an OutOfMemoryException.

Steps To Reproduce

Minimal repro project is here: https://github.com/jahnotto/BlazorWasmMemoryIssue

Try to click multiple times on "Allocate DotNet memory". The memory keeps increasing. Clicking "Free DotNet memory" does not make any difference. Same behavior is seen in debug, release and even AOT.

Allocating JS memory, on the other hand, works as expected.

Exceptions (if any)

OutOfMemoryException

.NET Version

6.0.101

Anything else?

Blazor wasm 6.0.1
Tested on both Edge and Chrome

Note: GCSettings.GCLatencyMode is set to Batch in Blazor wasm and can't be changed. I don't know if this relevant, but I'd expect it to be Interactive for an interactive client side application.

Author:	jahnotto
Assignees:	-
Labels:	arch-wasm, area-GC-coreclr, untriaged
Milestone:	-

[SamMonoRT]

@BrzVlad @lewing @vargaz

[ghost]

Tagging subscribers to this area: @BrzVlad
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

When allocating large arrays in Blazor wasm 6.0.1, the memory is never freed up when the reference is set to null. Consecutive allocations ends up with an OutOfMemoryException.

Either I'm doing something very wrong, or there is a memory leak.

Expected Behavior

The GC should free the memory that is no longer used, and should not result in an OutOfMemoryException.

Steps To Reproduce

Minimal repro project is here: https://github.com/jahnotto/BlazorWasmMemoryIssue

Try to click multiple times on "Allocate DotNet memory". The memory keeps increasing. Clicking "Free DotNet memory" does not make any difference. Same behavior is seen in debug, release and even AOT.

Allocating JS memory, on the other hand, works as expected.

Exceptions (if any)

OutOfMemoryException

.NET Version

6.0.101

Anything else?

Blazor wasm 6.0.1
Tested on both Edge and Chrome

Note: GCSettings.GCLatencyMode is set to Batch in Blazor wasm and can't be changed. I don't know if this relevant, but I'd expect it to be Interactive for an interactive client side application.

Author:	jahnotto
Assignees:	-
Labels:	arch-wasm, untriaged, area-GC-mono
Milestone:	-

[BrzVlad]

I'm not convinced this is a leak, simply because the array size is very large relative to the maximum memory size. The GC is not really deterministic: objects can sometimes remain pinned and not get collected, the GC can therefore increase the trigger limit for next collection and the memory usage can spiral out of control.

@jahnotto Could you please test the behavior with a smaller array size like 128MB ? If you do iterations of Alloc/Free would you ever encounter unbounded memory growth ?

[Fabi]

It's an issue in server and client side blazor. Even trying to force collecting memory won't work. it's all referenced somewhere in native memory behind the scenes. This is expected behavior according to all these MS guys but it should NOT happen. For months users are just getting blamed to wait for the GC to collect memory because stuff gets freed very late or never at some point or when you run out of memory. This should get fixed ASAP!

[jahnotto]

Thanks for following up @BrzVlad . I have done some more testing.

1. Allocating 128 MB instead of 512 MB seems to work better. The memory keeps increasing, but only up to a certain point. I'm not getting an exception anymore.
2. I have added a WPF test project in the repro project which is doing exactly the same thing (with 512 MB allocations). Here you can see that the GC is behaving very differently than in Blazor:

• Memory is increasing until a certain point, but does not result in an exception
• If I call GC.Collect(), the memory is freed. This seems to have no effect in Blazor
• GCSettings.LatencyMode is Interactive, as opposed to Batch in Blazor.

Edit @BrzVlad Not sure what you mean by "iterations of alloc/free"? I'm clicking allocate and free multiple times. With big arrays, memory consumption grows and end up with exception on Blazor, but not in WPF. Both are .NET 6.

[lambdageek]

In the sample _memoryUsed is populated by calling window.performance.memory.usedJSHeapSize in JS

I'm not sure if we shrink the .NET heap after we do a Mono GC and return that memory to JS. Assuming we don't, I wouldn't expect to see the JS heap size decrease after a mono collection. /cc @lewing

[BrzVlad]

@jahnotto On WASM the available memory is much lower, and 512MB is a significant portion of it. I don't think it is a fair comparison. So if any such huge object gets pinned the memory can quickly increase. I'm assuming this is the reason why it works with 128MB. It's unclear to me if the used memory is excessive or not. Also WPF and WASM use completely different GCs with different heuristics and capabilities. It would be useful to see some logs of the application with MONO_LOG_LEVEL=debug and MONO_LOG_MASK=gc enabled. @lewing Do we still have a way to enable this log ?

[jahnotto]

I see your point. Is there anything I can do somehow to free the memory?

[lewing]

It sounds like you are running into limitations of the current wasm memory model, see the discussion here WebAssembly/design#1397

[ilonatommy]

@maraf, can we do something to help here?

[jahnotto]

This was tagged with "milestone 7.0.0", whatever that means. Are there any changes to the Blazor memory model in .NET 7?

[Mindorf]

Agree, this seems to be an open issue. We are observing something like this as well.

[MikeReedGH]

Hi,

Is there any update on this issue?

We are running a .Net 7 Blazor Wasm app and having this exact issue. It is crippling our production deployment.
We either get an out of memory exception or the browser just force restarts the application.

Does anybody have a workaround or fix for this?

Thank you

[BrzVlad]

I think this issue is not actionable, not sure why it is open. The original submitted sample has allocation patterns that are not really relevant for real world applications. I think this issue might as well be closed.

Applications can run out of memory from various reason, including excessive memory consumption at the application level, especially given wasm has less available memory. Also mono gc can have somewhat increased memory consumption because we are conservatively keeping alive more objects than we could in theory, when we scan roots from stack for example. If you think your application is not allocating excessive memory or leaks memory and it's the GC's fault that it fails to collect it, then I would encourage you to submit a new issue with a sample reproduction app. Otherwise this is not currently actionable.

[jahnotto]

Our application is very much a real-world application, although the original submitted sample was simplified to isolate the problem. Allocating arrays of more than 200 MB (100 MB in some cases) causes the GC to never collect it, even when there are no references to it. It is causing major issues for us.

[BrzVlad]

It is expected for a ref to an object to linger around a little bit more on the stack after the object becomes logically dead, keeping the object still allocated. It is unclear to me if you are experiencing a few objects just living a bit longer, or if they are incorrectly pinned throughout the application leading to OOM. I can't tell or investigate without a proper sample application.

Also you could prevent this by reusing large arrays, which is the recommended approach anyway since large objects are allocated directly to the major generation, their allocation is expensive and problematic when the memory becomes fragmented.

[MikeReedGH]

@BrzVlad . Thanks for replying.

The problem comes as soon a user selects a photo/or takes a camera photo, then the WASM app just immediately crashes and restarts. It's not a silly bug in my handling of the IBrowserFile. I have quadrupled checked.
It happens seemingly randomly, it will work fine for hours and then suddenly just crash.
It just smells like a memory leak, and I really thought I had found it when reading about this bug.
I am running out of ideas :(

Thanks

[BrzVlad]

I would need some form of repro. Even if it is not always crashing, but exhibiting some high memory spikes that shouldn't happen.

[jahnotto]

It is expected for a ref to an object to linger around a little bit more on the stack after the object becomes logically dead, keeping the object still allocated. It is unclear to me if you are experiencing a few objects just living a bit longer, or if they are incorrectly pinned throughout the application leading to OOM. I can't tell or investigate without a proper sample application.

Also you could prevent this by reusing large arrays, which is the recommended approach anyway since large objects are allocated directly to the major generation, their allocation is expensive and problematic when the memory becomes fragmented.

Yep, that's the nature of the gc! However, any array larger than 100 MB is completely ignored by the gc forever. It seems like the data is never recollected, even after using the application for an hour. Eventually it will crash with an OutOfMemoryException.

We have tried reusing the arrays using a pool. However, the the number and size of the arrays will vary, and eventually new arrays that are bigger than arrays that are already in the pool will be required.

Even worse, even memory allocated by the framework itself is never reclaimed in many cases. This happens e.g. when in HttpClient's buffer when downloading large files (100+ MB).

I'll be happy to provide a repo, but I'm unsure exactly what you need. The original repo that I provided reproduces the exact problem.

[BrzVlad]

Overall these issues seem to be caused by the runtime conservatively scanning the entire stack and pinning all objects that have refs lingering on the stack, even if the objects might have been logically dead. Historically mono GC has always been conservative with scanning roots from regs/stack. On wasm-AOT it is particularly difficult to implement due to arch limitations and it is unlikely to happen any time soon. In the interpreted mode it is doable and a first step in this direction was done for .net 9 #100400