memory consumption for 1.9.0

[blumamir]

I am a maintainer of opentelemetry-js-contrib repo.
Recently, our browser tests started failing due to "JavaScript heap out of memory" when running webpack.

After doing some digging, I found that we are using this webpack loader @jsdevtools/coverage-istanbul-loader and it is causing the memory issues.

When I examined its dependencies, I see that it has a caret dependency on this package:

    "convert-source-map": "^1.7.0",

Which now automatically picks up version 1.9.0. It was released to npm on October 10 which is just when our CI started failing due to out of memory.

By comparing version 1.8.0 and 1.9.0, I found that it did some changes in decoding base64 which might be related.

I can see that you already published a new major version 2.0.0, but we cannot use it as this package is not a direct dependency for our project.

We will probably just increase our node memory in CI to work around this issue, but I wanted to document it here for anyone who might have the same issue and end up here.

Wondering if the change in #74 might have a memory leak or now consumes more memory than it used to?

[thlorenz]

Thanks for reporting this. 2.0.0 will most likely have a similar memory usage as the base64 change is in there as well.
@phated do you have ideas what specifically caused this memory increase and how we might improve this and backport to 1.x versions?

[phated]

None of the differences between 1.8 and 1.9 should have caused this. Did you check v1.8 instead of v1.9?

[phated]

Nevermind about 1.8, I thought that was the release with the lazy regexp. The only difference in v1.9 is that the Buffer.from API in nodejs is now being used instead of faking it with SafeBuffer. It is very possible that nodejs allocates more memory when using Buffer.from because it is an API with actual safety built-in.

That's not a change we would want to back out...

[blumamir]

Thank you for taking a look.
It might also be that we were on the edge with memory consumption in our tests and needed a minor push to cross the 2GB limit.

All I know is that on October 10 at noon our CI went from green to red with "JavaScript heap out of memory" with nothing that changed on the code itself. this is exactly when v1.9.0 is released. We are not using a lock file so the caret range picks new minor versions up automatically.
I don't know how much memory it was consuming previously (with v1.8.0) or how much it is using after (1.9.0), just that the 2GB limit is breached due to this change.

[blumamir]

@phated - this might be related to this bug in node.

Seems like moving to node's Buffer is now allocating the memory in a heap space that is not reclaimed by the GC, and only cleaned up by the event loop when the synchronous code finishes.
I assume that the running webpack code-coverage plugin is mostly synchronous, which means that each call consumes more and more memory which doesn't get the chance to be freed until the heap dies.