Build: avoid leaking memory from generate_plugins_version

[kares]

What does this PR do?

This PR handles memory leaks that exhibit when running the rake/gradle task.

NOTE: this is mostly a partial understanding what's going on by simply following the "huge" objects from memory dumps.
There might be better ways to handle the issue - without the patches (e.g. rewrite plugin generation to put each bundle resolution into a disposable runtime or run as a process).

Details on why each patch is relevant are to be found in code comments.

Why is it important/What is the impact to the user?

To be able to generate plugin docs we need to check available plugin versions.

How to test this PR locally

./gradlew generatePluginsVersion -Dorg.gradle.jvmargs="-XX:+HeapDumpOnOutOfMemoryError -Xmx2g"
should no longer end up with an out of memory error

Tested with Bundler 2.2.14 as well as latest 2.2.15

Related issues

resolves #12758
related: #10942

[yaauie]

Two very good finds. I've added a comment about a maybe-safer interception of the worker pool size. Feel free to accept or reject the suggestion.

[yaauie]

This is a really good find.

I'm a bit wary of the Bundler::Fetcher::CompactIndex#bundle_worker implementation changing out of under us -- if it were to perform an operation that had a side-effect of starting the worker threads, our interception here would be too late to make a difference.

An alternate approach would be to patch Bundler::Worker to cap the size of the pool, something like:

Worker.class_eval do
  alias_method :__initialize, :initialize
  def initialize(size, name, func)
    if size > 2
      $stderr.puts("Bundler::Worker: initializing limited worker pool (size: 2; requested: #{size})"
      size = 2
    end
    __initialize(size, name, func)
  end
end

[kares]

Thanks, I understand the concern and have thought about patching the Worker as well.
The worker was introduced 4.5+ years ago with Bundler 2.2.
(has bee used previously but existed inline - there wasn't a bundle_worker method here).

Would say the main use-case for Bundler::Worker is the parallel installation (from the installer), this can be disabled using the API if we choose to (command line --jobs option).

That being said I would rather not cause future surprises by affecting bundle install, unless we know it to be problematic. My general approach to patches tends to be to be contained and have minimal effect on other places.

Instantiating Bundle::Worker does not spawn threads, there needs to be an enq operation to do so (original code changing to do so would show up as changing arguments for the method).

Interestingly the 25 constant seems to have a life of its own, will try to raise this upstream with a reproducer.