Limit memory overhead

[tylersmalley]

V8 will not trigger a full mark-sweep & mark-compact until there is memory pressure for the max-old-space-size, which is 1.6GB on a 64bit system. This means that Kibana will, at times, consume over 1.6GB of RSS memory.

We will be setting max-old-space-size to 256MB unless it is ran with --dev, in which case we will use the node defaults.

Over-writing the max-old-space-size with NODE_OPTIONS is still possible.

[tylersmalley]

I am going to build and deploy this to DigitalOcean to get a definitive idea of what the usage is. But for now, let's discuss the language used in the documentation and the method for setting the default.

[tylersmalley]

I need to update this batch file to reflect the changes made in bin/kibana. Downloading a Windows VM to verify before adding to the PR.

[LeeDr]

Here's two ways. I think the first is the more common (but longer);

@REM Loop through any command line parameters and if we find "--dev" add "--max-old-space-size=256"
:parse
IF "%~1"=="" GOTO endparse
IF "%~1"=="--dev" set NODE_OPTIONS="--max-old-space-size=256 %NODE_OPTIONS%"
SHIFT
GOTO parse
:endparse

And a single line way;

@REM If the command line parameters do not contain "--dev" then add "--max-old-space-size=256"
echo "%*" | findstr /V /C:"--dev" && set NODE_OPTION="--max-old-space-size=256 %NODE_OPTION%"

[tylersmalley]

With max-old-space-size=256 the process' RSS is staying under 300M.

[tylersmalley]

@LeeDr - would you mind reviewing/testing the changes made to the batch file in this PR?

[Bargs]

Why remove the limit in dev? Dev should be as prod like as possible, otherwise we're just asking for pain down the road.

[tylersmalley]

@Bargs - the optimize process requires much more memory to run and is not used in production, except for plugin installation which goes through bin/kibana-plugin

[Bargs]

Hmmm, I'm guessing you've seen the optimizer crash with --max-old-space=256? I just started it up with 256 without trouble but it hasn't been running long. The optimizer gets started with cluster.fork so I think it should get its own heap. I thought we might be able to configure max-old-space separately if it's just the optimizer that needs more memory, but unfortunately that doesn't seem possible.

[tylersmalley]

This illustrates the memory usage as compared to 5.0.0-alpha1.

[rashidkpc]

512 here? I assume this is because some amount of memory will of course be used that isn't old space

[tylersmalley]

That is correct, it's just a recommendation. We could probably run on lower, but we do have things which are not old space.

[tylersmalley]

@rashidkpc, thoughts this verbiage?

[rashidkpc]

LGTM

[rashidkpc]

In total: LGTM

[LeeDr]

LGTM
I re-tested on Windows with no NODE_OPTIONS setting, and memory seemed to stay around 256M or lower.

With --dev and no NODE_OPTIONS, it went to about 614M while running functional tests.

With --max-old-space-size=50 it crashes with and without --dev.

With --max-old-space-size=5000 without --dev I saw the memory get to about 1G but then dropped down to around 500M while the functional tests were running.

[tylersmalley]

@epixa - what do you think about getting this into alpha2? It's ready to merge, but I can hold off if you would like to wait for beta1.

I also did further testing on Linux 32bit today.

[epixa]

Yep, let's do it.

[epixa]

I found a pretty big problem with this:

Optimizing does happen on the server in production in at least one scenario: If you remove a plugin, the next time you start up the server, the optimizer is run on the remaining plugins. I just removed x-pack on an alpha2 build and now I can't start kibana with the default settings because the optimizer makes it crash due to out-of-memory.