Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Question about default min_heap_size in exometer_probe #100

Closed
surik opened this issue Nov 22, 2017 · 3 comments
Closed

Question about default min_heap_size in exometer_probe #100

surik opened this issue Nov 22, 2017 · 3 comments

Comments

@surik
Copy link
Contributor

surik commented Nov 22, 2017

Each exometer_probe has set the default min_heap_size to 40000. If I understand it correctly it might be the reason for big memory consumption. According to Erlang documentation:

Initial Heap Size
...

The gain is twofold:

* Although the garbage collector grows the heap, it grows it step-by-step, 
  which is more costly than directly establishing a larger heap when the process is spawned.
* The garbage collector can also shrink the heap if it is much larger than 
  the amount of data stored on it; setting the minimum heap size prevents that.

I can observe interesting behavior in my test system where I have around 5000 histograms. That they immediately on startup allocate around 1.5Gb data and during the system running the amount of memory is growing and it seems that GC doesn't collect data from exometer_prob processes.

As soon I set min_heap_size to default value everything works better. Around 200Mb on startup for 5000 histograms and seems that GC cleans them time to time.

Since there is an ability to tune this value via probe options can we use min_heap_size with value by default?

Thanks in advance and sorry if it was already discussed somewhere.
@surik
Copy link
Contributor Author

surik commented Dec 14, 2017

See #102

@surik surik mentioned this issue Feb 21, 2018
Merged
surik added a commit to surik/eradius that referenced this issue Feb 21, 2018
@surik
Set min_heap_size to default value for exometer histograms
ee76d10 
Because the default value provided by exometer is 40000 which is high
for big amount of histograms, see Feuerlabs/exometer_core#100

We set it to 233 according to default value described in Erlang Efficiency Guide.
mgumz added a commit to travelping/gtp_u_edp that referenced this issue Jun 18, 2018
@mgumz
Improve memory consumption
9259021 
For every established GTP-U connection, several metric probes are
created upon connection (see src/gtp_u_edp_port.erl). Among the created
probes are 2 of type "histogram". An histogram is represented as a
process in the GTP-U-Node which will be initialized with a
`min_heap_size` of 40k WORDS (8byte on 64bit machines).

See [1] and [2] for a discussion about the issue.

Neither of the preconditions for increasing the `min_heap_size` are met:

* the histograms are kept over a long period of time (and are not short
  lived)
* the amount of processes keep growing with the amount of established
  connections

This commit sets the `min_heap_size` option of the spawned processes
responsible for the histogram to the default Erlang value and thus,
decreases the memory consumption of the histograms and allow the GC to
work.

[1]: Feuerlabs/exometer_core#100
[2]: http://erlang.org/doc/efficiency_guide/processes.html
@mgumz mgumz mentioned this issue Jun 18, 2018
Merged
RoadRunnr pushed a commit to travelping/gtp_u_edp that referenced this issue Jun 18, 2018
@mgumz @RoadRunnr
Improve memory consumption
47a45ad 
For every established GTP-U connection, several metric probes are
created upon connection (see src/gtp_u_edp_port.erl). Among the created
probes are 2 of type "histogram". An histogram is represented as a
process in the GTP-U-Node which will be initialized with a
`min_heap_size` of 40k WORDS (8byte on 64bit machines).

See [1] and [2] for a discussion about the issue.

Neither of the preconditions for increasing the `min_heap_size` are met:

* the histograms are kept over a long period of time (and are not short
  lived)
* the amount of processes keep growing with the amount of established
  connections

This commit sets the `min_heap_size` option of the spawned processes
responsible for the histogram to the default Erlang value and thus,
decreases the memory consumption of the histograms and allow the GC to
work.

[1]: Feuerlabs/exometer_core#100
[2]: http://erlang.org/doc/efficiency_guide/processes.html
mgumz added a commit to travelping/ergw-gtp-u-edp that referenced this issue Jun 19, 2018
@mgumz
Improve memory consumption
a24782a 
For every established GTP-U connection, several metric probes are
created upon connection. Among the created probes are 2 of type
"histogram". An histogram is represented as a process in the GTP-U-Node
which will be initialized with a `min_heap_size` of 40k WORDS (8byte on
64bit machines).

See [1] and [2] for a discussion about the issue.

Neither of the preconditions for increasing the `min_heap_size` are met:

* the histograms are kept over a long period of time (and are not short
  lived)
* the amount of processes keep growing with the amount of established
  connections

This commit upgrades the core dependency which set the `min_heap_size`
value to 233 words: `gtp_u_edp`. The other dependencies are upgraded
because they contain something similar (`eradius`) or because they have
almost a year of fixes ready to be used (`exometer_core`).

[1]: Feuerlabs/exometer_core#100
[2]: http://erlang.org/doc/efficiency_guide/processes.html
@mpope9
Copy link

mpope9 commented Jul 11, 2018

Is there any chance this will be merged?

@tolbrino
Copy link
Contributor

#102 has been merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tolbrino @surik @mpope9