Plug a memory leaks on every reload #91

kenhys · 2021-05-13T07:55:59Z

The resources must be freed during shutdown process when fluentd
graceful reload is executed. Without it, fluentd --under-supervisor
process keeps to consume memory.

Reproducible steps:

Use the following fluent.conf:

<system>
  log_level info
  rpc_endpoint "127.0.0.1:24444"
</system>

<source>
  @type sample
  auto_increment_key count
  tag sample
</source>

<match sample>
  @type stdout
</match>Before:

Launch fluentd -c fluent.conf
Execute curl http://127.0.0.1:24444/api/config.gracefulReload few times. less than 10 times is enough.

FYI: fluent/fluentd#3342

Before:

Every fluentd is reloaded, the value of top (RES) increases.

  % while true; do top -b -n 1|\grep ruby;sleep 1;done
  126501 kenhys    20   0  193052  46452  11004 S   0.0   0.1   0:00.52 ruby
  126506 kenhys    20   0 1247164  82512  47536 S   0.0   0.3   0:00.48 ruby
  126501 kenhys    20   0  193052  46452  11004 S   0.0   0.1   0:00.52 ruby
  126506 kenhys    20   0 1247324  82776  47536 S   0.0   0.3   0:00.49 ruby
  126501 kenhys    20   0  193052  46452  11004 S   0.0   0.1   0:00.52 ruby
  126506 kenhys    20   0 1777844  99824  64484 S   0.0   0.3   0:00.49 ruby
  126501 kenhys    20   0  193052  46452  11004 S   0.0   0.1   0:00.52 ruby
  126506 kenhys    20   0 1777844  99824  64484 S   0.0   0.3   0:00.49 ruby
  126501 kenhys    20   0  193052  46452  11004 S   0.0   0.1   0:00.53 ruby
  126506 kenhys    20   0 1777976 100072  64484 S   0.0   0.3   0:00.50 ruby
  126501 kenhys    20   0  193052  46452  11004 S   0.0   0.1   0:00.53 ruby
  126506 kenhys    20   0 2308496 119016  83348 S   0.0   0.4   0:00.51 ruby
  126501 kenhys    20   0  193052  46452  11004 S   0.0   0.1   0:00.54 ruby
  126506 kenhys    20   0 2308496 119016  83348 S   0.0   0.4   0:00.52 ruby
  126501 kenhys    20   0  193052  46452  11004 S   0.0   0.1   0:00.54 ruby
  126506 kenhys    20   0 2839016 138284 102556 S   0.0   0.4   0:00.52 ruby
  126501 kenhys    20   0  193052  46452  11004 S   0.0   0.1   0:00.54 ruby
  126506 kenhys    20   0 2839016 138284 102556 S   0.0   0.4   0:00.52 ruby
  126501 kenhys    20   0  193052  46664  11068 S   0.0   0.1   0:00.55 ruby
  126506 kenhys    20   0 2839016 138356 102620 S   0.0   0.4   0:00.52 ruby
  126501 kenhys    20   0  193052  46664  11068 S   0.0   0.1   0:00.55 ruby
  126506 kenhys    20   0 3369536 155020 119256 S   0.0   0.5   0:00.53 ruby
  126501 kenhys    20   0  193052  46664  11068 S   0.0   0.1   0:00.55 ruby
  126506 kenhys    20   0 3369536 155020 119256 S   0.0   0.5   0:00.54 ruby
  126501 kenhys    20   0  193052  46664  11068 S   6.7   0.1   0:00.56 ruby
  126506 kenhys    20   0 3900056 173852 138016 S   0.0   0.5   0:00.54 ruby
  126501 kenhys    20   0  193052  46664  11068 S   0.0   0.1   0:00.57 ruby
  126506 kenhys    20   0 3900056 173852 138016 S   0.0   0.5   0:00.54 ruby
  126501 kenhys    20   0  193052  46664  11068 S   0.0   0.1   0:00.57 ruby
  126506 kenhys    20   0 4430576 193100 157192 S   0.0   0.6   0:00.54 ruby
  126501 kenhys    20   0  193052  46780  11068 S   0.0   0.1   0:00.57 ruby
  126506 kenhys    20   0 4430576 193104 157192 S   0.0   0.6   0:00.54 ruby
  126501 kenhys    20   0  193052  46780  11068 S   0.0   0.1   0:00.57 ruby
  126506 kenhys    20   0 4430576 193152 157192 S   0.0   0.6   0:00.54 ruby

After:

Even though fluentd is reloaded, the value of RES does not increase.

  % while true; do top -b -n 1|\grep ruby;sleep 1;done
  117844 kenhys    20   0  186860  46308  10848 S   0.0   0.1   0:00.59 ruby
  117858 kenhys    20   0  712276  62804  28196 S   0.0   0.2   0:00.59 ruby
  117844 kenhys    20   0  186860  46308  10848 S   0.0   0.1   0:00.59 ruby
  117858 kenhys    20   0  712276  62804  28196 S   0.0   0.2   0:00.59 ruby
  117844 kenhys    20   0  186860  46308  10848 S   0.0   0.1   0:00.59 ruby
  117858 kenhys    20   0  712276  62804  28196 S   0.0   0.2   0:00.59 ruby
  117844 kenhys    20   0  186860  46308  10848 S   0.0   0.1   0:00.60 ruby
  117858 kenhys    20   0  712276  62804  28196 S   0.0   0.2   0:00.59 ruby
  117844 kenhys    20   0  186860  46308  10848 S   0.0   0.1   0:00.60 ruby
  117858 kenhys    20   0  712276  62804  28196 S   0.0   0.2   0:00.59 ruby
  117844 kenhys    20   0  186860  46308  10848 S   0.0   0.1   0:00.60 ruby
  117858 kenhys    20   0  712276  62804  28196 S   0.0   0.2   0:00.59 ruby
  117844 kenhys    20   0  186860  46308  10848 S   0.0   0.1   0:00.60 ruby
  117858 kenhys    20   0  712276  62804  28196 S   0.0   0.2   0:00.59 ruby
  117844 kenhys    20   0  193016  46308  10848 S   0.0   0.1   0:00.60 ruby
  117858 kenhys    20   0  186144  45892  11244 S   0.0   0.1   0:00.60 ruby
  117844 kenhys    20   0  193016  46308  10848 S   0.0   0.1   0:00.60 ruby
  117858 kenhys    20   0  716664  63160  28372 S   0.0   0.2   0:00.60 ruby
  117844 kenhys    20   0  193016  46388  10912 S   0.0   0.1   0:00.60 ruby
  117858 kenhys    20   0  716664  63216  28436 S   0.0   0.2   0:00.60 ruby

The resources must be freed during shutdown process when fluentd graceful reload is executed. Without it, fluentd --under-supervisor process keeps to consume memory. Before: Every fluentd is reloaded, the value of RES increases. % while true; do top -b -n 1|\grep ruby;sleep 1;done 126501 kenhys 20 0 193052 46452 11004 S 0.0 0.1 0:00.52 ruby 126506 kenhys 20 0 1247164 82512 47536 S 0.0 0.3 0:00.48 ruby 126501 kenhys 20 0 193052 46452 11004 S 0.0 0.1 0:00.52 ruby 126506 kenhys 20 0 1247324 82776 47536 S 0.0 0.3 0:00.49 ruby 126501 kenhys 20 0 193052 46452 11004 S 0.0 0.1 0:00.52 ruby 126506 kenhys 20 0 1777844 99824 64484 S 0.0 0.3 0:00.49 ruby 126501 kenhys 20 0 193052 46452 11004 S 0.0 0.1 0:00.52 ruby 126506 kenhys 20 0 1777844 99824 64484 S 0.0 0.3 0:00.49 ruby 126501 kenhys 20 0 193052 46452 11004 S 0.0 0.1 0:00.53 ruby 126506 kenhys 20 0 1777976 100072 64484 S 0.0 0.3 0:00.50 ruby 126501 kenhys 20 0 193052 46452 11004 S 0.0 0.1 0:00.53 ruby 126506 kenhys 20 0 2308496 119016 83348 S 0.0 0.4 0:00.51 ruby 126501 kenhys 20 0 193052 46452 11004 S 0.0 0.1 0:00.54 ruby 126506 kenhys 20 0 2308496 119016 83348 S 0.0 0.4 0:00.52 ruby 126501 kenhys 20 0 193052 46452 11004 S 0.0 0.1 0:00.54 ruby 126506 kenhys 20 0 2839016 138284 102556 S 0.0 0.4 0:00.52 ruby 126501 kenhys 20 0 193052 46452 11004 S 0.0 0.1 0:00.54 ruby 126506 kenhys 20 0 2839016 138284 102556 S 0.0 0.4 0:00.52 ruby 126501 kenhys 20 0 193052 46664 11068 S 0.0 0.1 0:00.55 ruby 126506 kenhys 20 0 2839016 138356 102620 S 0.0 0.4 0:00.52 ruby 126501 kenhys 20 0 193052 46664 11068 S 0.0 0.1 0:00.55 ruby 126506 kenhys 20 0 3369536 155020 119256 S 0.0 0.5 0:00.53 ruby 126501 kenhys 20 0 193052 46664 11068 S 0.0 0.1 0:00.55 ruby 126506 kenhys 20 0 3369536 155020 119256 S 0.0 0.5 0:00.54 ruby 126501 kenhys 20 0 193052 46664 11068 S 6.7 0.1 0:00.56 ruby 126506 kenhys 20 0 3900056 173852 138016 S 0.0 0.5 0:00.54 ruby 126501 kenhys 20 0 193052 46664 11068 S 0.0 0.1 0:00.57 ruby 126506 kenhys 20 0 3900056 173852 138016 S 0.0 0.5 0:00.54 ruby 126501 kenhys 20 0 193052 46664 11068 S 0.0 0.1 0:00.57 ruby 126506 kenhys 20 0 4430576 193100 157192 S 0.0 0.6 0:00.54 ruby 126501 kenhys 20 0 193052 46780 11068 S 0.0 0.1 0:00.57 ruby 126506 kenhys 20 0 4430576 193104 157192 S 0.0 0.6 0:00.54 ruby 126501 kenhys 20 0 193052 46780 11068 S 0.0 0.1 0:00.57 ruby 126506 kenhys 20 0 4430576 193152 157192 S 0.0 0.6 0:00.54 ruby After: Even though fluentd is reloaded, the value of RES does not increase. % while true; do top -b -n 1|\grep ruby;sleep 1;done 117844 kenhys 20 0 186860 46308 10848 S 0.0 0.1 0:00.59 ruby 117858 kenhys 20 0 712276 62804 28196 S 0.0 0.2 0:00.59 ruby 117844 kenhys 20 0 186860 46308 10848 S 0.0 0.1 0:00.59 ruby 117858 kenhys 20 0 712276 62804 28196 S 0.0 0.2 0:00.59 ruby 117844 kenhys 20 0 186860 46308 10848 S 0.0 0.1 0:00.59 ruby 117858 kenhys 20 0 712276 62804 28196 S 0.0 0.2 0:00.59 ruby 117844 kenhys 20 0 186860 46308 10848 S 0.0 0.1 0:00.60 ruby 117858 kenhys 20 0 712276 62804 28196 S 0.0 0.2 0:00.59 ruby 117844 kenhys 20 0 186860 46308 10848 S 0.0 0.1 0:00.60 ruby 117858 kenhys 20 0 712276 62804 28196 S 0.0 0.2 0:00.59 ruby 117844 kenhys 20 0 186860 46308 10848 S 0.0 0.1 0:00.60 ruby 117858 kenhys 20 0 712276 62804 28196 S 0.0 0.2 0:00.59 ruby 117844 kenhys 20 0 186860 46308 10848 S 0.0 0.1 0:00.60 ruby 117858 kenhys 20 0 712276 62804 28196 S 0.0 0.2 0:00.59 ruby 117844 kenhys 20 0 193016 46308 10848 S 0.0 0.1 0:00.60 ruby 117858 kenhys 20 0 186144 45892 11244 S 0.0 0.1 0:00.60 ruby 117844 kenhys 20 0 193016 46308 10848 S 0.0 0.1 0:00.60 ruby 117858 kenhys 20 0 716664 63160 28372 S 0.0 0.2 0:00.60 ruby 117844 kenhys 20 0 193016 46388 10912 S 0.0 0.1 0:00.60 ruby 117858 kenhys 20 0 716664 63216 28436 S 0.0 0.2 0:00.60 ruby Signed-off-by: Kentaro Hayashi <[email protected]>

cosmo0920 · 2021-05-17T00:04:00Z

@errm Could you kindly take a look?

ashie · 2021-05-26T09:04:33Z

@errm ping

errm

LGTM

I will fix up the rubocop issues before cuting a release!

kenhys · 2021-05-28T09:27:51Z

Thanks!

errm · 2021-05-28T19:42:26Z

Thanks for the fix @kenhys I just pushed 1.0.5 to rubygems

ashie · 2021-05-28T20:38:35Z

Thanks!

- remove ES plugin gem, not used - update system plugin, fixes a leak: fluent-plugins-nursery/fluent-plugin-systemd#91

- remove ES plugin gem, not used - update system plugin, fixes a leak: fluent-plugins-nursery/fluent-plugin-systemd#91 - update kubernetes_metadata_filter, that is compatible with fluentd 1.13

See fluent-plugins-nursery/fluent-plugin-systemd#91 TODO: make src-all when it is ready for newer release Signed-off-by: Kentaro Hayashi <[email protected]>

kenhys mentioned this pull request May 13, 2021

gracefulReloads are leaking memory fluent/fluentd#3342

Closed

errm approved these changes May 28, 2021

View reviewed changes

errm merged commit 0375a99 into fluent-plugins-nursery:master May 28, 2021

kenhys mentioned this pull request May 31, 2021

Memory leak fluent/fluentd#3401

Closed

george-angel added a commit to utilitywarehouse/fluentd that referenced this pull request Jun 11, 2021

Update fluentd to latest

fcc83b8

- remove ES plugin gem, not used - update system plugin, fixes a leak: fluent-plugins-nursery/fluent-plugin-systemd#91

kenhys mentioned this pull request Jul 15, 2021

Use fixed memory leak issue of fluent-plugin-systemd fluent/fluentd-kubernetes-daemonset#605

Merged

kenhys deleted the fix-memory-leaks branch January 27, 2025 02:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Plug a memory leaks on every reload #91

Plug a memory leaks on every reload #91

kenhys commented May 13, 2021 •

edited

Loading

cosmo0920 commented May 17, 2021

ashie commented May 26, 2021

errm left a comment

kenhys commented May 28, 2021

errm commented May 28, 2021

ashie commented May 28, 2021

Plug a memory leaks on every reload #91

Plug a memory leaks on every reload #91

Conversation

kenhys commented May 13, 2021 • edited Loading

cosmo0920 commented May 17, 2021

ashie commented May 26, 2021

errm left a comment

Choose a reason for hiding this comment

kenhys commented May 28, 2021

errm commented May 28, 2021

ashie commented May 28, 2021

kenhys commented May 13, 2021 •

edited

Loading