td-agent ignores log_level specified in a config file

[vvmnnnkv]

fluentd or td-agent version

td-agent-2.3.4-0.el7.x86_64, installed from official repo

Environment information

CentOS 7.x

Your configuration

The simplest configuration file that should set global log_level to trace:

 <system>
    log_level trace
  </system>

Your problem explanation

Running td-agent -c config.conf with config provided above would output only "info" level messages, while it is expected that log level is "trace":

2017-03-14 14:38:38 -0400 [info]: reading config file path="./fluent.conf"
2017-03-14 14:38:38 -0400 [info]: starting fluentd-0.14.12 pid=19209
2017-03-14 14:38:38 -0400 [info]: spawn command to main:  cmdline=["/opt/td-agent/embedded/bin/ruby", "-Eascii-8bit:ascii-8bit", "/sbin/td-agent", "-c", "./fluent.conf", "--under-supervisor"]
2017-03-14 14:38:39 -0400 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2017-03-14 14:38:39 -0400 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2017-03-14 14:38:39 -0400 [info]: gem 'fluent-plugin-kafka' version '0.4.1'
2017-03-14 14:38:39 -0400 [info]: gem 'fluent-plugin-mongo' version '0.7.16'
2017-03-14 14:38:39 -0400 [info]: gem 'fluent-plugin-mysql' version '0.2.1'
2017-03-14 14:38:39 -0400 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.5'
2017-03-14 14:38:39 -0400 [info]: gem 'fluent-plugin-s3' version '0.8.0'
2017-03-14 14:38:39 -0400 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2017-03-14 14:38:39 -0400 [info]: gem 'fluent-plugin-td' version '0.10.29'
2017-03-14 14:38:39 -0400 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2017-03-14 14:38:39 -0400 [info]: gem 'fluent-plugin-webhdfs' version '0.4.2'
2017-03-14 14:38:39 -0400 [info]: gem 'fluentd' version '0.14.12'
2017-03-14 14:38:39 -0400 [info]: gem 'fluentd' version '0.12.31'
2017-03-14 14:38:39 -0400 [info]: using configuration file: <ROOT>
  <system>
    log_level trace
  </system>
</ROOT>
2017-03-14 14:38:39 -0400 [info]: #0 starting fluentd worker pid=19213 ppid=19209 worker=0
2017-03-14 14:38:39 -0400 [info]: #0 fluentd worker is now running worker=0
^C2017-03-14 14:38:41 -0400 [info]: Received graceful stop
2017-03-14 14:38:42 -0400 [info]: #0 fluentd worker is now stopping worker=0
2017-03-14 14:38:42 -0400 [info]: #0 shutting down fluentd worker worker=0
2017-03-14 14:38:42 -0400 [info]: Worker 0 finished with status 0

However, running td-agent -vv -c config.conf will actually output debug-level logs:

2017-03-14 14:39:27 -0400 [info]: fluent/supervisor.rb:705:read_config: reading config file path="./fluent.conf"
2017-03-14 14:39:27 -0400 [info]: fluent/supervisor.rb:546:supervise: starting fluentd-0.14.12 pid=19216
2017-03-14 14:39:27 -0400 [info]: fluent/supervisor.rb:555:supervise: spawn command to main:  cmdline=["/opt/td-agent/embedded/bin/ruby", "-Eascii-8bit:ascii-8bit", "/sbin/td-agent", "-vv", "-c", "./fluent.conf", "--under-supervisor"]
2017-03-14 14:39:27 -0400 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2017-03-14 14:39:27 -0400 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2017-03-14 14:39:27 -0400 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-kafka' version '0.4.1'
2017-03-14 14:39:27 -0400 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-mongo' version '0.7.16'
2017-03-14 14:39:27 -0400 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-mysql' version '0.2.1'
2017-03-14 14:39:27 -0400 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.5'
2017-03-14 14:39:27 -0400 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-s3' version '0.8.0'
2017-03-14 14:39:27 -0400 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-scribe' version '0.10.14'
2017-03-14 14:39:27 -0400 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-td' version '0.10.29'
2017-03-14 14:39:27 -0400 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2017-03-14 14:39:27 -0400 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-webhdfs' version '0.4.2'
2017-03-14 14:39:27 -0400 [info]: fluent/engine.rb:114:block in configure: gem 'fluentd' version '0.14.12'
2017-03-14 14:39:27 -0400 [info]: fluent/engine.rb:114:block in configure: gem 'fluentd' version '0.12.31'
2017-03-14 14:39:27 -0400 [info]: fluent/engine.rb:150:configure: using configuration file: <ROOT>
  <system>
    log_level trace
  </system>
</ROOT>
2017-03-14 14:39:27 -0400 [info]: #0 fluent/engine.rb:208:run: starting fluentd worker pid=19220 ppid=19216 worker=0
2017-03-14 14:39:27 -0400 [info]: #0 fluent/engine.rb:217:run: fluentd worker is now running worker=0
^C2017-03-14 14:39:29 -0400 [debug]: #0 fluent/supervisor.rb:590:block in install_main_process_signal_handlers: fluentd main process get SIGINT
2017-03-14 14:39:29 -0400 [info]: serverengine/server.rb:51:stop: Received graceful stop
2017-03-14 14:39:30 -0400 [debug]: #0 fluent/supervisor.rb:601:block in install_main_process_signal_handlers: fluentd main process get SIGTERM
2017-03-14 14:39:30 -0400 [debug]: #0 fluent/supervisor.rb:604:block in install_main_process_signal_handlers: getting start to shutdown main process
2017-03-14 14:39:30 -0400 [info]: #0 fluent/engine.rb:219:run: fluentd worker is now stopping worker=0
2017-03-14 14:39:30 -0400 [info]: #0 fluent/engine.rb:236:run: shutting down fluentd worker worker=0
2017-03-14 14:39:30 -0400 [info]: serverengine/multi_process_server.rb:136:alive?: Worker 0 finished with status 0

It seems that log_level set via config file is ignored.