cannot use "enum" for specifying "rotate_age".

[mikankun1003]

Describe the bug

• From rotate_age の設定について教えてほしい #4226

In the configuration of rotate_age, you cannot specify "daily". While I haven't tested "weekly" or "monthly", if you set rotate_age to "daily", it will display the following error:

logger/period.rb:21:in 'next_rotate_time': invalid :shift_age :daily, should be daily, weekly, monthly, or everytime (ArgumentError).

To Reproduce

<system>
  workers 1
  log_level info
  <log>
    rotate_age daily
  </log>
</system>

の設定があれば再現する

Expected behavior

I wanted both the logs output by fluentd and the logs output by fluentd:out_file to be rotated daily using the rotate_age daily configuration. Additionally, I wanted to know at what time each day the rotation would occur.

Your Environment

- Fluentd version:1.16.1
- TD Agent version:
- Operating system:xxx
- Kernel version:xxx

Your Configuration

<system>
  workers 1
  log_level info
  <log>
    rotate_age daily
  </log>
</system>

Your Error Log

logger/period.rb:21:in 'next_rotate_time': invalid :shift_age :daily, should be daily, weekly, monthly, or everytime (ArgumentError).

Additional context

No response

[daipom]

Thanks for your report!

Looks like this is a bug of Fluentd.
Ruby Logger needs rotate_age as String, but Fluentd passes it as Symbol.

• https://github.com/ruby/ruby/blob/v3_2_2/lib/logger/period.rb#L9-L22

• fluentd/lib/fluent/system_config.rb

  Lines 63 to 75 in e20697e

  	config_param :rotate_age, default: nil do |v|
  	if Fluent::Log::LOG_ROTATE_AGE.include?(v)
  	v.to_sym
  	else
  	begin
  	Integer(v)
  	rescue ArgumentError => e
  	raise Fluent::ConfigError, e.message
  	else
  	v.to_i
  	end
  	end
  	end

• fluentd/lib/fluent/supervisor.rb

  Lines 707 to 711 in e20697e

  	logdev = Fluent::LogDeviceIO.new(
  	actual_log_path,
  	shift_age: @log_rotate_age,
  	shift_size: @log_rotate_size,
  	)

I can reproduce this at least since v1.15.3.
This probably occurs after rotate_age option is added to system_config.
The command line option works correctly. This is a problem of system_config.

[mrudrego]

Hi,
Can i work on this issue? This would be my first contribution.

Thanks,
Mahesh

[daipom]

@mrudrego Sure! Thanks!

[mrudrego]

Hi @daipom ,

I tried reproducing the issue by installing td-agent rpm "td-agent 4.5.1" which is bundled with fluentd version 1.16.2.
But i dont see the issue with rotating the file.

Configuration used:

 <system>
    workers 1
    log_level info
    <log>
      rotate_age daily
    </log>
  </system>
  <source>
    @type sample
    sample {"hello":"world"}
    tag "sample"
    rate 1
  </source>
  <match **>
    @type file
    path "/tmp/flu-log"
    <buffer time>
      path "/tmp/flu-log"
    </buffer>
  </match>

td-agent process is running for more than 2 days now and there are no errors logs as mentioned. Logs below:

2023-09-15 07:15:08 +0000 [info]: starting **fluentd-1.16.2** pid=8 ruby="2.7.8"
2023-09-15 07:15:08 +0000 [info]: spawn command to main:  cmdline=["/opt/td-agent/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/sbin/td-agent", "--under-supervisor"]
2023-09-15 07:15:10 +0000 [info]: #0 init worker0 logger path=nil **rotate_age=:daily** rotate_size=nil
2023-09-15 07:15:10 +0000 [info]: adding match pattern="**" type="file"
2023-09-15 07:15:10 +0000 [info]: adding source type="sample"
2023-09-15 07:15:10 +0000 [warn]: #0 both of Plugin @id and path for <storage> are not specified. Using on-memory store.
2023-09-15 07:15:10 +0000 [warn]: #0 both of Plugin @id and path for <storage> are not specified. Using on-memory store.
2023-09-15 07:15:10 +0000 [warn]: #0 define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2023-09-15 07:15:10 +0000 [info]: #0 starting fluentd worker pid=13 ppid=8 worker=0
2023-09-15 07:15:10 +0000 [info]: #0 fluentd worker is now running worker=0

Also the files are rotated as expected for every day as shown below:

bash-4.4$ ls -lrt /tmp/
total 7316
drwxr-sr-x. 3 td-agent render    4096 Sep 15 07:15 fluentd-tmp
srwx------. 1 td-agent render       0 Sep 15 07:15 SERVERENGINE_SOCKETMANAGER_2023-09-15T07:15:08Z_8
-rw-r--r--. 1 td-agent render 3074969 **Sep 16 00:10 flu-log.20230915_0.log**
-rw-r--r--. 1 td-agent render 4406400 **Sep 17 00:10 flu-log.20230916_0.log**
drwxr-xr-x. 2 td-agent render    4096 **Sep 17 00:10 flu-log**

Please let me know if i'm missing anything. Or the issue is actually fixed?

Snippet from the file /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.2/lib/fluent/system_config.rb inside the container.

config_section :log, required: false, init: true, multi: false do
      config_param :format, :enum, list: [:text, :json], default: :text
      config_param :time_format, :string, default: '%Y-%m-%d %H:%M:%S %z'
      config_param :rotate_age, default: nil do |v|
        if Fluent::Log::LOG_ROTATE_AGE.include?(v)
          v.to_sym
        else
          begin

Thanks,
Mahesh

[daipom]

@mrudrego
This setting is for Fluentd's own log and has nothing to do with Output(match) settings.
I think you ran Fluentd simply by the following command.

$ td-agent

If so, Fluentd outputs its own logs to the standard output.
In this case, this error doesn't occur because the log file is not created, and the settings for rotation are not used.
(You can use --log option to make Fluentd output its own logs to files.)

In fact, you don't need to execute the td-agent command by yourself.
After installing td-agent, td-agent is registered as a daemon.

• https://docs.fluentd.org/installation/install-by-rpm#step-2-launch-daemon

You can confirm the error when restarting the td-agent daemon by the following command.

$ sudo systemctl restart td-agent

By default, td-agent outputs Fluentd's logs to /var/log/td-agent/ (td-agent) or /var/log/fluentd/ (fluent-package).

Additional information:
td-agent on Linux does not use Fluentd's rotation feature, but logrotate.d.
You can see the default rotate setting in /etc/logrotate.d/td-agent.
(Fluentd's rotation feature is mainly for Windows. In Linux, we don't need to use it since we can use logrotate.d.)

[daipom]

It would be better to run Fluentd directly from Ruby to confirm its functionality, rather than focusing on the package.

• https://docs.fluentd.org/installation/install-from-source

[mrudrego]

@daipom, thank you very much for the guidance. Will check this.

Thanks,