Logs are missing when new file is rotated in tail plugin

[makaras]

With below configuration for source.conf file

      @type tail
      tag logs

      path = "/y=%Y/m=%m/d=%d/h=%H/**/*.json"
      pos_file /fluentd/log/fluentd.pos
      read_from_head true
      follow_inodes true
      open_on_every_update true
      refresh_interval 60
      pos_file_compaction_interval 2h

i have following issue.

On 12PM new log file is created on path: /y=2023/m=06/d=21/h=12/m=00/logs.json
According to refresh_interval 60 property, logs are read from json file every minute.
Json logs are rotated every hour so next file is created under /y=2023/m=06/d=21/h=13/m=00/logs.json path.

So if it became 12:59:01 time it reads a logs from file /y=2023/m=06/d=21/h=12/m=00/logs.json and on next interval on 13:00:01 it started to reading from file /y=2023/m=06/d=21/h=13/m=00/logs.json which means that logs between 12:59:01 till 13:00:00 from /y=2023/m=06/d=21/h=12/m=00/logs.json file is missed

rotate_wait does not work with that config with strftime on path

Is there any configuration example to not lost data every hour with strftime config?

[daipom]

According to refresh_interval 60 property, logs are read from json file every minute.

refresh_interval is the interval to refresh the list of watching files, not the interval to read data.
So the logs should be read immediately.
If it looks like the log is being read every 60 seconds, it may be because of the Output plugin setting.

So the logs appended to the previous file should still be read in this case.
If not, then something is wrong...

[makaras]

Thanks for your answer,

do you have some ideas to test some different configuration? i thought that rotate_wait should work but for some reason fluentd does not recognise new file in new folder that changes every hour as a rotation

or maybe some change in .pos file ? it also not seems that it works properly

[makaras]

thanks for your investigation

do you have any suggestions of further steps? should i focus on file privileges on pod stratup or there is some issue on fluentd?

[daipom]

Sorry, it seems there's nothing more I can find out currently.

Is there a possibility that in_tail cannot access files at No.2 (Immediately after starting fluentd worker)?

Since I can't reproduce this in my local environment, I think it is more likely to be the pod startup problem.

i have added initial delay to startup pod

Before doing this, in_tail could not access the target files at No.3.
After doing this, in_tail could access them at No.3.

Is it possible that access to files is unstable on pod startup?

There is only about 1 second difference between No.2 and No.3,
but doesn't it seem possible that delaying it more might fix it?

[makaras]

i have tried different configurations but with simple sleep 30 command pod cannot properly run

[daipom]

#4212 (reply in thread)

i found out empty

| existing = 

i think that this is an issue, there is no existing file which means that file is still not read at startup ?

Yes, this is wrong. If this is empty, then in_tail will read all records. This means the position file was empty at that point.

There is a possibility that Fluentd did not find the target files at startup, and wrongly recognized that all positions were no longer needed and cleared them.

I'm sorry. What I said here was wrong.

• https://github.com/fluent/fluentd/blob/v1.16.1/lib/fluent/plugin/in_tail.rb#L353-L363

existing = shows the targets that in_tail is currently tailing.
So, this is always empty at first. (in_tail set the tailing targets after that first log.)
in_tail should not read logs in duplicate if this is empty at first, if only the position file is alive.

In conclusion, the debug logs do not tell us whether the previous position data remained or not...
(I will improve this for the next version.)

Since I can't reproduce this in my local environment, I think it is more likely to be the pod startup problem.

I continue to feel the same way about this.

[daipom]

• in_tail: Add debug log for pos file compaction #4228

This PR is merged.
In the next version, we can check the status of the pos file at startup.