input tail plugin stops working after file rotation

[bertpassek]

Describe the bug

We use the input tail plugin for processing tracking data and we are struggling with missing data after file rotation. What happens ist, file gets rotated and fluentd does not recognize the new file after rotation. Data is written to the new file and will not be processed until next rotation happens. Unfortunately, it's rare and it just happens sometimes.

To Reproduce

not reproducible

Expected behavior

data gets processed after file rotation

Your Environment

- Fluentd version: 1.15.1
- TD Agent version:
- Operating system: Ubuntu 20.04.4 LTS
- Kernel version: 5.13.0-1031-aws

Your Configuration

<source >
    @type "tail"
    @id "input_tail"
    @log_level "info"

    <parse >
      @type "json"
      time_key "tt"
      time_type "unixtime"
      keep_time_key "true"
    </parse>

    path "/home/tracking/*.log,/home/tracking/retry/*.log,/home/tracking/error/*.log"
    follow_inodes "true"
    refresh_interval 5
    read_from_head "true"
    pos_file "/var/log/td-agent/tmp/tracking.pos"
    pos_file_compaction_interval "1h"
    rotate_wait 30
    tag "*"
  </source>

Your Error Log

no error logs

Additional context

No response

[ashie]

Same issue with #3614
At v1.15.1 we added a log message like the following to investigate this issue:

2022-08-01 17:57:19 +0900 [warn]: #0 Skip update_watcher because watcher has been already updated by other inotify event path="logs/hige.log" inode=59511737 inode_in_pos_file=0

If you see such log message, please report it to #3614

[bertpassek]

just update the input tail config, we will try to set enable_stat_watcher = false