input tail plugin stops working after logfile rotation

[bertpassek]

Describe the bug

We are using fluent for forwarding data logs to kafka/s3 with input tail plugin as data source. Somehow fluentd stopped working due to an exception when the file was rotated (done by python logging),

#0 [input_tail] undefined method unwatched=' for nil:NilClass #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:512:in update_watcher'
#0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:881:in call' #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:881:in on_rotate'
#0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1180:in on_notify' #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:814:in on_notify'
#0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:401:in block in setup_watcher' #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:734:in on_change'
#0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in run_once' #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in run'
#0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin_helper/event_loop.rb:93:in block in start' #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in block in thread_create'

We also noted that fluentd started working again, we think when the next lof rotation happened. The following image shows the input entry rate

After digging into the code we found out that rotated_tw (have a look at in_tail.rb at line 503 is using twice, once with and once without nil check. So rotated_tw might be nil and in one case we are running into the exception which stops the input plugin running.

To Reproduce

Expected behavior

Your Environment

- Fluentd version: 1.14.6
- TD Agent version: 1.14.6
- Operating system: Ubuntu 20.04.4 LTS
- Kernel version: 5.13.0-1023-aws

Your Configuration

<source >
  @type tail
  @id input_tail
  @label @TRACKING

  <parse >
    @type json
    time_key tt
    time_type unixtime
    keep_time_key true
  </parse>

  path /home/log/*.log
  follow_inodes true
  refresh_interval 5
  read_from_head true
  pos_file /var/log/td-agent/tmp/fluentd.pos
  pos_file_compaction_interval 72h
  rotate_wait 30
  tag *
</source>

Your Error Log

#0 [input_tail] undefined method `unwatched=' for nil:NilClass
#0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:512:in `update_watcher'
#0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:881:in `call'
#0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:881:in `on_rotate'
#0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:1180:in `on_notify'
#0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:814:in `on_notify'
#0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:401:in `block in setup_watcher'
#0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:734:in `on_change'
#0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
#0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
#0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
#0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

Additional context

No response