A "tag *" value in "<source> type tail" section yields errors in the log when fluent container starts

[dougbeauchene]

I am using fluentd v0.12.32.

I am tailing log files written to /var/lib/docker/containers/*

If I have tag * in my <source> section of type tail in order to pass on the entire tag to filter and match sections, there seems to be an error occuring as noted below in the log when I start my fluent container.

If I change the tag to a value other than * (for example tag doug or tag doug.*) no errors occur, everything works as expected.

When I prepend the tag like this tag doug.* the full tag, prepended with doug is passed on to the <filter> and <match> sections.

From the documentation, I was expecting to be able to specify tag * and just pass along the input tag as is and not have to prepend it with anything. Should it work this way?

<source>
  type tail
  path /var/lib/docker/containers/*/*-json.log
  tag *
  pos_file /var/lib/docker/containers/d4b.pos
  format json
  time_format %Y-%m-%dT%H:%M:%S
</source>
<filter fluent.*>
  @type record_transformer
  <record>
    message FLUENT:, ${record["message"]}
    loglevel ${tag_parts[1]}
    dlb_container_id ${hostname}
  </record>
</filter>
<filter var.lib.docker.containers.**>
  @type record_transformer
  <record>
    dlb_container_id ${hostname}
    dlb_tag ${tag}
  </record>
</filter>
<match **>
  type ibmlogging
  flush_interval 10s
  buffer_queue_full_action exception
  buffer_queue_limit 256
  buffer_chunk_limit 8m
  buffer_path /var/fluentd/log_buffer
  secrets_path /run/secrets/d4b-logmet-fluentd
  targetHost temp_value
  targetPort 0000
  retry_limit 8
  retry_wait 1s
  max_retry_wait 10s  # default is infinite
</match>

When I start my fluent container, it loads all the plugins, reads the above conf, starts tailing the logs of 4 running containers...then has a probelm.
Any idea what may be wrong with my fluent.conf?

{"log":"2017-03-20 18:37:36 +0000 [info]: fluent/supervisor.rb:471:read_config: reading config file path=\"/fluentd/etc/fluent.conf\"\n","stream":"stdout","time":"2017-03-20T18:37:36.748747383Z"}
{"log":"2017-03-20 18:37:36 +0000 [info]: fluent/supervisor.rb:337:supervise: starting fluentd-0.12.32\n","stream":"stdout","time":"2017-03-20T18:37:36.748850771Z"}
...
<load plugins>
<read fluent.conf>
...
{"log":"2017-03-20 18:37:38 +0000 [info]: plugin/in_tail.rb:567:initialize: following tail of /var/lib/docker/containers/d2dcaef0fda9e642b45e9915f138c939266812178ee13015652c218f18f0e967/d2dcaef0fda9e642b45e9915f138c939266812178ee13015652c218f18f0e967-json.log\n","stream":"stdout","time":"2017-03-20T18:37:38.089572586Z"}
{"log":"2017-03-20 18:37:38 +0000 [info]: plugin/in_tail.rb:567:initialize: following tail of /var/lib/docker/containers/3dc27558e10c45486201b0d3e4ea24336b272c078a15140371bce76e31732444/3dc27558e10c45486201b0d3e4ea24336b272c078a15140371bce76e31732444-json.log\n","stream":"stdout","time":"2017-03-20T18:37:38.089578468Z"}
{"log":"2017-03-20 18:37:38 +0000 [info]: plugin/in_tail.rb:567:initialize: following tail of /var/lib/docker/containers/822d56de1ccdf53476a10d3656ae7740bd309bf5617deb0940a826c8bcebfe02/822d56de1ccdf53476a10d3656ae7740bd309bf5617deb0940a826c8bcebfe02-json.log\n","stream":"stdout","time":"2017-03-20T18:37:38.089584208Z"}
{"log":"2017-03-20 18:37:38 +0000 [info]: plugin/in_tail.rb:567:initialize: following tail of /var/lib/docker/containers/6c9972bbc49d9d56cf09abc10d7cb627e6bb545b220b6a4eba3d4c8f5de05956/6c9972bbc49d9d56cf09abc10d7cb627e6bb545b220b6a4eba3d4c8f5de05956-json.log\n","stream":"stdout","time":"2017-03-20T18:37:38.089590218Z"}
{"log":"2017-03-20 18:37:39 +0000 [error]: plugin/in_tail.rb:616:rescue in on_notify: undefined method `+' for nil:NilClass\n","stream":"stdout","time":"2017-03-20T18:37:39.091939765Z"}
{"log":"  2017-03-20 18:37:39 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:288:in `receive_lines'\n","stream":"stdout","time":"2017-03-20T18:37:39.091967354Z"}
{"log":"  2017-03-20 18:37:39 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:411:in `wrap_receive_lines'\n","stream":"stdout","time":"2017-03-20T18:37:39.09197415Z"}
{"log":"  2017-03-20 18:37:39 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:606:in `on_notify'\n","stream":"stdout","time":"2017-03-20T18:37:39.091980226Z"}
{"log":"  2017-03-20 18:37:39 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:437:in `on_notify'\n","stream":"stdout","time":"2017-03-20T18:37:39.091996285Z"}
{"log":"  2017-03-20 18:37:39 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:522:in `on_timer'\n","stream":"stdout","time":"2017-03-20T18:37:39.092002859Z"}
{"log":"  2017-03-20 18:37:39 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/cool.io-1.4.6/lib/cool.io/loop.rb:88:in `run_once'\n","stream":"stdout","time":"2017-03-20T18:37:39.092008615Z"}
{"log":"  2017-03-20 18:37:39 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/cool.io-1.4.6/lib/cool.io/loop.rb:88:in `run'\n","stream":"stdout","time":"2017-03-20T18:37:39.092014037Z"}
{"log":"  2017-03-20 18:37:39 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:277:in `run'\n","stream":"stdout","time":"2017-03-20T18:37:39.092019617Z"}
{"log":"2017-03-20 18:37:40 +0000 [error]: plugin/in_tail.rb:616:rescue in on_notify: undefined method `+' for nil:NilClass\n","stream":"stdout","time":"2017-03-20T18:37:40.09018477Z"}
{"log":"  2017-03-20 18:37:40 +0000 [error]: plugin/in_tail.rb:522:on_timer: suppressed same stacktrace\n","stream":"stdout","time":"2017-03-20T18:37:40.090212339Z"}
{"log":"2017-03-20 18:37:41 +0000 [error]: plugin/in_tail.rb:616:rescue in on_notify: undefined method `+' for nil:NilClass\n","stream":"stdout","time":"2017-03-20T18:37:41.090561325Z"}
{"log":"  2017-03-20 18:37:41 +0000 [error]: plugin/in_tail.rb:522:on_timer: suppressed same stacktrace\n","stream":"stdout","time":"2017-03-20T18:37:41.091272632Z"}
{"log":"2017-03-20 18:37:42 +0000 [error]: plugin/in_tail.rb:616:rescue in on_notify: undefined method `+' for nil:NilClass\n","stream":"stdout","time":"2017-03-20T18:37:42.091080643Z"}
{"log":"  2017-03-20 18:37:42 +0000 [error]: plugin/in_tail.rb:522:on_timer: suppressed same stacktrace\n","stream":"stdout","time":"2017-03-20T18:37:42.091109684Z"}
{"log":"2017-03-20 18:37:43 +0000 [error]: plugin/in_tail.rb:616:rescue in on_notify: undefined method `+' for nil:NilClass\n","stream":"stdout","time":"2017-03-20T18:37:43.090042188Z"}
{"log":"  2017-03-20 18:37:43 +0000 [error]: plugin/in_tail.rb:522:on_timer: suppressed same stacktrace\n","stream":"stdout","time":"2017-03-20T18:37:43.090067049Z"}
{"log":"2017-03-20 18:37:43 +0000 [error]: plugin/in_tail.rb:616:rescue in on_notify: undefined method `+' for nil:NilClass\n","stream":"stdout","time":"2017-03-20T18:37:43.097276805Z"}
{"log":"  2017-03-20 18:37:43 +0000 [error]: plugin/in_tail.rb:538:on_change: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:288:in `receive_lines'\n","stream":"stdout","time":"2017-03-20T18:37:43.097751368Z"}
{"log":"  2017-03-20 18:37:43 +0000 [error]: plugin/in_tail.rb:538:on_change: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:411:in `wrap_receive_lines'\n","stream":"stdout","time":"2017-03-20T18:37:43.097769381Z"}
{"log":"  2017-03-20 18:37:43 +0000 [error]: plugin/in_tail.rb:538:on_change: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:606:in `on_notify'\n","stream":"stdout","time":"2017-03-20T18:37:43.097775425Z"}
{"log":"  2017-03-20 18:37:43 +0000 [error]: plugin/in_tail.rb:538:on_change: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:437:in `on_notify'\n","stream":"stdout","time":"2017-03-20T18:37:43.097781471Z"}
{"log":"  2017-03-20 18:37:43 +0000 [error]: plugin/in_tail.rb:538:on_change: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:538:in `on_change'\n","stream":"stdout","time":"2017-03-20T18:37:43.097787237Z"}
{"log":"  2017-03-20 18:37:43 +0000 [error]: plugin/in_tail.rb:538:on_change: /usr/lib/ruby/gems/2.3.0/gems/cool.io-1.4.6/lib/cool.io/loop.rb:88:in `run_once'\n","stream":"stdout","time":"2017-03-20T18:37:43.097805606Z"}
{"log":"  2017-03-20 18:37:43 +0000 [error]: plugin/in_tail.rb:538:on_change: /usr/lib/ruby/gems/2.3.0/gems/cool.io-1.4.6/lib/cool.io/loop.rb:88:in `run'\n","stream":"stdout","time":"2017-03-20T18:37:43.098485994Z"}
{"log":"  2017-03-20 18:37:43 +0000 [error]: plugin/in_tail.rb:538:on_change: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:277:in `run'\n","stream":"stdout","time":"2017-03-20T18:37:43.098503068Z"}
{"log":"2017-03-20 18:37:44 +0000 [error]: plugin/in_tail.rb:616:rescue in on_notify: undefined method `+' for nil:NilClass\n","stream":"stdout","time":"2017-03-20T18:37:44.09030848Z"}
{"log":"  2017-03-20 18:37:44 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:288:in `receive_lines'\n","stream":"stdout","time":"2017-03-20T18:37:44.090891622Z"}
{"log":"  2017-03-20 18:37:44 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:411:in `wrap_receive_lines'\n","stream":"stdout","time":"2017-03-20T18:37:44.090908959Z"}
{"log":"  2017-03-20 18:37:44 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:606:in `on_notify'\n","stream":"stdout","time":"2017-03-20T18:37:44.090915223Z"}
{"log":"  2017-03-20 18:37:44 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:437:in `on_notify'\n","stream":"stdout","time":"2017-03-20T18:37:44.090920823Z"}
{"log":"  2017-03-20 18:37:44 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:522:in `on_timer'\n","stream":"stdout","time":"2017-03-20T18:37:44.090926303Z"}
{"log":"  2017-03-20 18:37:44 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/cool.io-1.4.6/lib/cool.io/loop.rb:88:in `run_once'\n","stream":"stdout","time":"2017-03-20T18:37:44.090931927Z"}
{"log":"  2017-03-20 18:37:44 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/cool.io-1.4.6/lib/cool.io/loop.rb:88:in `run'\n","stream":"stdout","time":"2017-03-20T18:37:44.090937427Z"}
{"log":"  2017-03-20 18:37:44 +0000 [error]: plugin/in_tail.rb:522:on_timer: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.32/lib/fluent/plugin/in_tail.rb:277:in `run'\n"
...

[repeatedly]

Thanks for the report.
@tag_prefix ||= '' is needed here:

fluentd/lib/fluent/plugin/in_tail.rb

Line 130 in 85d5190

@tag_suffix ||= ''