concat filter bug

[hunkeelin]

Problem

Fluentd not sendings logs to es or in s3 successfully. I got a similar issue to #37
...

My configuration

<match fluent.**>
  @type null
</match>
<source>
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/es-containers.log.pos
  time_format %Y-%m-%dT%H:%M:%S.%NZ
  tag kubernetes.*
  format json
 
  read_from_head false
</source>
 
<filter kubernetes.**>
  @type concat
  key log
  multiline_start_regexp /(?<time>\d+(?:-\d+){2}\s+\d+(?::\d+){2}\.\d+)/
  flush_interval 10
</filter>
<filter kubernetes.**>
  @type kubernetes_metadata
</filter>
<match kubernetes.var.log.containers.**_kube-system_**>
  @type null
</match>
<match kubernetes.var.log.containers.**_test_**>
  @type null
</match>
<match kubernetes.var.log.containers.**_monitoring_**>
  @type null
</match>
<match kubernetes.var.log.containers.**_istio-system_**>
  @type null
</match>
<match kubernetes.var.log.containers.**_default_**>
  @type null
</match>
<match kubernetes.var.log.containers.**_fb-wilson_**>
  @type null
</match>
<match kubernetes.var.log.containers.**_infrastructure_**>
  @type null
</match>
<match kubernetes.var.log.containers.**_kube-public_**>
  @type null
</match>
<match kubernetes.var.log.containers.**_kube-node-lease_**>
  @type null
</match>
<match kubernetes.var.log.containers.**_istio-proxy_**>
  @type null
</match>
<match kubernetes.var.log.containers.**_fluentd-**>
  @type null
</match>
<match kubernetes.var.log.containers.**istio-system_**>
  @type null
</match>
<match **>
  @type copy
  deep_copy true
  <store>
   @type "aws-elasticsearch-service"
   @log_level info
   include_tag_key true
   port 443
   logstash_format true
   buffer_chunk_limit 5M
   flush_interval 5s
   max_retry_wait 30
   disable_retry_limit
   num_threads 15
   reload_connections false
   <endpoint>
   </endpoint>
  </store>
  <store>
   @type s3
   buffer_path /var/log/td-agent/s3
   time_slice_format %Y%m%d%H
   time_slice_wait 10m
   timekey_wait 10m
   flush_interval 60s
   buffer_chunk_limit 2M
   chunk_limit_size 256m
   buffer_queue_limit  250
   utc
   format_json true
   include_time_key true
   include_tag_key true
   buffer_chunk_limit 256m
  </store>
</match>

Warning message

2020-02-29 00:06:26 +0000 [info]: #0 Timeout flush: kubernetes.var.log.containers.foo-57bd446b67-5htz9_api_service-8724f4d1297417ec0572007a48dbc381dbdfbf38e634ff63c460a600b2ec2bd2.log:default
2020-02-29 00:06:36 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::ConcatFilter::TimeoutError error="Timeout flush: kubernetes.var.log.containers.fluentd-elasticsearch-4b5sb_logging_fluentd-elasticsearch-1886c1f72c4df4b2bfa247895d81141bd4b61e6b1a224f97ca836750fce7114b.log:default" location=nil tag="kubernetes.var.log.containers.fluentd-elasticsearch-4b5sb_logging_fluentd-elasticsearch-1886c1f72c4df4b2bfa247895d81141bd4b61e6b1a224f97ca836750fce7114b.log" time=#<Fluent::EventTime:0x007f3e94211b08 @sec=1582934796, @nsec=337412517> record={"log"=>"2020-02-29 00:06:26 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::ConcatFilter::TimeoutError error=\"Timeout flush: kubernetes.var.log.containers.foo-57bd446b67-5htz9_api_service-8724f4d1297417ec0572007a48dbc381dbdfbf38e634ff63c460a600b2ec2bd2.log:default\" location=nil tag=\"kubernetes.var.log.containers.foo-57bd446b67-5htz9_api_service-8724f4d1297417ec0572007a48dbc381dbdfbf38e634ff63c460a600b2ec2bd2.log\" time=#<Fluent::EventTime:0x007f3e91789890 @sec=1582934786, @nsec=336591187> record={\"log\"=>\"2020-02-29 00:06:16.065  INFO 1 --- [       Thread-3] b.c.c.NetworkCardTransactionEventHandler : Ignoring ISO8583 network management request with eventId:568e6a6d-5e00-4a30-be26-5d304c83357b\\n\", \"stream\"=>\"stdout\"}\n\n2020-02-29 00:06:26 +0000 [info]: #0 Timeout flush: kubernetes.var.log.containers.foo-57bd446b67-5htz9_api_service-8724f4d1297417ec0572007a48dbc381dbdfbf38e634ff63c460a600b2ec2bd2.log:default\n", "stream"=>"stdout"}

[okkez]

If you use k8s, you can try configuration for k8s in README.md. Please search k8s in README.md and check the sample.