Regression: filter_parse type json is dropping events

[timosellin]

My environment

• Fluentd version: 1.1.0
• Docker image: official fluent/fluentd:v1.1.0
• OS on the Host: Ubuntu 16.04.3 LTS
• Linux kernel: 4.13.0-32-generic
• Docker version: 17.12.0-ce

How to reproduce

Config file

$ cat fluent.conf
<source>
  @type tail
  path /x.log
  read_from_head true
  tag docker.*
  <parse>
    @type json
    time_key time
    time_format %Y-%m-%dT%H:%M:%S.%N%Z
    localtime true
  </parse>
</source>

<filter docker.**>
  @type parser
  format json
  key_name log
  reserve_data true
  emit_invalid_record_to_error false
</filter>

<match **>
  @type stdout
</match>

Test data

This is the log file I am reading with tail:

$ cat x.log
{"log":"+ cat /etc/resolv.conf\n","stream":"stderr","time":"2018-02-20T14:26:51.656915333Z"}
{"log":"nameserver 127.0.0.11\n","stream":"stdout","time":"2018-02-20T14:26:51.666966175Z"}
{"log":"options timeout:1 ndots:0\n","stream":"stdout","time":"2018-02-20T14:26:51.666988017Z"}

Expected behavior

Fluentd 1.0.1 shows the expected behavior:

$ docker run --rm -ti -v $PWD/x.log:/x.log -v $PWD/fluent.conf:/fluentd/etc/fluent.conf -e "FLUENTD_OPT=-qq" fluent/fluentd:v1.0.1
2018-02-20 14:26:51.656915333 +0000 docker.x.log: {"log":"+ cat /etc/resolv.conf\n","stream":"stderr"}
2018-02-20 14:26:51.666966175 +0000 docker.x.log: {"log":"nameserver 127.0.0.11\n","stream":"stdout"}
2018-02-20 14:26:51.666988017 +0000 docker.x.log: {"log":"options timeout:1 ndots:0\n","stream":"stdout"}

Actual behavior

Fluentd 1.1.0 drops 2 events:

$ docker run --rm -ti -v $PWD/x.log:/x.log -v $PWD/fluent.conf:/fluentd/etc/fluent.conf -e "FLUENTD_OPT=-qq" fluent/fluentd:v1.1.0
2018-02-20 14:26:51.666966175 +0000 docker.x.log: {"log":"nameserver 127.0.0.11\n","stream":"stdout"}

Cause

Looks like the problem is caused by a 3rd-party dependency (oj). It has been introduced in
fluent/fluentd-docker-image@a5792ee#diff-106c3e6808cb07c5ed182eed25142b97

Diff:

- && gem install oj -v 2.18.3 \
+ && gem install oj -v 3.3.10 \

With oj 2.18.3 I see the expected behavior with oj 3.3.10 events are dropped.

[repeatedly]

I confirmed this behaviour.
The problem is oj changes exception handling.
Will fix soon.