Remove time and tag within S3 file

[kzk]

.@x0jellio requested me that they only want JSON per line at S3 file, doesn't want to have tag and time within a same line. This is pretty reasonable and got the same inquiry from multiple users.

So we could introduce the option, to remove time and tag from S3 file and dump only JSON (maybe json_only or record_only?)

https://github.com/fluent/fluent-plugin-s3/blob/master/lib/fluent/plugin/out_s3.rb#L43

[kzk]

discussed with @frsyuki, and he prefers 'format_json' as an option name.

[tagomoris]

FYI: output_include_time and output_include_tag options available with:

• fluent-plugin-webhdfs
• fluent-plugin-file-alternative
• or other fluent-mixin-plaintextformatter included

[kzk]

btw, @x0jellio sent a note to me that he's only interested in records. Meaning no need to include time and tag to the record.

[repeatedly]

There are 3 situations. Configuration example below:

• Keep default format

format_json: false

2012-12-06T03:14:50Z    s3.hoge {"json":"message"}

• Remove time / tag from same line

format_json: true

{"json":"message"}

• Remove time / tag from same line and add removed time / tag to the record via default mixins

format_json: true
include_tag_key: true
include_time_key: true

{"json":"message", "tag":"s3.hoge", "time":1354763690}

What do you think?

[repeatedly]

All

I pushed first implementation to suppor-format-json branch.
Please check!

[repeatedly]

I think this issue is already resolved.

If you have more opinions, please reopen this issue.