support the compressing stream in forward and secure forward

[haroboro]

It's to save the bandwidth across the data centers, also accelerate the transfer speed and save the time.

[repeatedly]

For secure forward, please check fluent-plugin-secure-forward: https://github.com/tagomoris/fluent-plugin-secure-forward

[repeatedly]

We will consider supporting compression in v0.14's new APIs: #309 (comment)

[cswilliams]

+1, this would be a great feature to help reduce our bandwidth charges on AWS

[deitch]

Yes, I was just looking for this. You would need to support it in out_forward as well, so the receiver can decompress it.

I have a use case where the compressed logs are ~10% of the uncompressed logs. For LAN, that doesn't matter, but for shipping inter-DC or to the cloud, that is a huge deal.

+1

[repeatedly]

@deitch Do you use Gzip or Snappy like compression?

[deitch]

Don't actually care. We have gotten similar results with gzip and plain zip. But as long as in_forward / in_secure_forward and out_forward both support it, I can put a fluentd on each host to collect and forward, and have another cluster of them in the cloud, and everyone is happy.

[tsailiming]

+1

[repeatedly]

Does someone try this patch?

#637

[Manikantan22]

I am looking for a option to compress a message during out_forward. Could someone help me?

[tagomoris]

refer #1179