out_http: add gzip option

[rockliffelewis]

Which issue(s) this PR fixes:
Fixes #4410

What this PR does / why we need it:
Adds a compress gzip option to out_http to enable compressing the http request before sending it to save on bandwidth

Docs Changes:
fluent/fluentd-docs-gitbook#514

Release Note:

[daipom]

Thanks for the PR!
I think we need to take into account the compression of buffer.
I commentted below on the details, so please check.

[daipom]

I'd like to add test patterns like this: c684509

However, the case of sending the gzipped chunk as is fails in my environment.
(Only one record decompressed on the test server.)
I'm checking the cause.

Failure: test_write_with_gzip[buffer_compress: "gzip", json_array: false](HTTPOutputTest::GZIP)
/home/daipom/work/fluentd/fluentd/test/plugin/test_out_http.rb:564:in `test_write_with_gzip'
     561:         result.content_type
     562:       )
     563:       assert_equal 'gzip', result.headers['content-encoding']
  => 564:       assert_equal test_events, result.data
     565:       assert_not_empty result.headers
     566:     end
     567:   end
<[{"bool"=>true, "message"=>"hello", "num"=>10},
 {"bool"=>false, "message"=>"hello", "num"=>11}]> expected but was
<[{"bool"=>true, "message"=>"hello", "num"=>10}]>

diff:
? [{"bool"=>true, "message"=>"hello", "num"=>10},
?                                               ]
?                                               ?
-  {"bool"=>false, "message"=>"hello", "num"=>11}]

[rockliffelewis]

I'd like to add test patterns like this: c684509

However, the case of sending the gzipped chunk as is fails in my environment. (Only one record decompressed on the test server.) I'm checking the cause.

Failure: test_write_with_gzip[buffer_compress: "gzip", json_array: false](HTTPOutputTest::GZIP)
/home/daipom/work/fluentd/fluentd/test/plugin/test_out_http.rb:564:in `test_write_with_gzip'
     561:         result.content_type
     562:       )
     563:       assert_equal 'gzip', result.headers['content-encoding']
  => 564:       assert_equal test_events, result.data
     565:       assert_not_empty result.headers
     566:     end
     567:   end
<[{"bool"=>true, "message"=>"hello", "num"=>10},
 {"bool"=>false, "message"=>"hello", "num"=>11}]> expected but was
<[{"bool"=>true, "message"=>"hello", "num"=>10}]>

diff:
? [{"bool"=>true, "message"=>"hello", "num"=>10},
?                                               ]
?                                               ?
-  {"bool"=>false, "message"=>"hello", "num"=>11}]

I also noticed this in my testing, im assuming that the behavior is correct but not what the test is expecting.

Its delivering a single gzipped message rather than batching a few messages in the request.

I can add a few more tests to cover off on the different option combinations

[rockliffelewis]

The issue here is that the test harness doesn't really handle getting multiple http requests for one test, and in the gzipped buffer case the messages will come one by one. Im not entirely sure how to fix this properly without doing something like this:

  assert_equal(
    data[:buffer_compress] == 'gzip' && !data[:json_array] ? [test_events[0]] : test_events,
    result.data
  )

[daipom]

@rockliffelewis Thanks for considering it!
I found the cause!

The cause is that zlib (Ruby) does not support gzip concatenation.
The chunk of Fluentd handles it as follows.

fluentd/lib/fluent/plugin/compressable.rb

Lines 57 to 75 in c0cd1e6

	def string_decompress(compressed_data)
	io = StringIO.new(compressed_data)
	out = ''
	loop do
	gz = Zlib::GzipReader.new(io)
	out << gz.read
	unused = gz.unused
	gz.finish
	unless unused.nil?
	adjust = unused.length
	io.pos -= adjust
	end
	break if io.eof?
	end
	out
	end

So, as you say, it's generally not a problem.
However, if we consider a use case where forwarding the gzip data to in_http, this could be a problem.
Currently, in_http does not support gzip concatenation.

fluentd/lib/fluent/plugin/in_http.rb

Lines 509 to 523 in c0cd1e6

	# Content Encoding
	# =================
	# Decode payload according to the "Content-Encoding" header.
	# For now, we only support 'gzip' and 'deflate'.
	begin
	if @content_encoding == 'gzip'.freeze
	@body = Zlib::GzipReader.new(StringIO.new(@body)).read
	elsif @content_encoding == 'deflate'.freeze
	@body = Zlib::Inflate.inflate(@body)
	end
	rescue
	@log.warn 'fails to decode payload', error: $!.to_s
	send_response_and_close(RES_400_STATUS, {}, "")
	return
	end

So, if we consider a use case where forwarding the gzip data to in_http, it would be safer to start with a simple specification that always recompresses the chunk as the first implementation.

What do you think about this?

[rockliffelewis]

Ah i see! I didnt realise the buffer was doing something fancy like that. I wouldn't expect most HTTP receivers to implement that either so at best we would need a recompress flag.

In my usecase I want to use in_http as the receiver, and I want the maximum compression so I would end up buffering it uncompressed anyway.

[daipom]

Thanks! LGTM.
I will wait a bit to merge for the other maintainers.

[ashie]

LGTM
Thanks for your contribution!