Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compressable.rb lost some data on decompression #2546

Closed
komeda-shinji opened this issue Aug 8, 2019 · 0 comments · Fixed by #2547
Closed

compressable.rb lost some data on decompression #2546

komeda-shinji opened this issue Aug 8, 2019 · 0 comments · Fixed by #2547
Labels
bug Something isn't working

Comments

@komeda-shinji
Copy link

When sender fluend enable compress gzip, reciever fluend lost some data.

I put logging code to lib/fluent/plugin/compressable.rb, debug messages show unused is nil, but input stream is not EOF.
unused.nil? does not show end of stream.

--- compressable.rb-	2018-10-11 12:11:31.000000000 +0900
+++ compressable.rb	2019-08-08 13:26:25.589042228 +0900
@@ -64,6 +64,7 @@
           unused = gz.unused
           gz.finish
 
+          $log.debug "string_decompress", unused: (unused.nil? ? "nil" : "defined"), eof: (io.eof ? "EOF" : "not")
           break if unused.nil?
           adjust = unused.length
           io.pos -= adjust
2019-08-08 13:19:16 +0900 [debug]: #0 string_decompress unused="defined" eof="not"
2019-08-08 13:19:16 +0900 [debug]: #0 string_decompress unused="defined" eof="not"
2019-08-08 13:19:16 +0900 [debug]: #0 string_decompress unused="nil" eof="not"

I change check eof, then is works fine.

--- a/lib/fluent/plugin/compressable.rb
+++ b/lib/fluent/plugin/compressable.rb
@@ -64,9 +64,11 @@ module Fluent
           unused = gz.unused
           gz.finish
 
-          break if unused.nil?
-          adjust = unused.length
-          io.pos -= adjust
+          unless unused.nil?
+            adjust = unused.length
+            input.pos -= adjust
+          end
+          break if io.eof?
         end
 
         out
@komeda-shinji komeda-shinji added the bug Something isn't working label Aug 8, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant