From 34ef00002ea2071b5f7b37817e77197b048a9ba7 Mon Sep 17 00:00:00 2001 From: TAGOMORI Satoshi Date: Thu, 13 Jul 2017 11:40:24 +0900 Subject: [PATCH 1/2] Write metadata from head of file at first, then truncate the file to the size of metadata binary. It's to minimize the possibility to make empty metadata file (by failing write metadata after truncating the file), and it should work well with `#restore_metadata` method because MessagePack unpacker reads just one complete msgpack object from the head of file (and ignore rest). --- lib/fluent/plugin/buffer/file_chunk.rb | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/lib/fluent/plugin/buffer/file_chunk.rb b/lib/fluent/plugin/buffer/file_chunk.rb index 004260c7a2..6679e26530 100644 --- a/lib/fluent/plugin/buffer/file_chunk.rb +++ b/lib/fluent/plugin/buffer/file_chunk.rb @@ -244,9 +244,10 @@ def write_metadata(update: true) c: @created_at.to_i, m: (update ? Time.now : @modified_at).to_i, }) + bin = msgpack_packer.pack(data).to_s @meta.seek(0, IO::SEEK_SET) - @meta.truncate(0) - @meta.write(msgpack_packer.pack(data)) + @meta.write(bin) + @meta.truncate(bin.bytesize) end def file_rename(file, old_path, new_path, callback=nil) From ea86fc0dcbf249998073ca11d459c8682635c661 Mon Sep 17 00:00:00 2001 From: TAGOMORI Satoshi Date: Thu, 13 Jul 2017 11:43:45 +0900 Subject: [PATCH 2/2] write the metadata into file as soon as possible after initializing internal data --- lib/fluent/plugin/buffer/file_chunk.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/fluent/plugin/buffer/file_chunk.rb b/lib/fluent/plugin/buffer/file_chunk.rb index 6679e26530..38108d2219 100644 --- a/lib/fluent/plugin/buffer/file_chunk.rb +++ b/lib/fluent/plugin/buffer/file_chunk.rb @@ -285,6 +285,7 @@ def create_new_chunk(path, perm) @meta.set_encoding(Encoding::ASCII_8BIT) @meta.sync = true @meta.binmode + write_metadata(update: false) rescue => e # This case is easier than enqueued!. Just removing pre-create buffer file @chunk.close rescue nil