Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Write buffer metadata into file ASAP #1628

Merged
merged 2 commits into from
Jul 21, 2017

Conversation

tagomoris
Copy link
Member

As shown in #1589, file buffer chunks possibly make broken/empty .meta files for its metadata.
I found that some points to be improved to prevent such cases:

  • write metadata into files just after initializing buffer chunk object
    • before: .meta content is written at the time of #commit or enqueue! - it's too late
  • write metadata msgpack binary, then truncate .meta file into the size of metadata
    • before: .meta was truncated into 0 at first, then msgpack binary was written

…the size of metadata binary.

It's to minimize the possibility to make empty metadata file (by failing write metadata after truncating the file), and
it should work well with `#restore_metadata` method because MessagePack unpacker reads just one complete msgpack object
from the head of file (and ignore rest).
@tagomoris tagomoris requested a review from repeatedly July 13, 2017 02:48
@tagomoris
Copy link
Member Author

tagomoris commented Jul 13, 2017

The behavior of msgpack unpacker for non-truncated files is proven here: https://gist.github.com/tagomoris/4b22651072aadc74b0ab9ecdfcf658a2

[a loooooooooooooooooooooooong msgpack binary]
[overwriting short binary]oong msgpack binary] # => unpacker unpacks shorter one only

@repeatedly repeatedly added bug Something isn't working v0.14 labels Jul 13, 2017
@repeatedly repeatedly merged commit 3492a75 into master Jul 21, 2017
@repeatedly
Copy link
Member

repeatedly commented Jul 21, 2017

I tested this patch with several days and no problem on my mac.
So merged and will release rc version for testing at other environment.

@repeatedly repeatedly deleted the truncate-after-write-on-file-metadata branch March 9, 2018 23:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working v0.14
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants