[BREAKING] Fix(OOM): Don't unmarshal pb.Proposals until we need them #7059
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
✔️ Deploy preview for dgraph-docs ready! 🔨 Explore the source changes: c8f4c8b 🔍 Inspect the deploy logs: https://app.netlify.com/sites/dgraph-docs/deploys/5fd285f9db83b40007651c75 😎 Browse the preview: https://deploy-preview-7059--dgraph-docs.netlify.app |
manishrjain
changed the title
* Bump up max pending proposal size to 256 MB to increase proposal throughput.
manishrjain
changed the title
NamanJain8
pushed a commit
that referenced
this pull request
Dec 16, 2020
This PR updates the raftmigrate tool. We recently updated raft proposal format for alpha and zero, see #7059 . The tool now can migrate raftwal from old proposal format to new proposal format wherein we are encoding the proposal.Key within the entry.Data. This change will help to migrate raftwal from [RC-2,RC-3] to RC-4 onwards.
NamanJain8
pushed a commit
that referenced
this pull request
Dec 17, 2020
This PR updates the raftmigrate tool. We recently updated raft proposal format for alpha and zero, see #7059 . The tool now can migrate raftwal from old proposal format to new proposal format wherein we are encoding the proposal.Key within the entry.Data. This change will help to migrate raftwal from [RC-2,RC-3] to RC-4 onwards. (cherry picked from commit ab0f3ea)
NamanJain8
added a commit
that referenced
this pull request
Dec 17, 2020
chore: Update Raft migrate tool to handle new proposal format (#7123) This PR updates the raftmigrate tool. We recently updated raft proposal format for alpha and zero, see #7059 . The tool now can migrate raftwal from old proposal format to new proposal format wherein we are encoding the proposal.Key within the entry.Data. This change will help to migrate raftwal from [RC-2,RC-3] to RC-4 onwards. (cherry picked from commit ab0f3ea) Co-authored-by: Rahul Gurnani <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In worker/draft.go, in Raft.Ready() when we get committed entries, we were unmarshalling them to pb.Proposals upfront. Instead, push them to applyCh as it is, and only when we need them, we unmarshal them. This should fix an OOM issue we see with pb.Proposals unmarshal taking up a lot of memory.
We achieve this by switching proposal.Key which is a string to a uint64 key, which can be put directly in the raft entry.Data in the first 8 bytes. The rest are used to marshal proposal.
Additional changes:
This change is