[ABANDON] Support for single-primary, multi-secondary processes#4931
Closed
riversand963 wants to merge 5 commits intofacebook:masterfrom
Closed
[ABANDON] Support for single-primary, multi-secondary processes#4931riversand963 wants to merge 5 commits intofacebook:masterfrom
riversand963 wants to merge 5 commits intofacebook:masterfrom
Conversation
Summary: As titled. Returning error about non-existing path can help user better handle them. Test Plan: ``` $make clean && make -j32 all check ``` Reviewers: Subscribers: Tasks: Tags:
Add TryReadRecord method to log::Reader so that caller can call `TryReadRecord`
multiple times until a complete record is read. When a complete record is read,
`TryReadRecord` returns `true`; when a record has remaining part yet to be
read, or an error has occurred, `TryReadRecord` returns false. The caller can
implement different retry *policies*.
Also add unit test for non-blocking record read.
Example usage:
```
log::Reader reader(...);
Slice record;
std::string scratch;
while (reader.TryReadRecord(&record, &scratch)) {
// process record
}
if (reader.reader_error_) {
// handle error
} else {
// consider retry
}
```
Note that `reader_error_` is not exposed at the moment. Furthermore, we need
finer-grained error handling according to error type that is not exposed
either.
Update log test and rebase.
Test plan:
```
$make clean && make -j32 all check
$./log_test
$./log_test --gtest_filter=bool/RetriableLogTest.NonBlockingReadFullRecord/*
```
Test plan (to be updated) ``` $make clean && make -j32 all check ``` All tests must pass.
Summary: as title Test Plan: ``` $make clean && make -j32 db_secondary_test $./db_secondary_test ``` All tests must pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR allows RocksDB to run in single-primary, multi-secondary process mode.
The writer is a regular RocksDB (e.g. an
DBImpl) instance playing the role of a primary.Multiple
DBImplSecondaryprocesses (secondaries) share the same set of SST files, MANIFEST, WAL files with the primary. Secondaries tail the MANIFEST of the primary and apply updates to their own in-memory state of the file system, e.g.VersionStorageInfo.This PR has three components:
(Originally in Add PathNotFound subcode to IOError #4745). Add a 'PathNotFound' subcode to
IOErrorto denote the failure when a secondary tries to open a file which has been deleted by the primary.(Originally in Support non-blocking, retryable record reading #4602). Add
TryReadRecordmethod tolog::Readerso that caller can callTryReadRecordmultiple times until a complete record is read. When a complete record is read,
TryReadRecordreturnstrue; when a record has remaining part yet to beread, or an error has occurred,
TryReadRecordreturns false. The caller canimplement different retry policies.
Example usage:
Note that
reader_error_is not exposed at the moment. Furthermore, we needfiner-grained error handling according to error type that is not exposed
either.
DBImplSecondary.3.1 Tail the primary's MANIFEST during recovery.
3.2 Tail the primary's MANIFEST during normal processing by calling
ReadAndApply.3.3 Tailing WAL will be in a future PR.
Test plan
A general check of all tests.
Specifically, if you are interested in testing log reader related changes, run the following
To test the basic functionality of
DBImplSecondary, run the followingAll tests must pass.