-
Notifications
You must be signed in to change notification settings - Fork 122
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Restore record store, processors and logs from checkpoints #1984
Conversation
Caveats: - Didn't start source from checkpoint - Didn't checkpoint and restore processors - Didn't handle unpersisted cache data
And some styling fixes
I pushed another commit with a bug fix but Github has trouble verifying it. main...chubei:dozer:feat/checkpoint_restore |
Cursor seems like a re-implementation of the Furthermore, I think a lot of the |
I'm not using |
I meant that it also implements most of the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Generally looks very good. I think putting effort in using serde better is not a good use of our time right now, and the current implementation looks correct and performant.
This is the followup of #1854.
In this PR, in addition to writing checkpoints of the record store, we restore the record store and log from the checkpoints.
We also write checkpoints of processors. Now every processor implements
serialize
to dump its state to anObject
. Upon restart, theProcessorFactory::build
gets an additional parametercheckpoint_data
, which is the data that was dumped during one of theserialize
s. Every processor also restores its state fromcheckpoint_data
.However, due to the fact that
ConnectorSource
currently can't start from a checkpoint, the processor restoring functions won't actually run. We'll address the connector restart problem in another PR.