[HUDI-5511] Do not clean the CkpMetadata dir when restart the job #7620
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
At the beginning, we bootstrap the ckp metadata by cleaning all the messages. This brings in some corner case like 'the write task can not fetch the pending instant correctly when restartting the job', if a checkpoint succeed and the job crashes suddenly, the instant hasn't had time to commit, then the data loss happens, because the last pending instant would be rolled back, while the Flink engin thinks the checkpint/instant is successful. Q: Why we clean the messages here ? A: To prevent inconsistencies between timeline and the messages. Q: Why we decide to keep the messages ? A: There are two cases for the inconsistency: 1. the timeline instant is complete but the ckp message is inflight (for committing instant); 2. the timeline instant is pending while the ckp message does not start (for starting a new instant); For case1, there is no need to re-commit the instant, so it's okey the write task does not get any pending instant when recovering, for case2, the instant is basically pending, it would be rolled back which is in line with expectations. so keeping the ckp messages as is can actually maintain correctness.
Collaborator
XuQianJin-Stars
approved these changes
Jan 8, 2023
fengjian428
pushed a commit
to fengjian428/hudi
that referenced
this pull request
Jan 31, 2023
…ache#7620) In the beginning, we bootstrap the ckp metadata by cleaning all the messages. This introduces some corner case like 'the write task cannot fetch the pending instant correctly when restarting the job', if a checkpoint succeeds and the job crashes suddenly, the instant hasn't had time to commit, then the data loss happens, because the last pending instant would be rolled back, while the Flink engine thinks the checkpoint/instant is successful. Q: Why we clean the messages? A: To prevent inconsistencies between timeline and the messages. Q: Why we decide to keep the messages? A: There are two cases for the inconsistency: 1. the timeline instant is complete but the ckp message is inflight (for committing instant), 2. the timeline instant is pending while the ckp message does not start (for starting a new instant). For case1, there is no need to re-commit the instant, so it's okey the write task does not get any pending instant when recovering, for case2, the instant is basically pending, it would be rolled back which is in line with expectations. Keeping the ckp messages as it is can actually preserve correctness.
XuQianJin-Stars
pushed a commit
that referenced
this pull request
Jan 31, 2023
) In the beginning, we bootstrap the ckp metadata by cleaning all the messages. This introduces some corner case like 'the write task cannot fetch the pending instant correctly when restarting the job', if a checkpoint succeeds and the job crashes suddenly, the instant hasn't had time to commit, then the data loss happens, because the last pending instant would be rolled back, while the Flink engine thinks the checkpoint/instant is successful. Q: Why we clean the messages? A: To prevent inconsistencies between timeline and the messages. Q: Why we decide to keep the messages? A: There are two cases for the inconsistency: 1. the timeline instant is complete but the ckp message is inflight (for committing instant), 2. the timeline instant is pending while the ckp message does not start (for starting a new instant). For case1, there is no need to re-commit the instant, so it's okey the write task does not get any pending instant when recovering, for case2, the instant is basically pending, it would be rolled back which is in line with expectations. Keeping the ckp messages as it is can actually preserve correctness. (cherry picked from commit ff403c8)
4 tasks
nsivabalan
pushed a commit
to nsivabalan/hudi
that referenced
this pull request
Mar 22, 2023
…ache#7620) In the beginning, we bootstrap the ckp metadata by cleaning all the messages. This introduces some corner case like 'the write task cannot fetch the pending instant correctly when restarting the job', if a checkpoint succeeds and the job crashes suddenly, the instant hasn't had time to commit, then the data loss happens, because the last pending instant would be rolled back, while the Flink engine thinks the checkpoint/instant is successful. Q: Why we clean the messages? A: To prevent inconsistencies between timeline and the messages. Q: Why we decide to keep the messages? A: There are two cases for the inconsistency: 1. the timeline instant is complete but the ckp message is inflight (for committing instant), 2. the timeline instant is pending while the ckp message does not start (for starting a new instant). For case1, there is no need to re-commit the instant, so it's okey the write task does not get any pending instant when recovering, for case2, the instant is basically pending, it would be rolled back which is in line with expectations. Keeping the ckp messages as it is can actually preserve correctness.
4 tasks
fengjian428
pushed a commit
to fengjian428/hudi
that referenced
this pull request
Apr 5, 2023
…ache#7620) In the beginning, we bootstrap the ckp metadata by cleaning all the messages. This introduces some corner case like 'the write task cannot fetch the pending instant correctly when restarting the job', if a checkpoint succeeds and the job crashes suddenly, the instant hasn't had time to commit, then the data loss happens, because the last pending instant would be rolled back, while the Flink engine thinks the checkpoint/instant is successful. Q: Why we clean the messages? A: To prevent inconsistencies between timeline and the messages. Q: Why we decide to keep the messages? A: There are two cases for the inconsistency: 1. the timeline instant is complete but the ckp message is inflight (for committing instant), 2. the timeline instant is pending while the ckp message does not start (for starting a new instant). For case1, there is no need to re-commit the instant, so it's okey the write task does not get any pending instant when recovering, for case2, the instant is basically pending, it would be rolled back which is in line with expectations. Keeping the ckp messages as it is can actually preserve correctness.
h1ap
pushed a commit
to h1ap/hudi
that referenced
this pull request
Apr 12, 2023
We received several bug reports since apache#7620, for example: apache#8060, this patch revert the changes of CkpMetadata and always report the write metadata events for write task, the coordinator would decide whether to re-commit these metadata stats. # Conflicts: # hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/meta/TestCkpMetadata.java
stayrascal
pushed a commit
to stayrascal/hudi
that referenced
this pull request
Apr 20, 2023
We received several bug reports since apache#7620, for example: apache#8060, this patch revert the changes of CkpMetadata and always report the write metadata events for write task, the coordinator would decide whether to re-commit these metadata stats.
yihua
pushed a commit
to yihua/hudi
that referenced
this pull request
May 15, 2023
We received several bug reports since apache#7620, for example: apache#8060, this patch revert the changes of CkpMetadata and always report the write metadata events for write task, the coordinator would decide whether to re-commit these metadata stats.
KnightChess
pushed a commit
to KnightChess/hudi
that referenced
this pull request
Jan 2, 2024
We received several bug reports since apache#7620, for example: apache#8060, this patch revert the changes of CkpMetadata and always report the write metadata events for write task, the coordinator would decide whether to re-commit these metadata stats. Signed-off-by: liangjunning <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
At the beginning, we bootstrap the ckp metadata by cleaning all the messages. This brings in some corner case like 'the write task can not fetch the pending instant correctly when restartting the job', if a checkpoint succeed and the job crashes suddenly, the instant hasn't had time to commit, then the data loss happens, because the last pending instant would be rolled back, while the Flink engin thinks the checkpint/instant is successful.
Q: Why we clean the messages here ?
A: To prevent inconsistencies between timeline and the messages.
Q: Why we decide to keep the messages ?
A: There are two cases for the inconsistency:
For case1, there is no need to re-commit the instant, so it's okey the write task does not get any pending instant when recovering, for case2, the instant is basically pending, it would be rolled back which is in line with expectations. so keeping the ckp messages as is can actually maintain correctness.
Change Logs
Keep the ckp messages instead of cleaning it.
Impact
No impact
Risk level (write none, low medium or high below)
none
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change
ticket number here and follow the instruction to make
changes to the website.
Contributor's checklist