KAFKA-10199: Close pending active tasks to init on partitions lost #16545
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
With enabled state updater tasks that are created but not initialized are stored in a set. In each poll iteration the stream thread drains that set, intializes the tasks, and adds them to the state updater. On partition lost, all active tasks are closed. This commit ensures that active tasks pending initialization in the set mentioned above are closed cleanly on partition lost.
cadonna
commented
Jul 7, 2024
| if (stateUpdater != null) { | ||
| final Map<TaskId, CompletableFuture<StateUpdater.RemovedTaskResult>> futures = new LinkedHashMap<>(); | ||
| final Map<TaskId, RuntimeException> failedTasksDuringCleanClose = new HashMap<>(); | ||
| final Set<Task> tasksToCloseClean = new HashSet<>(); | ||
| final Set<Task> tasksToCloseClean = new HashSet<>(tasks.drainPendingActiveTasksToInit()); |
There was a problem hiding this comment.
This is the actual fix.
cadonna
commented
Jul 7, 2024
| @@ -107,6 +108,20 @@ public Set<Task> drainPendingTasksToInit() { | |||
| return result; | |||
| } | |||
|
|
|||
| @Override | |||
| public Set<Task> drainPendingActiveTasksToInit() { | |||
There was a problem hiding this comment.
Helper method for the fix.
lucasbru
approved these changes
Jul 8, 2024
|
The only failure we got in the last run was unrelated. |
cadonna
added a commit
to cadonna/kafka
that referenced
this pull request
Jul 8, 2024
…pache#16545) With enabled state updater tasks that are created but not initialized are stored in a set. In each poll iteration the stream thread drains that set, intializes the tasks, and adds them to the state updater. On partition lost, all active tasks are closed. This commit ensures that active tasks pending initialization in the set mentioned above are closed cleanly on partition lost.
3 tasks
cadonna
added a commit
that referenced
this pull request
Jul 8, 2024
…16545) (#16550) With enabled state updater tasks that are created but not initialized are stored in a set. In each poll iteration the stream thread drains that set, intializes the tasks, and adds them to the state updater. On partition lost, all active tasks are closed. This commit ensures that active tasks pending initialization in the set mentioned above are closed cleanly on partition lost. Reviewer: Lucas Brutschy <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With enabled state updater tasks that are created but not initialized are stored in a set. In each poll iteration the stream thread drains that set, intializes the tasks, and adds them to the state updater.
On partition lost, all active tasks are closed.
This commit ensures that active tasks pending initialization in the set mentioned above are closed cleanly on partition lost.
More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)