[CCR] Don't fail shard follow tasks in case of a non-retryable error #34404
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
martijnvg
added
WIP
:Distributed Indexing/CCR
Collaborator
|
Pinging @elastic/es-distributed |
jasontedor
approved these changes
Oct 11, 2018
Member
jasontedor
left a comment
jasontedor
left a comment
There was a problem hiding this comment.
I left one comment, looks good otherwise.
| .collect(Collectors.toList()); | ||
|
|
||
| if (shardFollowTaskIds.isEmpty()) { | ||
| listener.onFailure(new IllegalArgumentException("no shard follow tasks [" + request.getFollowIndex() + "]")); |
Member
There was a problem hiding this comment.
no shard follow tasks -> no shard follow tasks for
martijnvg
added a commit
that referenced
this pull request
Oct 16, 2018
kcm
pushed a commit
that referenced
this pull request
Oct 30, 2018
tlrx
added a commit
to tlrx/elasticsearch
that referenced
this pull request
Jul 24, 2019
Deleting a follower index does not delete its ShardFollowTasks, potentially leaving many persistent tasks in the cluster that cannot be allocated on nodes and unnecessary fill the logs. This commit adds a cluster state listener (ShardFollowTaskCleaner) that completes (with a failure) any persistent task that refers to a non existent follower index. I think that this bug has been introduced by elastic#34404: before this change the task would have been completed as failed and removed from the cluster state.
tlrx
added a commit
to tlrx/elasticsearch
that referenced
this pull request
Jul 24, 2019
Deleting a follower index does not delete its ShardFollowTasks, potentially leaving many persistent tasks in the cluster that cannot be allocated on nodes and unnecessary fill the logs. This commit adds a cluster state listener (ShardFollowTaskCleaner) that completes (with a failure) any persistent task that refers to a non existent follower index. I think that this bug has been introduced by elastic#34404: before this change the task would have been completed as failed and removed from the cluster state.
This was referenced Jul 24, 2019
tlrx
added a commit
that referenced
this pull request
Jul 25, 2019
Deleting a follower index does not delete its ShardFollowTasks, potentially leaving many persistent tasks in the cluster that cannot be allocated on nodes and unnecessary fill the logs. This commit adds a cluster state listener (ShardFollowTaskCleaner) that completes (with a failure) any persistent task that refers to a non existent follower index. I think that this bug has been introduced by #34404: before this change the task would have been completed as failed and removed from the cluster state. Backport of #44702 and #44801 on 7.x
tlrx
added a commit
that referenced
this pull request
Jul 25, 2019
Deleting a follower index does not delete its ShardFollowTasks, potentially leaving many persistent tasks in the cluster that cannot be allocated on nodes and unnecessary fill the logs. This commit adds a cluster state listener (ShardFollowTaskCleaner) that completes (with a failure) any persistent task that refers to a non existent follower index. I think that this bug has been introduced by #34404: before this change the task would have been completed as failed and removed from the cluster state. Backport of #44702 and #44801 to 7.3
tlrx
added a commit
that referenced
this pull request
Jul 25, 2019
Deleting a follower index does not delete its ShardFollowTasks, potentially leaving many persistent tasks in the cluster that cannot be allocated on nodes and unnecessary fill the logs. This commit adds a cluster state listener (ShardFollowTaskCleaner) that completes (with a failure) any persistent task that refers to a non existent follower index. I think that this bug has been introduced by #34404: before this change the task would have been completed as failed and removed from the cluster state. Backport of #44702 and #44801 on 6.8
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I had to make a few unrelated changes to the ccr stats and pause APIs. In case a follower index has been removed and we keep the shard follow tasks around then we have no way to read stats from these tasks or pause them, because the requests are routed based on the follower index.