Skip to content

Make Snapshot Deletes Less Racy #54765

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Apr 6, 2020
Merged

Make Snapshot Deletes Less Racy #54765

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
7580ede
Make Snapshot Deletes Less Racy
original-brownbear Apr 4, 2020
6fea237
revert unrelated changes
original-brownbear Apr 4, 2020
3766083
Merge remote-tracking branch 'elastic/master' into cleanup-slm-tests
original-brownbear Apr 5, 2020
a0cdd17
fix deadlock
original-brownbear Apr 5, 2020
36aa15d
Merge remote-tracking branch 'elastic/master' into cleanup-slm-tests
original-brownbear Apr 6, 2020
7f4deaf
CR: add after delete log messages
original-brownbear Apr 6, 2020
157c938
CR: extract helper for in progrss snapshot search
original-brownbear Apr 6, 2020
ecd90a5
CR: update snapshot list safer
original-brownbear Apr 6, 2020
7c3a24e
CR: preserve other deletes
original-brownbear Apr 6, 2020
6cd5631
Merge remote-tracking branch 'elastic/master' into cleanup-slm-tests
original-brownbear Apr 6, 2020
c78ec9a
CR: list it is
original-brownbear Apr 6, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions ...rg/elasticsearch/action/admin/cluster/snapshots/delete/TransportDeleteSnapshotAction.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ public TransportDeleteSnapshotAction(TransportService transportService, ClusterS

@Override
protected String executor() {
return ThreadPool.Names.GENERIC;
return ThreadPool.Names.SAME;
Copy link
Contributor Author

@original-brownbear original-brownbear Apr 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to fork to the generic pool here any more, we instantly move to the cluster state thread now in the snapshots service. We'll use the generic pool there once we have to inspect the repository data

}

@Override
Expand All @@ -72,6 +72,6 @@ protected ClusterBlockException checkBlock(DeleteSnapshotRequest request, Cluste
protected void masterOperation(Task task, final DeleteSnapshotRequest request, ClusterState state,
final ActionListener<AcknowledgedResponse> listener) {
snapshotsService.deleteSnapshot(request.repository(), request.snapshot(),
ActionListener.map(listener, v -> new AcknowledgedResponse(true)), false);
ActionListener.map(listener, v -> new AcknowledgedResponse(true)));
}
}
3 changes: 3 additions & 0 deletions server/src/main/java/org/elasticsearch/cluster/SnapshotDeletionsInProgress.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
import org.elasticsearch.common.io.stream.Writeable;
import org.elasticsearch.common.unit.TimeValue;
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.elasticsearch.repositories.RepositoryData;
import org.elasticsearch.repositories.RepositoryOperation;
import org.elasticsearch.snapshots.Snapshot;

Expand Down Expand Up @@ -174,6 +175,8 @@ public Entry(Snapshot snapshot, long startTime, long repositoryStateId) {
this.snapshot = snapshot;
this.startTime = startTime;
this.repositoryStateId = repositoryStateId;
assert repositoryStateId > RepositoryData.EMPTY_REPO_GEN :
Copy link
Contributor Author

@original-brownbear original-brownbear Apr 6, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We would incorrectly get a -1 (empty repo gen) here when we deleted/aborted an initializing snapshot (because the repo gen is at -2 during the snapshot INIT stage). This wouldn't have caused any corruption and only made deletes fail but still, this assertion would've avoided not catching this race for so long :)

"Can't delete based on an empty or unknown repository generation but saw [" + repositoryStateId + "]";
}

public Entry(StreamInput in) throws IOException {
Expand Down
Loading