[Remote Translog] Deleting remote translog considering latest remote metadata by gbbafna · Pull Request #5869 · opensearch-project/OpenSearch

gbbafna · 2023-01-16T05:08:50Z

Signed-off-by: Gaurav Bafna gbbafna@amazon.com

Description

We should delete remote translog considering the latest metadata file uploaded. The tlog/ckp files referenced by that metadata file cannot be deleted . By doing this we will be able to restore translog from that metadata.

Issues Resolved

#5845

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-01-16T05:37:43Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationRelocationIT.testRelocateWhileContinuouslyIndexingAndWaitingForRefresh

URL: https://build.ci.opensearch.org/job/gradle-check/9422/
CommitID: 8643461
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov-commenter · 2023-01-16T05:39:47Z

Codecov Report

Merging #5869 (246924d) into main (32dd9e2) will increase coverage by 0.02%.
The diff coverage is 88.23%.

@@             Coverage Diff              @@
##               main    #5869      +/-   ##
============================================
+ Coverage     70.88%   70.91%   +0.02%     
- Complexity    58720    58776      +56     
============================================
  Files          4768     4768              
  Lines        280575   280584       +9     
  Branches      40514    40516       +2     
============================================
+ Hits         198881   198971      +90     
+ Misses        65334    65327       -7     
+ Partials      16360    16286      -74

Impacted Files	Coverage Δ
...rg/opensearch/index/translog/RemoteFsTranslog.java	`68.53% <81.81%> (+0.88%)`	⬆️
...h/index/translog/transfer/FileTransferTracker.java	`94.59% <100.00%> (+0.47%)`	⬆️
...dex/translog/transfer/TranslogTransferManager.java	`82.60% <100.00%> (-0.19%)`	⬇️
...port/ResponseHandlerFailureTransportException.java	`0.00% <0.00%> (-60.00%)`	⬇️
...a/org/opensearch/client/cluster/ProxyModeInfo.java	`0.00% <0.00%> (-55.00%)`	⬇️
.../java/org/opensearch/node/NodeClosedException.java	`50.00% <0.00%> (-50.00%)`	⬇️
...rch/client/transport/NoNodeAvailableException.java	`28.57% <0.00%> (-42.86%)`	⬇️
...cluster/coordination/PublishClusterStateStats.java	`33.33% <0.00%> (-37.51%)`	⬇️
...n/admin/indices/readonly/AddIndexBlockRequest.java	`35.71% <0.00%> (-35.72%)`	⬇️
...search/search/aggregations/pipeline/EwmaModel.java	`24.44% <0.00%> (-33.34%)`	⬇️
... and 449 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

server/src/main/java/org/opensearch/index/translog/RemoteFsTranslog.java

ashking94 · 2023-01-16T12:05:37Z

server/src/main/java/org/opensearch/index/translog/RemoteFsTranslog.java

+        }
+    }
+
+    public void trimUnreferencedReaders() throws IOException {


Add @Override annotation?

ashking94 · 2023-01-16T12:09:10Z

server/src/main/java/org/opensearch/index/translog/RemoteFsTranslog.java

+                logger.trace("delete remote translog generation file [{}], not referenced by metadata anymore", generation);
+                deleteRemoteGeneration(generation);
+            } else {
+                break;


Why are we breaking out of the for loop?

I can think of that we may want to retry the uploads - in which case should we start the for loop from minimum generation number that exists?

this would increase the flush times to a very high number. The clean up can be taken as an async job on its own .

ashking94 · 2023-01-16T12:11:53Z

server/src/main/java/org/opensearch/index/translog/RemoteFsTranslog.java

+            String translogFilename = Translog.getFilename(generation);
+            if (fileTransferTracker.uploaded(translogFilename)) {
+                logger.trace("delete remote translog generation file [{}], not referenced by metadata anymore", generation);
+                deleteRemoteGeneration(generation);


deleteRemoteGeneration method uses the current primaryTerm. This probably would not be handled when there is failover and the primary term has increased. Could we rely on metadata to fetch the right generation to primary term mapping for cleaning up the files?

That's a great catch @ashking94 . The issue here is the latest metadata doesn't know about the right primary term for this generation. I will create a backlog item for cleaning up the older primary term translog files . This can be done on every failover or an async job . I don't want to complicate the usual deletion flow due to this corner case .

ashking94

pls address the comments.

github-actions · 2023-01-17T04:16:00Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9452/
CommitID: 9be02e5
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-17T04:47:49Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/9453/
CommitID: 246924d

github-actions · 2023-01-18T05:35:54Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9518/
CommitID: b718c86
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-30T11:25:37Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/10157/
CommitID: 2422c10
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-30T12:03:33Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/10158/
CommitID: 950c2bc
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

Signed-off-by: Ashish Singh <ssashish@amazon.com>

github-actions · 2023-01-30T17:44:05Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/10200/
CommitID: 61344d4
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

ashking94 · 2023-01-30T17:47:31Z

Created PR #6086 for rest of the development due to access issue.

gbbafna added skip-changelog backport 2.x Backport to 2.x branch labels Jan 16, 2023

gbbafna marked this pull request as ready for review January 16, 2023 10:39

gbbafna requested review from Bukhtawar, CEHENKLE, Rishikesh1159, VachaShah, adnapibar, anasalkouz, andrross, dblock, dreamer-89, kartg, kotwanikunal, mch2, nknize, owaiskazi19, reta, ryanbogan, saratvemulapalli, setiah, shwetathareja, tlfeng and xuezhou25 as code owners January 16, 2023 10:39

gbbafna requested review from ashking94 and sachinpkale January 16, 2023 10:39

ashking94 reviewed Jan 16, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/translog/RemoteFsTranslog.java Show resolved Hide resolved

ashking94 reviewed Jan 16, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/translog/RemoteFsTranslog.java Outdated Show resolved Hide resolved

ashking94 reviewed Jan 16, 2023

View reviewed changes

ashking94 requested changes Jan 16, 2023

View reviewed changes

gbbafna changed the title ~~Deleting remote translog considering latest remote metadata~~ [Remote Translog ]Deleting remote translog considering latest remote metadata Jan 17, 2023

gbbafna changed the title ~~[Remote Translog ]Deleting remote translog considering latest remote metadata~~ [Remote Translog] Deleting remote translog considering latest remote metadata Jan 17, 2023

gbbafna force-pushed the tx-safe-remote-delete branch from 9be02e5 to 246924d Compare January 17, 2023 04:14

gbbafna marked this pull request as draft January 18, 2023 05:27

gbbafna force-pushed the tx-safe-remote-delete branch from 2422c10 to 950c2bc Compare January 30, 2023 11:42

ashking94 added Storage:Durability Issues and PRs related to the durability framework v2.6.0 'Issues and PRs related to version v2.6.0' labels Jan 30, 2023

gbbafna and others added 5 commits January 30, 2023 19:52

Deleting remote translog considering latest remote metadata

82124d1

Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

PR comments

546a551

Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

Cleaning up translogs from older primaries

307ecbd

Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

Cleaning remote translog of older primary term in async manner

78bdb15

Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

Refactor code

61344d4

Signed-off-by: Ashish Singh <ssashish@amazon.com>

ashking94 force-pushed the tx-safe-remote-delete branch from 950c2bc to 61344d4 Compare January 30, 2023 17:33

ashking94 closed this Jan 30, 2023

Poojita-Raj removed the v2.6.0 'Issues and PRs related to version v2.6.0' label Feb 21, 2023

Comments

Conversation

gbbafna commented Jan 16, 2023

Description

Issues Resolved

Check List

Uh oh!

github-actions bot commented Jan 16, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

codecov-commenter commented Jan 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

ashking94 Jan 16, 2023

Choose a reason for hiding this comment

Uh oh!

ashking94 Jan 16, 2023

Choose a reason for hiding this comment

Uh oh!

gbbafna Jan 17, 2023

Choose a reason for hiding this comment

Uh oh!

ashking94 Jan 16, 2023

Choose a reason for hiding this comment

Uh oh!

gbbafna Jan 17, 2023

Choose a reason for hiding this comment

Uh oh!

ashking94 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 17, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Jan 17, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Jan 18, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Jan 30, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Jan 30, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Jan 30, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

ashking94 commented Jan 30, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov-commenter commented Jan 16, 2023 •

edited

Loading