Purge remote translog basis the latest metadata for remote-backed indexes

[ashking94]

Description

During failover or relocation of primary shard, during initialisation of RemoteFsTranslog, we read remote translog metadata and download translog and checkpoint file for the corresponding primary term and generation as present in remote translog metadata file. After a flush, we clear *.tlog and *.ckp files, but the remote metadata file continues to hold references to the deleted generations. We need to handle the same.

We should delete remote translog considering the latest metadata file uploaded. The tlog/ckp files referenced by that metadata file should not be deleted . By doing this we will be able to restore translog from that metadata.

Issues Resolved

#5845

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/10377/
• CommitID: b5c0f7f
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockIsRemovedWhenAnyNodesNotExceedHighWatermark

• URL: https://build.ci.opensearch.org/job/gradle-check/10378/
• CommitID: 0942b2d
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/10455/
• CommitID: 6ab321a
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/10460/
• CommitID: ff5de9f
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/10462/
• CommitID: 6e89c9e

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/10473/
• CommitID: 7dd90b9

[gbbafna]

Build failed due to

> Task :modules:ingest-common:jarHell FAILED
| Output for C:\hostedtoolcache\windows\Java_Adopt_jdk\11.0.18-10\x64\bin\java.exe:#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 57616 bytes for Chunk::new
# An error report file with more information is saved as:
# D:\a\OpenSearch\OpenSearch\modules\ingest-common\hs_err_pid6884.log

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

• URL: https://build.ci.opensearch.org/job/gradle-check/10482/
• CommitID: df2b6dd
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[ashking94]

Build succeeded. @gbbafna