Treat last fetch timestamp of pinned timestamp as one of the pinned timestamps

[sachinpkale]

Description

• In Bugfix in snapshot V2 restore flow #16332, we provided a solution to always keep last metadata file even if index is deleted.
• But this still does not solve a problem with following scenario:

Metadata 1 - 10:30
Pin Timestamp Fetch - 11:00
Snapshot 1 - 11:30
Snapshot 2 - 12:30
Metadata 2 - 13:10
Snapshot 3 - 13:30
DELETE INDEX

• In this case, with the existing solution, we will make sure not to delete Metadata 2 but Metadata 1 will be deleted creating data loss issues for Snapshot 1 and Snapshot 2.
• To solve this problem completely, we need to understand which metadata files need to be kept based on timestamp of pinned timestamps fetch and snapshot timestamp.
• We have following three scenarios:

// Scenario 1
Pin Timestamp Fetch
Metadata 1
Snapshot 1
Metadata 2
Delete Index
// Here, as both the metadata file are created post pin timestamp fetch, we do not delete them. This logic already exists.

// Scenario 2
Metadata 1 - 10:30
Pin Timestamp Fetch - 11:00
Snapshot 1
Metadata 2
Delete Index
// Here, Snapshot 1 refers to Metadata 1 but while deleting index, we delete Metadata 1. This creates data loss issue for Snapshot 1. Need to fix this.

// Scenario 3
Metadata 1
Snapshot 1
Pin Timestamp Fetch
Metadata 2
Delete Index
// Here, pin timestamp fetch already has snapshot 1 and metadata 1 info. So, while deleting the index, we do not delete metadata 1 as it is pinned and we do not delete metadata 2 as it was created post pin timestamp fetch. This logic already exists.

// Scenario 4
Metadata 1
Snapshot 1
Metadata 2
Pin Timestamp Fetch
Delete Index
// Here, pin timestamp fetch has a knowledge of metadata 1 referred by snapshot 1. While deleting index, we keep last metadata file, so metadata 2 is not deleted. And metadata 1 is not deleted as it is pinned. We have added metadata 2 not deleting logic as part of https://github.com/opensearch-project/OpenSearch/pull/16332.

• So, we need to fix Scenario 2. In this scenario, we need to keep the last metadata file before the pin timestamp fetch. Only keeping last md file is sufficient as any snapshot that is created will only refer to the last md file.

• In this PR, we add last fetch timestamp to the pinned timestamp list to avoid deletion of last metadata file created prior to last fetch.

• With change in this PR, we don't need changes done in Bugfix in snapshot V2 restore flow #16332 to always keep last deleted file. This improves stale data deletion as well.

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 6d86b38: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 7837786: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for 7837786: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.93%. Comparing base (f1c98a4) to head (7837786).
Report is 4 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #16392      +/-   ##
============================================
- Coverage     71.98%   71.93%   -0.05%     
+ Complexity    64810    64759      -51     
============================================
  Files          5307     5308       +1     
  Lines        302719   302727       +8     
  Branches      43734    43733       -1     
============================================
- Hits         217912   217779     -133     
- Misses        66915    67043     +128     
- Partials      17892    17905      +13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dbwiddis]

Code changes LGTM. A few comments for consideration, but feel free to have a good reason to do otherwise.

[gbbafna]

Why cluster.remote_store.pinned_timestamps.lookback_interval didn't prevent below from happening ?

Metadata 1 - 10:30
Pin Timestamp Fetch - 11:00
Snapshot 1 - 11:30
Snapshot 2 - 12:30
Metadata 2 - 13:10
Snapshot 3 - 13:30
DELETE INDEX

As per that , we we need to keep files since last successful run of scheduler

[sachinpkale]

Why cluster.remote_store.pinned_timestamps.lookback_interval didn't prevent below from happening ?

We only prevent deletion of metadata files if the metadata timestamp > last fetch timestamp - (2 * cluster.remote_store.pinned_timestamps.lookback_interval). Default lookback interval is 1 min. In this example, metadata was created 30 mins before pin timestamp fetch, so the garbage collection flow will delete Metadata 1