[Remote Store] Get SeqNo from Segment Info and integ tests for restore flow by sachinpkale · Pull Request #6067 · opensearch-project/OpenSearch

sachinpkale · 2023-01-30T08:38:56Z

Description

Add integration tests for remote store restore flow.
Few changes discovered as part of integ test runs.

Issues Resolved

[BUG] [Dev Setup] Assertion failure while restoring an index from a remote store #5872

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-01-30T09:06:58Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/10151/
CommitID: 1afdc3e

gbbafna · 2023-02-02T10:05:59Z

server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java

-        assert indexShard.getEngine() instanceof InternalEngine : "Expected shard with InternalEngine, got: "
-            + indexShard.getEngine().getClass();
-        final long lastRefreshedCheckpoint = ((InternalEngine) indexShard.getEngine()).lastRefreshedCheckpoint();
+        final long lastRefreshedCheckpoint = indexShard.getEngine().getMaxSeqNoFromSegmentInfos(segmentInfosSnapshot);


we should rename lastRefreshedCheckpoint to maxSeqNoFromSegmentIfnos ?

gbbafna · 2023-02-02T10:19:16Z

server/src/internalClusterTest/java/org/opensearch/remotestore/RemoteStoreIT.java

+        ensureYellowAndNoInitializingShards(INDEX_NAME);
+        ensureGreen(INDEX_NAME);
+
+        Map<String, Long> indexStats = indexData();


how do we sure that remote segment store is up to date here ? Should we check the local fs , in case we don't have any stats.

We trigger explicit refresh/flush which calls RemoteStoreRefreshListener in the same flow. So, if remote store is available, the segments should be uploaded. Test failure means some problem with the code and we need to fix it.

github-actions · 2023-02-03T03:48:08Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/10568/
CommitID: 6888f87
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Sachin Kale <kalsac@amazon.com>

sachinpkale · 2023-02-03T04:41:31Z

Gradle Check (Jenkins) Run Completed with:

* **RESULT:** FAILURE ❌

* **URL:** https://build.ci.opensearch.org/job/gradle-check/10568/

* **CommitID:** [6888f87](https://github.com/opensearch-project/OpenSearch/commit/6888f87bb146f2ed512e20e5383fa5e0a5cdcd9e)
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure [a flaky test](https://github.com/opensearch-project/OpenSearch/blob/main/DEVELOPER_GUIDE.md#flaky-tests) unrelated to your change?

org.opensearch.remotestore.RemoteStoreIT.testRemoteTranslogRestore test was flaky due to the bug in #5845. After merge of #6086, rebased the changes, should be fine now.

github-actions · 2023-02-03T05:10:36Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.geo.search.aggregations.bucket.GeoTileGridIT.testGeoShapes

URL: https://build.ci.opensearch.org/job/gradle-check/10571/
CommitID: 96a1678
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov-commenter · 2023-02-03T05:11:24Z

Codecov Report

Merging #6067 (96a1678) into main (54ce423) will decrease coverage by 0.16%.
The diff coverage is 42.85%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main    #6067      +/-   ##
============================================
- Coverage     70.90%   70.74%   -0.16%     
+ Complexity    58904    58799     -105     
============================================
  Files          4778     4778              
  Lines        281149   281150       +1     
  Branches      40622    40622              
============================================
- Hits         199346   198906     -440     
- Misses        65480    65881     +401     
- Partials      16323    16363      +40

Impacted Files	Coverage Δ
...uster/routing/allocation/IndexMetadataUpdater.java	`86.39% <0.00%> (ø)`
...a/org/opensearch/test/OpenSearchIntegTestCase.java	`55.33% <0.00%> (-1.59%)`	⬇️
...search/index/shard/RemoteStoreRefreshListener.java	`81.30% <100.00%> (+1.49%)`	⬆️
...search/indices/recovery/RecoveryTargetHandler.java	`0.00% <0.00%> (-100.00%)`	⬇️
...adonly/AddIndexBlockClusterStateUpdateRequest.java	`0.00% <0.00%> (-75.00%)`	⬇️
...a/org/opensearch/client/cluster/ProxyModeInfo.java	`0.00% <0.00%> (-60.00%)`	⬇️
...ava/org/opensearch/action/NoSuchNodeException.java	`0.00% <0.00%> (-50.00%)`	⬇️
...ch/transport/ReceiveTimeoutTransportException.java	`50.00% <0.00%> (-50.00%)`	⬇️
...main/java/org/opensearch/ingest/ProcessorInfo.java	`21.05% <0.00%> (-47.37%)`	⬇️
...rc/main/java/org/opensearch/ingest/IngestInfo.java	`34.48% <0.00%> (-44.83%)`	⬇️
... and 499 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

…e flow (#6067) * Get SeqNo from Segment Info and add integ tests for restore flow from remote store Signed-off-by: Sachin Kale <kalsac@amazon.com> (cherry picked from commit f01d3d2) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…e flow (#6067) (#6171) * Get SeqNo from Segment Info and add integ tests for restore flow from remote store Signed-off-by: Sachin Kale <kalsac@amazon.com> (cherry picked from commit f01d3d2) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…e flow (opensearch-project#6067) * Get SeqNo from Segment Info and add integ tests for restore flow from remote store Signed-off-by: Sachin Kale <kalsac@amazon.com>

sachinpkale added the skip-changelog label Jan 30, 2023

gbbafna reviewed Feb 2, 2023

View reviewed changes

sachinpkale force-pushed the remote-store-it branch from 1afdc3e to 6888f87 Compare February 3, 2023 03:16

Sachin Kale added 2 commits February 3, 2023 10:00

Add integ tests for restore flow from remote store

804697b

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Rename variable as per PR comment

96a1678

Signed-off-by: Sachin Kale <kalsac@amazon.com>

sachinpkale force-pushed the remote-store-it branch from 6888f87 to 96a1678 Compare February 3, 2023 04:35

gbbafna changed the title ~~[Remote Store] Add integ tests for restore flow from remote store~~ [Remote Store] Get SeqNo from Segment Info and integ tests for restore flow Feb 3, 2023

gbbafna approved these changes Feb 3, 2023

View reviewed changes

gbbafna merged commit f01d3d2 into opensearch-project:main Feb 3, 2023

gbbafna added the backport 2.x Backport to 2.x branch label Feb 3, 2023

opensearch-trigger-bot bot mentioned this pull request Feb 3, 2023

[Backport 2.x] [Remote Store] Get SeqNo from Segment Info and integ tests for restore flow #6171

Merged

reta mentioned this pull request Feb 8, 2023

[BUG] Flaky test org.opensearch.remotestore.RemoteStoreIT.testRemoteStoreRestoreFromRemoteSegmentStore #6248

Closed

sachinpkale mentioned this pull request Feb 20, 2023

[BUG] [Dev Setup] Assertion failure while restoring an index from a remote store #5872

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

[Remote Store] Get SeqNo from Segment Info and integ tests for restore flow#6067

[Remote Store] Get SeqNo from Segment Info and integ tests for restore flow#6067
gbbafna merged 2 commits intoopensearch-project:mainfrom
sachinpkale:remote-store-it

sachinpkale commented Jan 30, 2023 •

edited

Loading

Uh oh!

github-actions bot commented Jan 30, 2023

Uh oh!

gbbafna Feb 2, 2023

Uh oh!

gbbafna Feb 2, 2023

Uh oh!

sachinpkale Feb 3, 2023

Uh oh!

github-actions bot commented Feb 3, 2023

Uh oh!

sachinpkale commented Feb 3, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Feb 3, 2023

Uh oh!

codecov-commenter commented Feb 3, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

Conversation

sachinpkale commented Jan 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues Resolved

Check List

Uh oh!

github-actions bot commented Jan 30, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

gbbafna Feb 2, 2023

Choose a reason for hiding this comment

Uh oh!

gbbafna Feb 2, 2023

Choose a reason for hiding this comment

Uh oh!

sachinpkale Feb 3, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 3, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

sachinpkale commented Feb 3, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Feb 3, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

codecov-commenter commented Feb 3, 2023

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sachinpkale commented Jan 30, 2023 •

edited

Loading