Use pluggable DirectoryFactory in shard store validation (draft) - #21247
Use pluggable DirectoryFactory in shard store validation (draft)#21247krishna-ggk wants to merge 53 commits into
Conversation
Store.readMetadataSnapshot() and Store.tryOpenIndex() hardcode NIOFSDirectory when reading segment metadata during shard allocation. This prevents IndexStorePlugin implementations (e.g. encryption plugins) from reading their custom-formatted segment files during the gateway allocation path after node restart, causing shards to be reported as having no valid copy. Add overloaded versions of readMetadataSnapshot() and tryOpenIndex() that accept a DirectoryFactory and IndexSettings. When provided, the factory is used to create the directory; otherwise falls back to NIOFSDirectory for backward compatibility. Update TransportNodesListShardStoreMetadataHelper and TransportNodesGatewayStartedShardHelper to resolve the DirectoryFactory from the index store type setting and pass it through. Expose directoryFactories via IndicesService.getDirectoryFactories(). Add unit tests verifying custom factory invocation, null-factory fallback, and FilterDirectory-wrapping factory behavior. Signed-off-by: Gopala Krishna A <gopalak@amazon.com>
PR Code Analyzer ❗AI-powered 'Code-Diff-Analyzer' found issues on commit f68b433. 'Diff too large, requires skip by maintainers after manual review' Pull Requests Author(s): Please update your Pull Request according to the report above. Repository Maintainer(s): You can Thanks. |
|
Seeking early feedback if there are any major concerns with the approach - @shwetathareja @itiyamas @udabhas @RajatGupta02 |
| IndexMetadata metadata = clusterService.state().metadata().index(shardId.getIndex()); | ||
| if (metadata != null) { |
There was a problem hiding this comment.
Question: Will this be expensive to be invoked in recover path?
Signed-off-by: Gopala Krishna A <gopalak@amazon.com>
PR Reviewer Guide 🔍(Review updated until commit 85f08bf)Here are some key observations to aid the review process:
|
021329c to
c71c6b9
Compare
|
Persistent review updated to latest commit c71c6b9 |
PR Code Suggestions ✨Latest suggestions up to 85f08bf Explore these optional code suggestions:
Previous suggestionsSuggestions up to commit 45e8321
Suggestions up to commit 63ce95b
Suggestions up to commit fff4b09
Suggestions up to commit c71c6b9
|
|
❌ Gradle check result for c71c6b9: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
…red storage (opensearch-project#21285) Signed-off-by: Mayank Harsh <mayankmh@amazon.com>
…NPE (opensearch-project#19725) (opensearch-project#21345) testProgressListenerExceptionsAreCaught fires the partial-merge-failure accumulator lambda directly (without the null-gated compareAndSet(null, ...) path used in production sites like TransportSearchAction and GroupedActionListener), so the AtomicReference state is null on the first call. Under a JIT tier-1 race the JVM can invoke the BinaryOperator with prev=null before the initial accumulateAndGet write is observed by the second shard thread, producing: java.lang.NullPointerException: Cannot suppress a null exception. at java.util.Objects.requireNonNull(Objects.java:246) at java.lang.Throwable.addSuppressed(Throwable.java:1103) at QueryPhaseResultConsumerTests.lambda$testProgressListenerExceptionsAreCaught$1(QueryPhaseResultConsumerTests.java:136) Matches the sibling StreamQueryPhaseResultConsumerTests which already guards with 'if (prev != null) curr.addSuppressed(prev);'. Production accumulator sites are NOT affected — both gate the lambda behind 'compareAndSet(null, e) == false', which guarantees neither argument can be null when the BinaryOperator fires. Verified: failing seed 283CB275127A6AC7 now passes; 20 consecutive iters green locally. Signed-off-by: Andre Kurait <andrekurait@gmail.com>
…#21299) * Add Lucene engine implementation for Pluggable data formats Signed-off-by: Mohit Godwani <mgodwan@amazon.com>
* Added substrait converter for the fragments Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * spotless fix Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * fixed pr comments Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> * fix failing test Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com> --------- Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>
…project#21350) ArrayList.removeAll(ArrayList) is O(n*m) due to linear contains() checks. Wrap the argument in HashSet for O(1) lookups, reducing the complexity to O(n). This was causing CPU spikes on the remote_purge thread when metadata file counts grew large. Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
Signed-off-by: Andriy Redko <drreta@gmail.com>
…#21249) Signed-off-by: Divya <divyruhil999@gmail.com> Co-authored-by: DIVYA2 <DIVYA2@ibm.com> Co-authored-by: Divya <divyruhil999@gmail.com> Co-authored-by: Andrew Ross <andrross@amazon.com>
… env param instead of org.bouncycastle.fips.approved_only (opensearch-project#21366) Signed-off-by: Craig Perkins <cwperx@amazon.com>
… LuceneTestCase (opensearch-project#21363) BlockTransferManagerTests was extending LuceneTestCase directly which causes sysout check failures since the test uses loggers that print to console. Changed to extend OpenSearchTestCase which already includes @SuppressSysoutChecks and follows the project convention for all server tests. Signed-off-by: Mayank Harsh <mayankmh@amazon.com> Co-authored-by: Mayank Harsh <mayankmh@amazon.com>
…ject#21128) * Adding CompositeMergeHandler and CompositeMergePolicy Signed-off-by: Sagar Darji <darsaga@amazon.com> # Conflicts: # sandbox/plugins/composite-engine/src/main/java/org/opensearch/composite/CompositeIndexingExecutionEngine.java * Addressing comments Signed-off-by: Sagar Darji <darsaga@amazon.com> * Split the monolithic CompositeMergeHandler into classes with clear responsibilities: Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com> * Fix up tests Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com> * Addressing commits Signed-off-by: Sagar Darji <darsaga@amazon.com> * Integrating the merge flow with the DataFormatAwareEngine Signed-off-by: Sagar Darji <darsaga@amazon.com> # Conflicts: # server/src/main/java/org/opensearch/index/engine/DataFormatAwareEngine.java # server/src/main/java/org/opensearch/index/engine/exec/coord/CatalogSnapshotManager.java # server/src/test/java/org/opensearch/index/engine/exec/coord/CatalogSnapshotManagerTests.java # Conflicts: # server/src/main/java/org/opensearch/index/engine/DataFormatAwareEngine.java * Addressed the comments Signed-off-by: Sagar Darji <darsaga@amazon.com> --------- Signed-off-by: Sagar Darji <darsaga@amazon.com> Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com> Co-authored-by: Sagar Darji <darsaga@amazon.com> Co-authored-by: Bukhtawar Khan <bukhtawa@amazon.com>
…1408) * Fix incorrect defaults in FieldStorageResolver. Signed-off-by: Marc Handalian <marc.handalian@gmail.com> * test fixes Signed-off-by: Marc Handalian <marc.handalian@gmail.com> --------- Signed-off-by: Marc Handalian <marc.handalian@gmail.com>
45e8321 to
e517a29
Compare
Store.readMetadataSnapshot() and Store.tryOpenIndex() hardcode NIOFSDirectory when reading segment metadata during shard allocation. This prevents IndexStorePlugin implementations (e.g. encryption plugins) from reading their custom-formatted segment files during the gateway allocation path after node restart, causing shards to be reported as having no valid copy. Add overloaded versions of readMetadataSnapshot() and tryOpenIndex() that accept a DirectoryFactory and IndexSettings. When provided, the factory is used to create the directory; otherwise falls back to NIOFSDirectory for backward compatibility. Update TransportNodesListShardStoreMetadataHelper and TransportNodesGatewayStartedShardHelper to resolve the DirectoryFactory from the index store type setting and pass it through. Expose directoryFactories via IndicesService.getDirectoryFactories(). Add unit tests verifying custom factory invocation, null-factory fallback, and FilterDirectory-wrapping factory behavior. Signed-off-by: Gopala Krishna A <krishna.ggk@gmail.com>
Signed-off-by: Gopala Krishna A <krishna.ggk@gmail.com>
Signed-off-by: Gopala Krishna A <krishna.ggk@gmail.com>
e605661 to
f68b433
Compare
|
Persistent review updated to latest commit 85f08bf |
|
❌ Gradle check result for 85f08bf: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Description
Store.readMetadataSnapshot()andStore.tryOpenIndex()hardcodenew NIOFSDirectory(path)when reading Lucene segment metadata during shard allocation. This preventsIndexStorePlugin.DirectoryFactoryimplementations (e.g., encryption plugins - pr-169) from reading their custom-formatted segment files during the gateway allocation path after a node restart.Impact: After a node restart, the shard allocator calls
TransportNodesListShardStoreMetadataHelperandTransportNodesGatewayStartedShardHelperto validate shard data on disk. These helpers callStore.readMetadataSnapshot()/Store.tryOpenIndex(), which create a plainNIOFSDirectory— bypassing any plugin-provided directory. If the plugin writessegments_*and.sifiles in a custom format (e.g., encrypted), the plain directory cannot read them, and the shard is reported as havingno_valid_shard_copy, preventing recovery.Changes
Store.java: Added overloadedreadMetadataSnapshot()andtryOpenIndex()that acceptDirectoryFactory+IndexSettings. Added privateopenDirectory()helper that delegates to the factory when provided, falling back toNIOFSDirectory.IndicesService.java: ExposeddirectoryFactoriesviagetDirectoryFactories()getter.TransportNodesListShardStoreMetadataHelper.java: ResolvesDirectoryFactoryfrom index settings and passes it toStore.readMetadataSnapshot().TransportNodesGatewayStartedShardHelper.java: ResolvesDirectoryFactoryfrom index settings and passes it toStore.tryOpenIndex().StoreTests.java: Added 5 unit tests covering custom factory invocation, null-factory fallback, and wrapping factory behavior.Testing
testNodeRestartWithEncryptedIndicesnow passes with encrypted.si/segments_*files (previously failed withno_valid_shard_copy)StoreTests(e.g.,testCanOpenIndex,testMetadataSnapshotStreaming) continue to passCheck List
--signoff