Skip to content

Use pluggable DirectoryFactory in shard store validation (draft) - #21247

Open
krishna-ggk wants to merge 53 commits into
opensearch-project:mainfrom
krishna-ggk:pluggable-directory-store-validation
Open

Use pluggable DirectoryFactory in shard store validation (draft)#21247
krishna-ggk wants to merge 53 commits into
opensearch-project:mainfrom
krishna-ggk:pluggable-directory-store-validation

Conversation

@krishna-ggk

@krishna-ggk krishna-ggk commented Apr 16, 2026

Copy link
Copy Markdown

Description

Store.readMetadataSnapshot() and Store.tryOpenIndex() hardcode new NIOFSDirectory(path) when reading Lucene segment metadata during shard allocation. This prevents IndexStorePlugin.DirectoryFactory implementations (e.g., encryption plugins - pr-169) from reading their custom-formatted segment files during the gateway allocation path after a node restart.

Impact: After a node restart, the shard allocator calls TransportNodesListShardStoreMetadataHelper and TransportNodesGatewayStartedShardHelper to validate shard data on disk. These helpers call Store.readMetadataSnapshot() / Store.tryOpenIndex(), which create a plain NIOFSDirectory — bypassing any plugin-provided directory. If the plugin writes segments_* and .si files in a custom format (e.g., encrypted), the plain directory cannot read them, and the shard is reported as having no_valid_shard_copy, preventing recovery.

Changes

  • Store.java: Added overloaded readMetadataSnapshot() and tryOpenIndex() that accept DirectoryFactory + IndexSettings. Added private openDirectory() helper that delegates to the factory when provided, falling back to NIOFSDirectory.
  • IndicesService.java: Exposed directoryFactories via getDirectoryFactories() getter.
  • TransportNodesListShardStoreMetadataHelper.java: Resolves DirectoryFactory from index settings and passes it to Store.readMetadataSnapshot().
  • TransportNodesGatewayStartedShardHelper.java: Resolves DirectoryFactory from index settings and passes it to Store.tryOpenIndex().
  • StoreTests.java: Added 5 unit tests covering custom factory invocation, null-factory fallback, and wrapping factory behavior.

Testing

  • All new unit tests pass
  • Verified with opensearch-storage-encryption plugin: testNodeRestartWithEncryptedIndices now passes with encrypted .si/segments_* files (previously failed with no_valid_shard_copy)
  • Existing StoreTests (e.g., testCanOpenIndex, testMetadataSnapshotStreaming) continue to pass

Check List

  • New functionality includes testing
  • Commits are signed per the DCO using --signoff

Store.readMetadataSnapshot() and Store.tryOpenIndex() hardcode
NIOFSDirectory when reading segment metadata during shard allocation.
This prevents IndexStorePlugin implementations (e.g. encryption plugins)
from reading their custom-formatted segment files during the gateway
allocation path after node restart, causing shards to be reported as
having no valid copy.

Add overloaded versions of readMetadataSnapshot() and tryOpenIndex()
that accept a DirectoryFactory and IndexSettings. When provided, the
factory is used to create the directory; otherwise falls back to
NIOFSDirectory for backward compatibility.

Update TransportNodesListShardStoreMetadataHelper and
TransportNodesGatewayStartedShardHelper to resolve the DirectoryFactory
from the index store type setting and pass it through.

Expose directoryFactories via IndicesService.getDirectoryFactories().

Add unit tests verifying custom factory invocation, null-factory
fallback, and FilterDirectory-wrapping factory behavior.

Signed-off-by: Gopala Krishna A <gopalak@amazon.com>
@github-actions

github-actions Bot commented Apr 16, 2026

Copy link
Copy Markdown
Contributor

PR Code Analyzer ❗

AI-powered 'Code-Diff-Analyzer' found issues on commit f68b433.

'Diff too large, requires skip by maintainers after manual review'


Pull Requests Author(s): Please update your Pull Request according to the report above.

Repository Maintainer(s): You can bypass diff analyzer by adding label skip-diff-analyzer after reviewing the changes carefully, then re-run failed actions. To re-enable the analyzer, remove the label, then re-run all actions.


⚠️ Note: The Code-Diff-Analyzer helps protect against potentially harmful code patterns. Please ensure you have thoroughly reviewed the changes beforehand.

Thanks.

@krishna-ggk krishna-ggk changed the title Use pluggable DirectoryFactory in shard store validation Use pluggable DirectoryFactory in shard store validation (draft) Apr 16, 2026
@krishna-ggk
krishna-ggk marked this pull request as draft April 16, 2026 11:33
@krishna-ggk

Copy link
Copy Markdown
Author

Seeking early feedback if there are any major concerns with the approach - @shwetathareja @itiyamas @udabhas @RajatGupta02

Comment on lines +161 to +162
IndexMetadata metadata = clusterService.state().metadata().index(shardId.getIndex());
if (metadata != null) {

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: Will this be expensive to be invoked in recover path?

Signed-off-by: Gopala Krishna A <gopalak@amazon.com>
@github-actions

github-actions Bot commented Apr 21, 2026

Copy link
Copy Markdown
Contributor

PR Reviewer Guide 🔍

(Review updated until commit 85f08bf)

Here are some key observations to aid the review process:

🧪 PR contains tests
🔒 No security concerns identified
✅ No TODO sections
🔀 Multiple PR themes

Sub-PR theme: Add pluggable DirectoryFactory support to Store read methods

Relevant files:

  • server/src/main/java/org/opensearch/index/store/Store.java
  • server/src/test/java/org/opensearch/index/store/StoreTests.java

Sub-PR theme: Wire DirectoryFactory into gateway shard validation helpers

Relevant files:

  • server/src/main/java/org/opensearch/indices/IndicesService.java
  • server/src/main/java/org/opensearch/gateway/TransportNodesGatewayStartedShardHelper.java
  • server/src/main/java/org/opensearch/indices/store/TransportNodesListShardStoreMetadataHelper.java

⚡ Recommended focus areas for review

ShardPath Construction

In openDirectory(), the ShardPath is constructed using indexLocation.getParent() for both shardStatePath and dataPath. This assumes the index directory is always one level below the shard root. If the actual shard layout differs (e.g., custom data paths or shadow replicas), the constructed ShardPath may be incorrect, causing the DirectoryFactory to open the wrong directory or fail.

if (directoryFactory != null && indexSettings != null) {
    ShardPath shardPath = new ShardPath(false, indexLocation.getParent(), indexLocation.getParent(), shardId);
    return directoryFactory.newDirectory(indexSettings, shardPath);
Mutable Map Exposure

getDirectoryFactories() returns the internal directoryFactories map directly without wrapping it in an unmodifiable view. External callers could potentially modify the map, affecting all subsequent directory factory lookups.

public Map<String, IndexStorePlugin.DirectoryFactory> getDirectoryFactories() {
    return directoryFactories;
}
Duplicate Logic

resolveDirectoryFactory() and resolveIndexSettings() in TransportNodesGatewayStartedShardHelper are nearly identical to the same methods in TransportNodesListShardStoreMetadataHelper. This duplication increases maintenance burden and risk of divergence. Consider extracting to a shared utility class.

private static IndexStorePlugin.DirectoryFactory resolveDirectoryFactory(
    ShardId shardId,
    IndicesService indicesService,
    Settings settings,
    ClusterService clusterService
) {
    IndexSettings indexSettings = resolveIndexSettings(shardId, indicesService, settings, clusterService);
    if (indexSettings == null) {
        return null;
    }
    String storeType = IndexModule.INDEX_STORE_TYPE_SETTING.get(indexSettings.getSettings());
    if (storeType.isEmpty()) {
        return null;
    }
    return indicesService.getDirectoryFactories().get(storeType);
}

/**
 * Resolves the {@link IndexSettings} for the given shard's index.
 */
private static IndexSettings resolveIndexSettings(
    ShardId shardId,
    IndicesService indicesService,
    Settings settings,
    ClusterService clusterService
) {
    IndexService indexService = indicesService.indexService(shardId.getIndex());
    if (indexService != null) {
        return indexService.getIndexSettings();
    }
    IndexMetadata metadata = clusterService.state().metadata().index(shardId.getIndex());
    if (metadata != null) {
        return new IndexSettings(metadata, settings);
    }
    return null;
}
Double resolveIndexSettings Call

resolveDirectoryFactory() calls resolveIndexSettings() internally, and then the call site also calls resolveIndexSettings() separately. This results in two redundant cluster state lookups per shard validation. The index settings should be resolved once and reused.

Store.tryOpenIndex(
    shardPath.resolveIndex(),
    shardId,
    nodeEnv::shardLock,
    logger,
    resolveDirectoryFactory(shardId, indicesService, settings, clusterService),
    resolveIndexSettings(shardId, indicesService, settings, clusterService)
);
Null IndexSettings with Non-null Factory

In openDirectory(), if directoryFactory is non-null but indexSettings is null (e.g., when cluster state has no metadata for the index), the method silently falls back to NIOFSDirectory. This silent fallback could mask misconfiguration for encrypted indices, potentially causing data corruption or silent read failures instead of a clear error.

if (directoryFactory != null && indexSettings != null) {
    ShardPath shardPath = new ShardPath(false, indexLocation.getParent(), indexLocation.getParent(), shardId);
    return directoryFactory.newDirectory(indexSettings, shardPath);
}
return new NIOFSDirectory(indexLocation);

@krishna-ggk
krishna-ggk force-pushed the pluggable-directory-store-validation branch from 021329c to c71c6b9 Compare April 21, 2026 08:31
@github-actions

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit c71c6b9

@github-actions

github-actions Bot commented Apr 21, 2026

Copy link
Copy Markdown
Contributor

PR Code Suggestions ✨

Latest suggestions up to 85f08bf

Explore these optional code suggestions:

CategorySuggestion                                                                                                                                    Impact
General
Avoid redundant cluster state lookups

resolveDirectoryFactory calls resolveIndexSettings internally, but the caller in
getShardInfoOnLocalNode also calls resolveIndexSettings separately. This means
resolveIndexSettings is invoked twice for the same shard, which involves a cluster
state lookup each time. The two methods should be refactored so
resolveDirectoryFactory accepts an already-resolved IndexSettings to avoid the
redundant lookup.

server/src/main/java/org/opensearch/gateway/TransportNodesGatewayStartedShardHelper.java [131-146]

 private static IndexStorePlugin.DirectoryFactory resolveDirectoryFactory(
-    ShardId shardId,
     IndicesService indicesService,
-    Settings settings,
-    ClusterService clusterService
+    IndexSettings indexSettings
 ) {
-    IndexSettings indexSettings = resolveIndexSettings(shardId, indicesService, settings, clusterService);
     if (indexSettings == null) {
         return null;
     }
     String storeType = IndexModule.INDEX_STORE_TYPE_SETTING.get(indexSettings.getSettings());
     if (storeType.isEmpty()) {
         return null;
     }
     return indicesService.getDirectoryFactories().get(storeType);
 }
Suggestion importance[1-10]: 5

__

Why: The suggestion correctly identifies that resolveIndexSettings is called twice (once inside resolveDirectoryFactory and once directly by the caller), causing redundant cluster state lookups. Refactoring to pass the already-resolved IndexSettings would be a valid optimization, though the same duplication exists in TransportNodesListShardStoreMetadataHelper.java too.

Low
Return unmodifiable map to prevent mutation

The method returns the internal directoryFactories map directly, exposing a mutable
reference. If the underlying map is mutable, callers could modify it
unintentionally. Return an unmodifiable view to prevent accidental mutation.

server/src/main/java/org/opensearch/indices/IndicesService.java [1424-1426]

 public Map<String, IndexStorePlugin.DirectoryFactory> getDirectoryFactories() {
-    return directoryFactories;
+    return Collections.unmodifiableMap(directoryFactories);
 }
Suggestion importance[1-10]: 4

__

Why: Returning an unmodifiable view of directoryFactories is a good defensive practice to prevent accidental mutation by callers. However, the impact is limited since this is an internal service method and the risk of unintended mutation is low in practice.

Low
Document and assert shard path structure assumption

The ShardPath is constructed with indexLocation.getParent() for both the shard path
and the data path, but indexLocation already points to the index subdirectory inside
the shard directory. Using indexLocation.getParent() as the shard root is correct,
but this assumption may silently break if the directory structure differs. More
critically, ShardPath.resolveIndex() appends "index" to the shard path, so if
indexLocation is already the index directory, the resolved path will be correct —
but this implicit coupling is fragile. Consider adding an assertion or comment to
document this invariant, or derive the path more explicitly.

server/src/main/java/org/opensearch/index/store/Store.java [695-706]

 private static Directory openDirectory(
     Path indexLocation,
     IndexStorePlugin.DirectoryFactory directoryFactory,
     IndexSettings indexSettings,
     ShardId shardId
 ) throws IOException {
     if (directoryFactory != null && indexSettings != null) {
-        ShardPath shardPath = new ShardPath(false, indexLocation.getParent(), indexLocation.getParent(), shardId);
+        // indexLocation points to the 'index' subdirectory; its parent is the shard root
+        Path shardRoot = indexLocation.getParent();
+        assert shardRoot.resolve("index").equals(indexLocation)
+            : "Expected indexLocation to be the 'index' subdirectory of the shard root";
+        ShardPath shardPath = new ShardPath(false, shardRoot, shardRoot, shardId);
         return directoryFactory.newDirectory(indexSettings, shardPath);
     }
     return new NIOFSDirectory(indexLocation);
 }
Suggestion importance[1-10]: 3

__

Why: The suggestion adds an assertion to document the implicit coupling between indexLocation and the shard root path. While this improves code clarity, it's a minor defensive programming improvement and the improved_code only adds a comment and assertion without changing logic.

Low

Previous suggestions

Suggestions up to commit 45e8321
CategorySuggestion                                                                                                                                    Impact
General
Avoid duplicate cluster state lookups per shard

resolveDirectoryFactory calls resolveIndexSettings internally, and then the call
site also calls resolveIndexSettings separately. This means resolveIndexSettings is
invoked twice per shard validation, which involves a cluster state lookup each time.
The two methods should be refactored so resolveIndexSettings is called only once and
its result is shared, avoiding redundant cluster state access.

server/src/main/java/org/opensearch/gateway/TransportNodesGatewayStartedShardHelper.java [131-146]

+// At the call site, call resolveIndexSettings once and pass the result to both:
+IndexSettings indexSettings = resolveIndexSettings(shardId, indicesService, settings, clusterService);
+Store.tryOpenIndex(
+    shardPath.resolveIndex(),
+    shardId,
+    nodeEnv::shardLock,
+    logger,
+    resolveDirectoryFactory(indexSettings, indicesService),
+    indexSettings
+);
+
 private static IndexStorePlugin.DirectoryFactory resolveDirectoryFactory(
-    ShardId shardId,
-    IndicesService indicesService,
-    Settings settings,
-    ClusterService clusterService
+    IndexSettings indexSettings,
+    IndicesService indicesService
 ) {
-    IndexSettings indexSettings = resolveIndexSettings(shardId, indicesService, settings, clusterService);
     if (indexSettings == null) {
         return null;
     }
     String storeType = IndexModule.INDEX_STORE_TYPE_SETTING.get(indexSettings.getSettings());
     if (storeType.isEmpty()) {
         return null;
     }
     return indicesService.getDirectoryFactories().get(storeType);
 }
Suggestion importance[1-10]: 5

__

Why: The duplicate call to resolveIndexSettings (once inside resolveDirectoryFactory and once at the call site) is a real inefficiency involving cluster state lookups. The refactoring suggestion is valid and would improve performance, though the same duplication exists in TransportNodesListShardStoreMetadataHelper.java as well and would need to be addressed there too.

Low
Return unmodifiable map to prevent external mutation

The method returns the internal directoryFactories map directly, exposing a mutable
reference. Callers could accidentally or maliciously modify the map. Return an
unmodifiable view to prevent external mutation.

server/src/main/java/org/opensearch/indices/IndicesService.java [1424-1426]

 public Map<String, IndexStorePlugin.DirectoryFactory> getDirectoryFactories() {
-    return directoryFactories;
+    return Collections.unmodifiableMap(directoryFactories);
 }
Suggestion importance[1-10]: 5

__

Why: Returning a direct reference to the internal directoryFactories map is a valid defensive programming concern. Wrapping with Collections.unmodifiableMap() is a straightforward fix that prevents accidental external mutation of the internal state.

Low
Possible issue
Validate ShardPath resolves to correct index directory

The ShardPath is constructed using indexLocation.getParent() for both the shard
state path and the data path. However, indexLocation is the index subdirectory
inside the shard directory, so indexLocation.getParent() is the shard root. This is
correct for the shard path, but the ShardPath constructor expects the data path to
be the shard root (not the index subdirectory). Verify that shardPath.resolveIndex()
inside the factory will correctly resolve back to indexLocation; if the shard UUID
or custom data path is involved, this assumption may break and the factory could
open the wrong directory.

server/src/main/java/org/opensearch/index/store/Store.java [695-706]

 private static Directory openDirectory(
     Path indexLocation,
     IndexStorePlugin.DirectoryFactory directoryFactory,
     IndexSettings indexSettings,
     ShardId shardId
 ) throws IOException {
     if (directoryFactory != null && indexSettings != null) {
-        ShardPath shardPath = new ShardPath(false, indexLocation.getParent(), indexLocation.getParent(), shardId);
+        // indexLocation is the "index" subdirectory; its parent is the shard root
+        Path shardRoot = indexLocation.getParent();
+        ShardPath shardPath = new ShardPath(false, shardRoot, shardRoot, shardId);
+        // Sanity check: the factory must resolve back to the same index directory
+        assert shardPath.resolveIndex().equals(indexLocation)
+            : "ShardPath resolves to " + shardPath.resolveIndex() + " but expected " + indexLocation;
         return directoryFactory.newDirectory(indexSettings, shardPath);
     }
     return new NIOFSDirectory(indexLocation);
 }
Suggestion importance[1-10]: 4

__

Why: The concern about ShardPath construction is valid - using indexLocation.getParent() for both paths assumes a standard directory layout. However, the improved_code only adds an assertion comment without changing the actual logic, making it more of a verification suggestion. The existing code in tests uses createShardIndexDir which creates the proper structure, suggesting the author is aware of this constraint.

Low
Suggestions up to commit 63ce95b
CategorySuggestion                                                                                                                                    Impact
General
Avoid redundant double resolution of index settings

resolveDirectoryFactory calls resolveIndexSettings internally, but the caller in
getShardInfoOnLocalNode also calls resolveIndexSettings separately. This results in
resolveIndexSettings being called twice (once inside resolveDirectoryFactory and
once explicitly), which is redundant and potentially inconsistent if the cluster
state changes between calls. The factory resolution should accept the
already-resolved IndexSettings as a parameter to avoid the double call.

server/src/main/java/org/opensearch/gateway/TransportNodesGatewayStartedShardHelper.java [131-146]

 private static IndexStorePlugin.DirectoryFactory resolveDirectoryFactory(
-    ShardId shardId,
     IndicesService indicesService,
-    Settings settings,
-    ClusterService clusterService
+    IndexSettings indexSettings
 ) {
-    IndexSettings indexSettings = resolveIndexSettings(shardId, indicesService, settings, clusterService);
     if (indexSettings == null) {
         return null;
     }
     String storeType = IndexModule.INDEX_STORE_TYPE_SETTING.get(indexSettings.getSettings());
     if (storeType.isEmpty()) {
         return null;
     }
     return indicesService.getDirectoryFactories().get(storeType);
 }
Suggestion importance[1-10]: 5

__

Why: The suggestion correctly identifies that resolveIndexSettings is called twice — once inside resolveDirectoryFactory and once at the call site. Refactoring resolveDirectoryFactory to accept pre-resolved IndexSettings would eliminate redundancy and potential inconsistency. The same pattern exists in TransportNodesListShardStoreMetadataHelper.java as well.

Low
Return unmodifiable map to prevent external mutation

The method returns the internal directoryFactories map directly, which exposes the
mutable internal state of IndicesService. Callers could accidentally (or
maliciously) modify the map. Return an unmodifiable view to prevent external
mutation.

server/src/main/java/org/opensearch/indices/IndicesService.java [1424-1426]

 public Map<String, IndexStorePlugin.DirectoryFactory> getDirectoryFactories() {
-    return directoryFactories;
+    return Collections.unmodifiableMap(directoryFactories);
 }
Suggestion importance[1-10]: 5

__

Why: Returning the internal directoryFactories map directly exposes mutable state. Wrapping it with Collections.unmodifiableMap() is a good defensive practice to prevent accidental modification by callers, which is a valid correctness concern.

Low
Add assertion to validate directory path assumption

The ShardPath is constructed using indexLocation.getParent() for both the shard
state path and the data path. However, indexLocation already points to the index
subdirectory inside the shard directory, so indexLocation.getParent() is the shard
root — this is correct for the shard path. But ShardPath.resolveIndex() internally
appends "index" to the data path, so passing indexLocation.getParent() is correct.
However, if indexLocation is not always the index subdirectory (e.g., it could be
the shard root itself in some code paths), this assumption could break. You should
add a guard or assertion to ensure indexLocation is indeed the index subdirectory
before calling getParent().

server/src/main/java/org/opensearch/index/store/Store.java [695-706]

 private static Directory openDirectory(
     Path indexLocation,
     IndexStorePlugin.DirectoryFactory directoryFactory,
     IndexSettings indexSettings,
     ShardId shardId
 ) throws IOException {
     if (directoryFactory != null && indexSettings != null) {
+        // indexLocation must be the 'index' subdirectory; its parent is the shard root
+        assert indexLocation.getFileName() != null && indexLocation.getFileName().toString().equals("index")
+            : "Expected indexLocation to be the 'index' subdirectory, got: " + indexLocation;
         ShardPath shardPath = new ShardPath(false, indexLocation.getParent(), indexLocation.getParent(), shardId);
         return directoryFactory.newDirectory(indexSettings, shardPath);
     }
     return new NIOFSDirectory(indexLocation);
 }
Suggestion importance[1-10]: 3

__

Why: The assertion adds a defensive check for the assumption that indexLocation is always the index subdirectory. While this is a valid concern, the PR consistently passes shardPath.resolveIndex() which always returns the index subdirectory, making this a low-impact defensive measure. Assertions are also disabled by default in production JVMs.

Low
Suggestions up to commit fff4b09
CategorySuggestion                                                                                                                                    Impact
General
Avoid duplicate index settings resolution calls

resolveDirectoryFactory calls resolveIndexSettings internally, but the caller in
getShardInfoOnLocalNode also calls resolveIndexSettings separately, resulting in two
redundant calls to resolveIndexSettings for the same shard. This is inefficient and
could be inconsistent if cluster state changes between calls. Refactor so that
resolveIndexSettings is called once and the result is passed to both usages.

server/src/main/java/org/opensearch/gateway/TransportNodesGatewayStartedShardHelper.java [131-146]

 private static IndexStorePlugin.DirectoryFactory resolveDirectoryFactory(
     ShardId shardId,
     IndicesService indicesService,
-    Settings settings,
-    ClusterService clusterService
+    IndexSettings indexSettings
 ) {
-    IndexSettings indexSettings = resolveIndexSettings(shardId, indicesService, settings, clusterService);
     if (indexSettings == null) {
         return null;
     }
     String storeType = IndexModule.INDEX_STORE_TYPE_SETTING.get(indexSettings.getSettings());
     if (storeType.isEmpty()) {
         return null;
     }
     return indicesService.getDirectoryFactories().get(storeType);
 }
Suggestion importance[1-10]: 5

__

Why: The suggestion correctly identifies that resolveIndexSettings is called twice — once inside resolveDirectoryFactory and once at the call site in getShardInfoOnLocalNode. Refactoring to pass the already-resolved IndexSettings avoids redundant computation and potential inconsistency, though the same duplication exists in TransportNodesListShardStoreMetadataHelper.java and would need to be addressed there too.

Low
Return unmodifiable map to protect internal state

The method returns the internal directoryFactories map directly, exposing the
mutable internal state of IndicesService. This could allow callers to modify the
map. Return an unmodifiable view of the map to prevent unintended mutations.

server/src/main/java/org/opensearch/indices/IndicesService.java [1417-1419]

 public Map<String, IndexStorePlugin.DirectoryFactory> getDirectoryFactories() {
-    return directoryFactories;
+    return Collections.unmodifiableMap(directoryFactories);
 }
Suggestion importance[1-10]: 5

__

Why: Returning a direct reference to the internal directoryFactories map exposes mutable state. Wrapping it with Collections.unmodifiableMap is a straightforward defensive improvement that prevents accidental or malicious modification of the internal map by callers.

Low
Document and validate shard path construction assumption

The ShardPath is constructed using indexLocation.getParent() for both the shard path
and the data path, but indexLocation already points to the index subdirectory inside
the shard directory. The ShardPath constructor expects the shard state path and the
data path (the shard directory itself), not the index subdirectory. Using
indexLocation.getParent() is correct here, but it's worth verifying that
resolveIndex() on this ShardPath will return the original indexLocation. If
ShardPath.resolveIndex() appends "index" to the data path, then passing
indexLocation.getParent() is correct; however, if indexLocation is not always a
direct child named "index", this assumption may break. Add an assertion or comment
to document this invariant.

server/src/main/java/org/opensearch/index/store/Store.java [695-706]

 private static Directory openDirectory(
     Path indexLocation,
     IndexStorePlugin.DirectoryFactory directoryFactory,
     IndexSettings indexSettings,
     ShardId shardId
 ) throws IOException {
     if (directoryFactory != null && indexSettings != null) {
-        ShardPath shardPath = new ShardPath(false, indexLocation.getParent(), indexLocation.getParent(), shardId);
+        // indexLocation points to the "index" subdirectory; shardDir is its parent
+        Path shardDir = indexLocation.getParent();
+        assert shardDir.resolve(ShardPath.INDEX_FOLDER_NAME).equals(indexLocation)
+            : "indexLocation must be the 'index' subdirectory of the shard directory";
+        ShardPath shardPath = new ShardPath(false, shardDir, shardDir, shardId);
         return directoryFactory.newDirectory(indexSettings, shardPath);
     }
     return new NIOFSDirectory(indexLocation);
 }
Suggestion importance[1-10]: 3

__

Why: The suggestion asks to add an assertion to document the invariant that indexLocation is always the "index" subdirectory. While this is a valid defensive programming practice, the improved_code is functionally equivalent to the existing_code (only adds a comment and assertion), making it a low-impact improvement.

Low
Suggestions up to commit c71c6b9
CategorySuggestion                                                                                                                                    Impact
General
Avoid duplicate index settings resolution calls

resolveDirectoryFactory calls resolveIndexSettings internally, but the caller in
getShardInfoOnLocalNode also calls resolveIndexSettings separately, resulting in two
redundant calls to resolveIndexSettings for the same shard. This is inefficient and
could lead to inconsistency if the cluster state changes between calls. Consider
refactoring so resolveIndexSettings is called once and the result is passed to both
usages.

server/src/main/java/org/opensearch/gateway/TransportNodesGatewayStartedShardHelper.java [131-146]

 private static IndexStorePlugin.DirectoryFactory resolveDirectoryFactory(
     ShardId shardId,
     IndicesService indicesService,
-    Settings settings,
-    ClusterService clusterService
+    IndexSettings indexSettings
 ) {
-    IndexSettings indexSettings = resolveIndexSettings(shardId, indicesService, settings, clusterService);
     if (indexSettings == null) {
         return null;
     }
     String storeType = IndexModule.INDEX_STORE_TYPE_SETTING.get(indexSettings.getSettings());
     if (storeType.isEmpty()) {
         return null;
     }
     return indicesService.getDirectoryFactories().get(storeType);
 }
Suggestion importance[1-10]: 5

__

Why: The suggestion correctly identifies that resolveIndexSettings is called twice — once inside resolveDirectoryFactory and once separately in the caller. Refactoring to pass the already-resolved IndexSettings avoids redundant computation and potential inconsistency, though the performance impact is minor.

Low
Return unmodifiable map to prevent external mutation

The method returns the internal directoryFactories map directly, exposing a mutable
reference. If the underlying map is mutable, callers could inadvertently modify it.
Return an unmodifiable view to protect the internal state.

server/src/main/java/org/opensearch/indices/IndicesService.java [1417-1419]

 public Map<String, IndexStorePlugin.DirectoryFactory> getDirectoryFactories() {
-    return directoryFactories;
+    return Collections.unmodifiableMap(directoryFactories);
 }
Suggestion importance[1-10]: 5

__

Why: Returning a direct reference to the internal directoryFactories map exposes it to external mutation. Wrapping it with Collections.unmodifiableMap is a valid defensive programming practice to protect internal state integrity.

Low
Add assertion to validate directory path structure assumption

The ShardPath is constructed using indexLocation.getParent() for both the shard
state path and the data path. However, indexLocation already points to the index
subdirectory inside the shard directory, so indexLocation.getParent() is the shard
root — this is correct for the shard path, but the resolveIndex() call inside
DirectoryFactory.newDirectory() will append "index" again, resulting in the correct
path. However, if indexLocation is not structured as /index, this assumption breaks
silently. The construction should be validated or documented clearly to avoid
misuse.

server/src/main/java/org/opensearch/index/store/Store.java [695-706]

 private static Directory openDirectory(
     Path indexLocation,
     IndexStorePlugin.DirectoryFactory directoryFactory,
     IndexSettings indexSettings,
     ShardId shardId
 ) throws IOException {
     if (directoryFactory != null && indexSettings != null) {
-        ShardPath shardPath = new ShardPath(false, indexLocation.getParent(), indexLocation.getParent(), shardId);
+        // indexLocation is expected to be <shardRoot>/index; shardRoot is its parent
+        Path shardRoot = indexLocation.getParent();
+        assert shardRoot != null && indexLocation.getFileName().toString().equals("index")
+            : "indexLocation must point to the 'index' subdirectory of the shard root, got: " + indexLocation;
+        ShardPath shardPath = new ShardPath(false, shardRoot, shardRoot, shardId);
         return directoryFactory.newDirectory(indexSettings, shardPath);
     }
     return new NIOFSDirectory(indexLocation);
 }
Suggestion importance[1-10]: 3

__

Why: The suggestion adds a defensive assertion to validate that indexLocation points to the index subdirectory, which is a reasonable safety check. However, the improved_code only adds a comment and assertion without changing logic, making it a low-impact documentation/validation improvement.

Low

@github-actions

Copy link
Copy Markdown
Contributor

❌ Gradle check result for c71c6b9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

MayankHarsh03 and others added 11 commits April 28, 2026 00:58
…red storage (opensearch-project#21285)

Signed-off-by: Mayank Harsh <mayankmh@amazon.com>
…NPE (opensearch-project#19725) (opensearch-project#21345)

testProgressListenerExceptionsAreCaught fires the partial-merge-failure accumulator lambda directly (without the null-gated compareAndSet(null, ...) path used in production sites like TransportSearchAction and GroupedActionListener), so the AtomicReference state is null on the first call. Under a JIT tier-1 race the JVM can invoke the BinaryOperator with prev=null before the initial accumulateAndGet write is observed by the second shard thread, producing:

java.lang.NullPointerException: Cannot suppress a null exception.
    at java.util.Objects.requireNonNull(Objects.java:246)
    at java.lang.Throwable.addSuppressed(Throwable.java:1103)
    at QueryPhaseResultConsumerTests.lambda$testProgressListenerExceptionsAreCaught$1(QueryPhaseResultConsumerTests.java:136)

Matches the sibling StreamQueryPhaseResultConsumerTests which already guards with 'if (prev != null) curr.addSuppressed(prev);'.

Production accumulator sites are NOT affected — both gate the lambda behind 'compareAndSet(null, e) == false', which guarantees neither argument can be null when the BinaryOperator fires.

Verified: failing seed 283CB275127A6AC7 now passes; 20 consecutive iters green locally.

Signed-off-by: Andre Kurait <andrekurait@gmail.com>
…#21299)

* Add Lucene engine implementation for Pluggable data formats

Signed-off-by: Mohit Godwani <mgodwan@amazon.com>
* Added substrait converter for the fragments

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

* spotless fix

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

* fixed pr comments

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

* fix failing test

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

---------

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>
…project#21350)

ArrayList.removeAll(ArrayList) is O(n*m) due to linear contains() checks. Wrap the argument in HashSet for O(1) lookups, reducing the complexity to O(n). This was causing  CPU spikes on the remote_purge thread when metadata file counts grew large.

Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
Signed-off-by: Andriy Redko <drreta@gmail.com>
…#21249)

Signed-off-by: Divya <divyruhil999@gmail.com>
Co-authored-by: DIVYA2 <DIVYA2@ibm.com>
Co-authored-by: Divya <divyruhil999@gmail.com>
Co-authored-by: Andrew Ross <andrross@amazon.com>
… env param instead of org.bouncycastle.fips.approved_only (opensearch-project#21366)

Signed-off-by: Craig Perkins <cwperx@amazon.com>
… LuceneTestCase (opensearch-project#21363)

BlockTransferManagerTests was extending LuceneTestCase directly which causes
sysout check failures since the test uses loggers that print to console.
Changed to extend OpenSearchTestCase which already includes
@SuppressSysoutChecks and follows the project convention for all server tests.

Signed-off-by: Mayank Harsh <mayankmh@amazon.com>
Co-authored-by: Mayank Harsh <mayankmh@amazon.com>
…ject#21128)

* Adding CompositeMergeHandler and CompositeMergePolicy

Signed-off-by: Sagar Darji <darsaga@amazon.com>

# Conflicts:
#	sandbox/plugins/composite-engine/src/main/java/org/opensearch/composite/CompositeIndexingExecutionEngine.java

* Addressing comments

Signed-off-by: Sagar Darji <darsaga@amazon.com>

* Split the monolithic CompositeMergeHandler into classes with clear responsibilities:

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

* Fix up tests

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

* Addressing commits

Signed-off-by: Sagar Darji <darsaga@amazon.com>

* Integrating the merge flow with the DataFormatAwareEngine

Signed-off-by: Sagar Darji <darsaga@amazon.com>

# Conflicts:
#	server/src/main/java/org/opensearch/index/engine/DataFormatAwareEngine.java
#	server/src/main/java/org/opensearch/index/engine/exec/coord/CatalogSnapshotManager.java
#	server/src/test/java/org/opensearch/index/engine/exec/coord/CatalogSnapshotManagerTests.java

# Conflicts:
#	server/src/main/java/org/opensearch/index/engine/DataFormatAwareEngine.java

* Addressed the comments

Signed-off-by: Sagar Darji <darsaga@amazon.com>

---------

Signed-off-by: Sagar Darji <darsaga@amazon.com>
Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>
Co-authored-by: Sagar Darji <darsaga@amazon.com>
Co-authored-by: Bukhtawar Khan <bukhtawa@amazon.com>
…1408)

* Fix incorrect defaults in FieldStorageResolver.

Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

* test fixes

Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

---------

Signed-off-by: Marc Handalian <marc.handalian@gmail.com>
@krishna-ggk
krishna-ggk force-pushed the pluggable-directory-store-validation branch from 45e8321 to e517a29 Compare April 28, 2026 07:58
Gopala Krishna A added 3 commits April 28, 2026 01:14
Store.readMetadataSnapshot() and Store.tryOpenIndex() hardcode
NIOFSDirectory when reading segment metadata during shard allocation.
This prevents IndexStorePlugin implementations (e.g. encryption plugins)
from reading their custom-formatted segment files during the gateway
allocation path after node restart, causing shards to be reported as
having no valid copy.

Add overloaded versions of readMetadataSnapshot() and tryOpenIndex()
that accept a DirectoryFactory and IndexSettings. When provided, the
factory is used to create the directory; otherwise falls back to
NIOFSDirectory for backward compatibility.

Update TransportNodesListShardStoreMetadataHelper and
TransportNodesGatewayStartedShardHelper to resolve the DirectoryFactory
from the index store type setting and pass it through.

Expose directoryFactories via IndicesService.getDirectoryFactories().

Add unit tests verifying custom factory invocation, null-factory
fallback, and FilterDirectory-wrapping factory behavior.

Signed-off-by: Gopala Krishna A <krishna.ggk@gmail.com>
Signed-off-by: Gopala Krishna A <krishna.ggk@gmail.com>
Signed-off-by: Gopala Krishna A <krishna.ggk@gmail.com>
@krishna-ggk
krishna-ggk marked this pull request as ready for review April 28, 2026 13:41
@github-actions

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 85f08bf

@github-actions

Copy link
Copy Markdown
Contributor

❌ Gradle check result for 85f08bf: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.