[BUG] NullPointerException when creating remote index from snapshot

[Kyle-Cooley]

Describe the bug

A NullPointerException can be thrown when calculating the total size of existing shards created from remote snapshots. This appears to happen while shards are being allocated.

Related component

Search:Searchable Snapshots

To Reproduce

Use the POST /_snapshot/<repository>/<snapshot>/_restore to restore indices from multiple snapshots using remote_snapshot as the storage_type in a tight loop.

Expected behavior

The indices are restored.

Additional Details

Plugins
Please list all plugins currently enabled.

• all defaults
• repository-s3
• prometheus-exporter

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: Amazon Linux container
• Version 3.2.0

Additional context
Add any other context about the problem here.

java.lang.NullPointerException: null
        at java.base/java.util.stream.ReferencePipeline$5$1.accept(ReferencePipeline.java:249) ~[?:?]
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:215) ~[?:?]
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1716) ~[?:?]
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570) ~[?:?]
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:560) ~[?:?]
        at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265) ~[?:?]
        at java.base/java.util.stream.LongPipeline.reduce(LongPipeline.java:502) ~[?:?]
        at java.base/java.util.stream.LongPipeline.sum(LongPipeline.java:460) ~[?:?]
        at org.opensearch.snapshots.RestoreService$1.validateSearchableSnapshotRestorable(RestoreService.java:906) ~[opensearch-3.2.0.jar:3.2.0]
        at org.opensearch.snapshots.RestoreService$1.execute(RestoreService.java:588) ~[opensearch-3.2.0.jar:3.2.0]
        at org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:67) ~[opensearch-3.2.0.jar:3.2.0]
        at org.opensearch.cluster.service.ClusterManagerService.executeTasks(ClusterManagerService.java:890) ~[opensearch-3.2.0.jar:3.2.0]
        at org.opensearch.cluster.service.ClusterManagerService.calculateTaskOutputs(ClusterManagerService.java:441) ~[opensearch-3.2.0.jar:3.2.0]
        at org.opensearch.cluster.service.ClusterManagerService.runTasks(ClusterManagerService.java:301) [opensearch-3.2.0.jar:3.2.0]
        at org.opensearch.cluster.service.ClusterManagerService$Batcher.run(ClusterManagerService.java:214) [opensearch-3.2.0.jar:3.2.0]
        at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:206) [opensearch-3.2.0.jar:3.2.0]
        at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:264) [opensearch-3.2.0.jar:3.2.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:916) [opensearch-3.2.0.jar:3.2.0]
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:299) [opensearch-3.2.0.jar:3.2.0]
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:262) [opensearch-3.2.0.jar:3.2.0]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:1447) [?:?]

[andrross]

Catch All Triage - 1 2

[thomas315]

We experience exactly the same NullPointerException but in a different scenario and with different impact.
Restoring Snapshots via the ISM action convert_index_to_remote with multiple retries. Therefore the indices are restored mostly in the second try already. BUT when the NullPointerException happened the restored index will not get the ISM policy attached.
If the exception is not thrown, everything works fine.

Opensearch 3.2.0
Having daily indices.
On some days it works completely fine.
On others it fails for all of them.
Indices are restored at different times during the day as well.

[lawofcycles]

I've successfully reproduced and investigated this issue. I'll open a PR to fix it soon.

Root Cause

NPE occurs in RestoreService.java:914-915 when ClusterInfo.getShardSize() returns null for a shard that exists in the routing table but lacks size information in the ClusterInfo cache.

long totalRestoredRemoteIndexesSize = shardsIterator.getShardRoutings()
    .stream()
    .map(clusterInfo::getShardSize)  // Returns null
    .mapToLong(Long::longValue)      // NPE here
    .sum();

ClusterInfo.getShardSize() returns null when:

• Shard was recently allocated (after last ClusterInfo update)
• ClusterInfo update failed or timed out
• Shard is in INITIALIZING state without size data

Reproduction Steps

Environment: OpenSearch 3.2.0 (Docker)

1. Configure cluster to delay ClusterInfo updates to ensure newly created shards won't have size information.

PUT _cluster/settings
{
"transient": {
"cluster.info.update.interval": "60m"
}
}

2. Create snapshot repository
   PUT _snapshot/my_repository
   {
   "type": "fs",
   "settings": {
   "location": "/mnt/snapshots/my_repository"
   }
   }

3. Create test indices
   PUT test-index-1
   {"settings": {"number_of_shards": 2, "number_of_replicas": 0}}
   ... (repeat for test-index-2 through test-index-10)

4. Create snapshot and delete original indices:
   PUT _snapshot/my_repository/snapshot_all?wait_for_completion=true
   {
   "indices": "test-index-*"
   }

DELETE test-index-*

5. Restart cluster to clear ClusterInfo cache to ensure ClusterInfo is completely empty.

6. Execute multiple remote snapshot restores simultaneously:
   POST _snapshot/my_repository/snapshot_all/restore
   {
   "indices": "test-index-1",
   "storage_type": "remote_snapshot",
   "rename_pattern": "(.+)",
   "rename_replacement": "remote1$1"
   }

7. Immediately execute multiple similar requests
   With these steps, getShardSize() returns null and triggers NPE.
   First restore creates remote snapshot shards then it was added to routing table. Second restore triggers calculation for total size of existing remote shards but ClusterInfo has no size data.