Default max concurrent search req. to the avg shards size per index

We moved to 1 shard by default which caused some issues in how many
concurrent shard requests we allow by default. For instance searching
a 5 shard index on a single node will now be executed serially per shard
while we want these cases to have a good concurrency out of the box. This
change moves to defaults based on the avg. number of shards per index in
the current search request to provide a good out of the box concurrency.

Relates to elastic#30783
Closes elastic#30994

Author: s1monw

server/src/main/java/org/elasticsearch/action/search/TransportSearchAction.java

@@ -345,8 +345,13 @@ private void executeSearch(SearchTask task, SearchTimeProvider timeProvider, Sea
              * We try to set a default of max concurrent shard requests based on the node count but upper-bound it by 256 by default to keep
              * it sane. A single search request that fans out to lots of shards should not hit a cluster too hard while 256 is already a
              * lot.
+             * Yet we are still trying to guarantee some concurrency ie. if you have an index size of N shards and you are searching a
+             * single index on a single node we still search it concurrently.
              */
-            searchRequest.setMaxConcurrentShardRequests(Math.min(256, nodeCount));
+            final int numShards = shardIterators.size();
+            final int numIndices = remoteClusterIndices.size() + concreteIndices.length;
+            assert numShards >= numIndices : "found schroedingers index numShards: " + numShards + " vs. numIndices: " + numIndices;
+            searchRequest.setMaxConcurrentShardRequests(Math.min(256, nodeCount * (numShards / numIndices)));
         }
         boolean preFilterSearchShards = shouldPreFilterSearchShards(searchRequest, shardIterators);
         searchAsyncAction(task, searchRequest, shardIterators, timeProvider, connectionLookup, clusterState.version(),