@jed326 / @sohami - Do you guys have some thoughts here?

Thanks for the feature request @kkewwei!

Concurrent fetch is something I had briefly discussed with @sohami before, but it's not something we saw any use cases for (until now). When we were looking at concurrent search the performance on aggregations is what we were focusing on, and in a lot of cases users will do aggregations with either size=0 or just the default size so in those so concurrent fetch was not really needed or useful.

I'd be happy to review any designs or PRs for this!

@jed326 / @sohami - Do you guys have some thoughts here?

Thanks for the feature request @kkewwei!

Concurrent fetch is something I had briefly discussed with @sohami before, but it's not something we saw any use cases for (until now). When we were looking at concurrent search the performance on aggregations is what we were focusing on, and in a lot of cases users will do aggregations with either size=0 or just the default size so in those so concurrent fetch was not really needed or useful.

I'd be happy to review any designs or PRs for this!

@jed326, If you haven't started on it, can you assign it to me? I'm pleased to have a try. Of course, I will make a draft first

can you assign it to me?

@kkewwei done!

It seems that serial execution of each document's innerhit can speed up due to querycache, while concurrent executing will result in performance degradation. I'm still testing.

@jed326 @sohami, during I experimentation with concurrent execution in the innerhit phase, I discovered another logic that requires optimization.

It is known that TermQuery is never cached in QueryCache, but innerhit phase extensively utilizes TermQuery, resulting in inefficient(https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/search/UsageTrackingQueryCachingPolicy.java#L57). The following improvements can be made:

1. In findNestedObjectMapper, if context is stance of InnerHitSubContext, we are able to rapidly identify the ObjectMapper, thus reducing the number of TermQuery executions.
2. In getInternalNestedIdentity, childFilter, which is a simple TermQuery, will be cached by bitsetFilterCache, which
   can also help reduce redundant query executions.

The details is as follows, I will pull a pr about this.
kkewwei@9a47717de54

I have test several times, and restarted the node each time to eliminate cache interference. The detail tooks(ms) is as follows:
Baseline(main branch), avg: 836ms
841
853
828
774
811
914

Optimized:, avg: 551ms
569
506
666
522
526
518

Took descrease 34%

Thanks @kkewwei! Feel free to trigger some performance benchmarks on your PR as well, see: https://github.com/opensearch-project/OpenSearch/blob/main/PERFORMANCE_BENCHMARKS.md

In our product, offset in innerhit serves no practical purpose. When dealing with complex nested structures, computing it will cost much time, In the next step, I will test to gauge the potential time savings by omitting the offset computation. If the performance improves a lot, I will try to adding parameter to bypass computing offset