Skip to content

Conversation

@andrevandeven
Copy link
Contributor

Description

There are 3 types of fetch phases that are important to account for in profiling:

  • Standard fetch phase: This is the fundamental second stage of every standard search query. It's a broad operation that retrieves the full source document (_source) for the top-ranking documents identified in the initial query phase.
  • Inner Hits Fetch: This is a specialized type of fetch used to pinpoint exactly why a document with nested or parent-child data matched your query. It retrieves the specific nested objects that caused the match, not just the top-level document.
  • Top Hits Aggregation (Fetch Top Hits): This is not a phase of the search itself, but rather a powerful aggregation. It allows you to fetch the top-matching documents per group or per bucket after a primary aggregation has run.
    In the first stage of implementation, the fetch phase profile only supports profiling the standard fetch phase.

This change allows inner hits fetch and top hits aggregation fetch to be profiled and included as separate fetch phases alongside the standard fetch phase in the profile. If multiple separate inner hits searches are run or top hits aggregations are run, they will appear as separate breakdowns. Each individual fetch phase within a search or aggregation will be consolidated under the breakdown of its respective operation rather than creating a new breakdown for each individual fetch phase that is run. The logic for this is that top hits aggregation often runs fetch phase many times, depending on the number of documents aggregated, and lack of consolidation would lead to a verbose and crowded search profile.

Related Issues

Resolves #18862

#18864

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions github-actions bot added enhancement Enhancement or improvement to existing feature or request Search:Query Insights labels Aug 5, 2025
@andrevandeven andrevandeven changed the title Inner hits top hits fetch Expand fetch phase profiling to support inner hits and top hits aggregation phases Aug 5, 2025
andrevandeven and others added 2 commits August 5, 2025 16:33
Signed-off-by: Andre van de Ven <[email protected]>
@github-actions
Copy link
Contributor

github-actions bot commented Aug 6, 2025

❌ Gradle check result for d760f9f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

andrevandeven and others added 2 commits August 6, 2025 10:32
Signed-off-by: Andre van de Ven <[email protected]>
@github-actions
Copy link
Contributor

github-actions bot commented Aug 6, 2025

❌ Gradle check result for d14fe2d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Andre van de Ven <[email protected]>
@github-actions
Copy link
Contributor

github-actions bot commented Aug 6, 2025

❌ Gradle check result for 96bc00f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Andre van de Ven <[email protected]>
@github-actions
Copy link
Contributor

github-actions bot commented Aug 6, 2025

❌ Gradle check result for 46b9140: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Andre van de Ven <[email protected]>
@github-actions
Copy link
Contributor

github-actions bot commented Aug 7, 2025

❌ Gradle check result for 5db9865: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Andre van de Ven <[email protected]>
@github-actions
Copy link
Contributor

github-actions bot commented Aug 7, 2025

❌ Gradle check result for 7b76295: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Andre van de Ven added 3 commits August 7, 2025 11:03
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
@github-actions
Copy link
Contributor

github-actions bot commented Aug 7, 2025

❌ Gradle check result for c8e2ae1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

…ation due to concurrency

Signed-off-by: Andre van de Ven <[email protected]>
@github-actions
Copy link
Contributor

github-actions bot commented Aug 7, 2025

❌ Gradle check result for 7267d03: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@andrevandeven andrevandeven force-pushed the inner-hits-top-hits-fetch branch from 00c407f to 9738ccb Compare August 11, 2025 21:09
@github-actions
Copy link
Contributor

✅ Gradle check result for 9738ccb: SUCCESS

@andrevandeven
Copy link
Contributor Author

andrevandeven commented Aug 11, 2025

@msfroh I updated the rest tests to skip versions before 3.3, so should be good to go now! EDIT: Gradle check needs to run again because I fixed changelog merge issues. Will ping again when it finishes running!

@github-actions
Copy link
Contributor

✅ Gradle check result for 2ed7df4: SUCCESS

@andrevandeven
Copy link
Contributor Author

@msfroh good to go now!

@msfroh msfroh merged commit 13e2f2d into opensearch-project:main Aug 12, 2025
31 checks passed
RajatGupta02 pushed a commit to RajatGupta02/OpenSearch that referenced this pull request Aug 18, 2025
…gation phases (opensearch-project#18936)

---------

Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>
RajatGupta02 pushed a commit to RajatGupta02/OpenSearch that referenced this pull request Aug 18, 2025
…gation phases (opensearch-project#18936)

---------

Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>
RajatGupta02 pushed a commit to RajatGupta02/OpenSearch that referenced this pull request Aug 26, 2025
…gation phases (opensearch-project#18936)

---------

Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>
cwperks added a commit that referenced this pull request Aug 27, 2025
* Add overload for channelFactory

Signed-off-by: Rajat Gupta <[email protected]>

* Fix tests

Signed-off-by: Rajat Gupta <[email protected]>

* Add Changelog entry

Signed-off-by: Rajat Gupta <[email protected]>

* Fix conflicts

Signed-off-by: Rajat Gupta <[email protected]>

* When update operations fail during preparation (e.g., version conflicts), (#18917)

TransportShardBulkAction still triggers refresh even though no actual
  writes occurred. This fix checks if locationToSync is null (indicating
  no writes) and prevents refresh in such cases.

  Fixes #15261

Signed-off-by: Atri Sharma <[email protected]>

* Remove all entries from changelog to be released in 3.2 (#18989)

Signed-off-by: Andrew Ross <[email protected]>

* Add temporal routing processors for time-based document routing (#18966)

Implements TemporalRoutingProcessor for ingest pipelines and
TemporalRoutingSearchProcessor for search pipelines based on RFC #18920.

 Features:
 - Route documents to shards based on timestamp fields
 - Support hour, day, week, and month granularities
 - Optional hash bucketing for better distribution
 - Automatic search routing to relevant time ranges
 - ISO week format support

The processors enable efficient time-based data organization for
log and metrics workloads by co-locating documents from the same
time period on the same shards.

---------

Signed-off-by: Atri Sharma <[email protected]>

* Add CompletionStage variants to methods in the Client Interface and default to ActionListener impl (#18998)

* Add CompletableFuture variables to methods in the Client Interface and default to ActionListener impl

Signed-off-by: Craig Perkins <[email protected]>

* Add to CHANGELOG

Signed-off-by: Craig Perkins <[email protected]>

* Fix typo in CHANGELOG

Signed-off-by: Craig Perkins <[email protected]>

* Switch to CompletionStage

Signed-off-by: Craig Perkins <[email protected]>

* Update CHANGELOG entry

Signed-off-by: Craig Perkins <[email protected]>

---------

Signed-off-by: Craig Perkins <[email protected]>

* Expand fetch phase profiling to support inner hits and top hits aggregation phases (#18936)


---------

Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>

* IllegalArgumentException when scroll ID has a node no longer part of the Cluster (#19031)



---------

Signed-off-by: Anurag Rai <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>

* Add Changelog entry

Signed-off-by: Rajat Gupta <[email protected]>

* Add secondary constructor

Signed-off-by: Rajat Gupta <[email protected]>

* Modify changelog

Signed-off-by: Rajat Gupta <[email protected]>

* Update changelog

Signed-off-by: Rajat Gupta <[email protected]>

* Add another constructor to fix breaking change check

Signed-off-by: Rajat Gupta <[email protected]>

---------

Signed-off-by: Rajat Gupta <[email protected]>
Signed-off-by: Atri Sharma <[email protected]>
Signed-off-by: Andrew Ross <[email protected]>
Signed-off-by: Craig Perkins <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>
Co-authored-by: Rajat Gupta <[email protected]>
Co-authored-by: Atri Sharma <[email protected]>
Co-authored-by: Andrew Ross <[email protected]>
Co-authored-by: Craig Perkins <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>
Co-authored-by: Anurag Rai <[email protected]>
atris pushed a commit to atris/OpenSearch that referenced this pull request Aug 28, 2025
…gation phases (opensearch-project#18936)

---------

Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>
atris added a commit to atris/OpenSearch that referenced this pull request Aug 28, 2025
* Add overload for channelFactory

Signed-off-by: Rajat Gupta <[email protected]>

* Fix tests

Signed-off-by: Rajat Gupta <[email protected]>

* Add Changelog entry

Signed-off-by: Rajat Gupta <[email protected]>

* Fix conflicts

Signed-off-by: Rajat Gupta <[email protected]>

* When update operations fail during preparation (e.g., version conflicts), (opensearch-project#18917)

TransportShardBulkAction still triggers refresh even though no actual
  writes occurred. This fix checks if locationToSync is null (indicating
  no writes) and prevents refresh in such cases.

  Fixes opensearch-project#15261

Signed-off-by: Atri Sharma <[email protected]>

* Remove all entries from changelog to be released in 3.2 (opensearch-project#18989)

Signed-off-by: Andrew Ross <[email protected]>

* Add temporal routing processors for time-based document routing (opensearch-project#18966)

Implements TemporalRoutingProcessor for ingest pipelines and
TemporalRoutingSearchProcessor for search pipelines based on RFC opensearch-project#18920.

 Features:
 - Route documents to shards based on timestamp fields
 - Support hour, day, week, and month granularities
 - Optional hash bucketing for better distribution
 - Automatic search routing to relevant time ranges
 - ISO week format support

The processors enable efficient time-based data organization for
log and metrics workloads by co-locating documents from the same
time period on the same shards.

---------

Signed-off-by: Atri Sharma <[email protected]>

* Add CompletionStage variants to methods in the Client Interface and default to ActionListener impl (opensearch-project#18998)

* Add CompletableFuture variables to methods in the Client Interface and default to ActionListener impl

Signed-off-by: Craig Perkins <[email protected]>

* Add to CHANGELOG

Signed-off-by: Craig Perkins <[email protected]>

* Fix typo in CHANGELOG

Signed-off-by: Craig Perkins <[email protected]>

* Switch to CompletionStage

Signed-off-by: Craig Perkins <[email protected]>

* Update CHANGELOG entry

Signed-off-by: Craig Perkins <[email protected]>

---------

Signed-off-by: Craig Perkins <[email protected]>

* Expand fetch phase profiling to support inner hits and top hits aggregation phases (opensearch-project#18936)


---------

Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>

* IllegalArgumentException when scroll ID has a node no longer part of the Cluster (opensearch-project#19031)



---------

Signed-off-by: Anurag Rai <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>

* Add Changelog entry

Signed-off-by: Rajat Gupta <[email protected]>

* Add secondary constructor

Signed-off-by: Rajat Gupta <[email protected]>

* Modify changelog

Signed-off-by: Rajat Gupta <[email protected]>

* Update changelog

Signed-off-by: Rajat Gupta <[email protected]>

* Add another constructor to fix breaking change check

Signed-off-by: Rajat Gupta <[email protected]>

---------

Signed-off-by: Rajat Gupta <[email protected]>
Signed-off-by: Atri Sharma <[email protected]>
Signed-off-by: Andrew Ross <[email protected]>
Signed-off-by: Craig Perkins <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>
Co-authored-by: Rajat Gupta <[email protected]>
Co-authored-by: Atri Sharma <[email protected]>
Co-authored-by: Andrew Ross <[email protected]>
Co-authored-by: Craig Perkins <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>
Co-authored-by: Anurag Rai <[email protected]>
pranikum pushed a commit to pranikum/OpenSearch that referenced this pull request Sep 4, 2025
* Add overload for channelFactory

Signed-off-by: Rajat Gupta <[email protected]>

* Fix tests

Signed-off-by: Rajat Gupta <[email protected]>

* Add Changelog entry

Signed-off-by: Rajat Gupta <[email protected]>

* Fix conflicts

Signed-off-by: Rajat Gupta <[email protected]>

* When update operations fail during preparation (e.g., version conflicts), (opensearch-project#18917)

TransportShardBulkAction still triggers refresh even though no actual
  writes occurred. This fix checks if locationToSync is null (indicating
  no writes) and prevents refresh in such cases.

  Fixes opensearch-project#15261

Signed-off-by: Atri Sharma <[email protected]>

* Remove all entries from changelog to be released in 3.2 (opensearch-project#18989)

Signed-off-by: Andrew Ross <[email protected]>

* Add temporal routing processors for time-based document routing (opensearch-project#18966)

Implements TemporalRoutingProcessor for ingest pipelines and
TemporalRoutingSearchProcessor for search pipelines based on RFC opensearch-project#18920.

 Features:
 - Route documents to shards based on timestamp fields
 - Support hour, day, week, and month granularities
 - Optional hash bucketing for better distribution
 - Automatic search routing to relevant time ranges
 - ISO week format support

The processors enable efficient time-based data organization for
log and metrics workloads by co-locating documents from the same
time period on the same shards.

---------

Signed-off-by: Atri Sharma <[email protected]>

* Add CompletionStage variants to methods in the Client Interface and default to ActionListener impl (opensearch-project#18998)

* Add CompletableFuture variables to methods in the Client Interface and default to ActionListener impl

Signed-off-by: Craig Perkins <[email protected]>

* Add to CHANGELOG

Signed-off-by: Craig Perkins <[email protected]>

* Fix typo in CHANGELOG

Signed-off-by: Craig Perkins <[email protected]>

* Switch to CompletionStage

Signed-off-by: Craig Perkins <[email protected]>

* Update CHANGELOG entry

Signed-off-by: Craig Perkins <[email protected]>

---------

Signed-off-by: Craig Perkins <[email protected]>

* Expand fetch phase profiling to support inner hits and top hits aggregation phases (opensearch-project#18936)


---------

Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>

* IllegalArgumentException when scroll ID has a node no longer part of the Cluster (opensearch-project#19031)



---------

Signed-off-by: Anurag Rai <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>

* Add Changelog entry

Signed-off-by: Rajat Gupta <[email protected]>

* Add secondary constructor

Signed-off-by: Rajat Gupta <[email protected]>

* Modify changelog

Signed-off-by: Rajat Gupta <[email protected]>

* Update changelog

Signed-off-by: Rajat Gupta <[email protected]>

* Add another constructor to fix breaking change check

Signed-off-by: Rajat Gupta <[email protected]>

---------

Signed-off-by: Rajat Gupta <[email protected]>
Signed-off-by: Atri Sharma <[email protected]>
Signed-off-by: Andrew Ross <[email protected]>
Signed-off-by: Craig Perkins <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>
Co-authored-by: Rajat Gupta <[email protected]>
Co-authored-by: Atri Sharma <[email protected]>
Co-authored-by: Andrew Ross <[email protected]>
Co-authored-by: Craig Perkins <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>
Co-authored-by: Anurag Rai <[email protected]>
kh3ra pushed a commit to kh3ra/OpenSearch that referenced this pull request Sep 5, 2025
…gation phases (opensearch-project#18936)

---------

Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>
kh3ra pushed a commit to kh3ra/OpenSearch that referenced this pull request Sep 5, 2025
* Add overload for channelFactory

Signed-off-by: Rajat Gupta <[email protected]>

* Fix tests

Signed-off-by: Rajat Gupta <[email protected]>

* Add Changelog entry

Signed-off-by: Rajat Gupta <[email protected]>

* Fix conflicts

Signed-off-by: Rajat Gupta <[email protected]>

* When update operations fail during preparation (e.g., version conflicts), (opensearch-project#18917)

TransportShardBulkAction still triggers refresh even though no actual
  writes occurred. This fix checks if locationToSync is null (indicating
  no writes) and prevents refresh in such cases.

  Fixes opensearch-project#15261

Signed-off-by: Atri Sharma <[email protected]>

* Remove all entries from changelog to be released in 3.2 (opensearch-project#18989)

Signed-off-by: Andrew Ross <[email protected]>

* Add temporal routing processors for time-based document routing (opensearch-project#18966)

Implements TemporalRoutingProcessor for ingest pipelines and
TemporalRoutingSearchProcessor for search pipelines based on RFC opensearch-project#18920.

 Features:
 - Route documents to shards based on timestamp fields
 - Support hour, day, week, and month granularities
 - Optional hash bucketing for better distribution
 - Automatic search routing to relevant time ranges
 - ISO week format support

The processors enable efficient time-based data organization for
log and metrics workloads by co-locating documents from the same
time period on the same shards.

---------

Signed-off-by: Atri Sharma <[email protected]>

* Add CompletionStage variants to methods in the Client Interface and default to ActionListener impl (opensearch-project#18998)

* Add CompletableFuture variables to methods in the Client Interface and default to ActionListener impl

Signed-off-by: Craig Perkins <[email protected]>

* Add to CHANGELOG

Signed-off-by: Craig Perkins <[email protected]>

* Fix typo in CHANGELOG

Signed-off-by: Craig Perkins <[email protected]>

* Switch to CompletionStage

Signed-off-by: Craig Perkins <[email protected]>

* Update CHANGELOG entry

Signed-off-by: Craig Perkins <[email protected]>

---------

Signed-off-by: Craig Perkins <[email protected]>

* Expand fetch phase profiling to support inner hits and top hits aggregation phases (opensearch-project#18936)


---------

Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>

* IllegalArgumentException when scroll ID has a node no longer part of the Cluster (opensearch-project#19031)



---------

Signed-off-by: Anurag Rai <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>

* Add Changelog entry

Signed-off-by: Rajat Gupta <[email protected]>

* Add secondary constructor

Signed-off-by: Rajat Gupta <[email protected]>

* Modify changelog

Signed-off-by: Rajat Gupta <[email protected]>

* Update changelog

Signed-off-by: Rajat Gupta <[email protected]>

* Add another constructor to fix breaking change check

Signed-off-by: Rajat Gupta <[email protected]>

---------

Signed-off-by: Rajat Gupta <[email protected]>
Signed-off-by: Atri Sharma <[email protected]>
Signed-off-by: Andrew Ross <[email protected]>
Signed-off-by: Craig Perkins <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>
Co-authored-by: Rajat Gupta <[email protected]>
Co-authored-by: Atri Sharma <[email protected]>
Co-authored-by: Andrew Ross <[email protected]>
Co-authored-by: Craig Perkins <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>
Co-authored-by: Anurag Rai <[email protected]>
jainankitk pushed a commit to jainankitk/OpenSearch that referenced this pull request Sep 22, 2025
* Add overload for channelFactory

Signed-off-by: Rajat Gupta <[email protected]>

* Fix tests

Signed-off-by: Rajat Gupta <[email protected]>

* Add Changelog entry

Signed-off-by: Rajat Gupta <[email protected]>

* Fix conflicts

Signed-off-by: Rajat Gupta <[email protected]>

* When update operations fail during preparation (e.g., version conflicts), (opensearch-project#18917)

TransportShardBulkAction still triggers refresh even though no actual
  writes occurred. This fix checks if locationToSync is null (indicating
  no writes) and prevents refresh in such cases.

  Fixes opensearch-project#15261

Signed-off-by: Atri Sharma <[email protected]>

* Remove all entries from changelog to be released in 3.2 (opensearch-project#18989)

Signed-off-by: Andrew Ross <[email protected]>

* Add temporal routing processors for time-based document routing (opensearch-project#18966)

Implements TemporalRoutingProcessor for ingest pipelines and
TemporalRoutingSearchProcessor for search pipelines based on RFC opensearch-project#18920.

 Features:
 - Route documents to shards based on timestamp fields
 - Support hour, day, week, and month granularities
 - Optional hash bucketing for better distribution
 - Automatic search routing to relevant time ranges
 - ISO week format support

The processors enable efficient time-based data organization for
log and metrics workloads by co-locating documents from the same
time period on the same shards.

---------

Signed-off-by: Atri Sharma <[email protected]>

* Add CompletionStage variants to methods in the Client Interface and default to ActionListener impl (opensearch-project#18998)

* Add CompletableFuture variables to methods in the Client Interface and default to ActionListener impl

Signed-off-by: Craig Perkins <[email protected]>

* Add to CHANGELOG

Signed-off-by: Craig Perkins <[email protected]>

* Fix typo in CHANGELOG

Signed-off-by: Craig Perkins <[email protected]>

* Switch to CompletionStage

Signed-off-by: Craig Perkins <[email protected]>

* Update CHANGELOG entry

Signed-off-by: Craig Perkins <[email protected]>

---------

Signed-off-by: Craig Perkins <[email protected]>

* Expand fetch phase profiling to support inner hits and top hits aggregation phases (opensearch-project#18936)


---------

Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>

* IllegalArgumentException when scroll ID has a node no longer part of the Cluster (opensearch-project#19031)



---------

Signed-off-by: Anurag Rai <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>

* Add Changelog entry

Signed-off-by: Rajat Gupta <[email protected]>

* Add secondary constructor

Signed-off-by: Rajat Gupta <[email protected]>

* Modify changelog

Signed-off-by: Rajat Gupta <[email protected]>

* Update changelog

Signed-off-by: Rajat Gupta <[email protected]>

* Add another constructor to fix breaking change check

Signed-off-by: Rajat Gupta <[email protected]>

---------

Signed-off-by: Rajat Gupta <[email protected]>
Signed-off-by: Atri Sharma <[email protected]>
Signed-off-by: Andrew Ross <[email protected]>
Signed-off-by: Craig Perkins <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>
Co-authored-by: Rajat Gupta <[email protected]>
Co-authored-by: Atri Sharma <[email protected]>
Co-authored-by: Andrew Ross <[email protected]>
Co-authored-by: Craig Perkins <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>
Co-authored-by: Anurag Rai <[email protected]>
jainankitk pushed a commit to jainankitk/OpenSearch that referenced this pull request Sep 22, 2025
* Add overload for channelFactory

Signed-off-by: Rajat Gupta <[email protected]>

* Fix tests

Signed-off-by: Rajat Gupta <[email protected]>

* Add Changelog entry

Signed-off-by: Rajat Gupta <[email protected]>

* Fix conflicts

Signed-off-by: Rajat Gupta <[email protected]>

* When update operations fail during preparation (e.g., version conflicts), (opensearch-project#18917)

TransportShardBulkAction still triggers refresh even though no actual
  writes occurred. This fix checks if locationToSync is null (indicating
  no writes) and prevents refresh in such cases.

  Fixes opensearch-project#15261

Signed-off-by: Atri Sharma <[email protected]>

* Remove all entries from changelog to be released in 3.2 (opensearch-project#18989)

Signed-off-by: Andrew Ross <[email protected]>

* Add temporal routing processors for time-based document routing (opensearch-project#18966)

Implements TemporalRoutingProcessor for ingest pipelines and
TemporalRoutingSearchProcessor for search pipelines based on RFC opensearch-project#18920.

 Features:
 - Route documents to shards based on timestamp fields
 - Support hour, day, week, and month granularities
 - Optional hash bucketing for better distribution
 - Automatic search routing to relevant time ranges
 - ISO week format support

The processors enable efficient time-based data organization for
log and metrics workloads by co-locating documents from the same
time period on the same shards.

---------

Signed-off-by: Atri Sharma <[email protected]>

* Add CompletionStage variants to methods in the Client Interface and default to ActionListener impl (opensearch-project#18998)

* Add CompletableFuture variables to methods in the Client Interface and default to ActionListener impl

Signed-off-by: Craig Perkins <[email protected]>

* Add to CHANGELOG

Signed-off-by: Craig Perkins <[email protected]>

* Fix typo in CHANGELOG

Signed-off-by: Craig Perkins <[email protected]>

* Switch to CompletionStage

Signed-off-by: Craig Perkins <[email protected]>

* Update CHANGELOG entry

Signed-off-by: Craig Perkins <[email protected]>

---------

Signed-off-by: Craig Perkins <[email protected]>

* Expand fetch phase profiling to support inner hits and top hits aggregation phases (opensearch-project#18936)

---------

Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>

* IllegalArgumentException when scroll ID has a node no longer part of the Cluster (opensearch-project#19031)

---------

Signed-off-by: Anurag Rai <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>

* Add Changelog entry

Signed-off-by: Rajat Gupta <[email protected]>

* Add secondary constructor

Signed-off-by: Rajat Gupta <[email protected]>

* Modify changelog

Signed-off-by: Rajat Gupta <[email protected]>

* Update changelog

Signed-off-by: Rajat Gupta <[email protected]>

* Add another constructor to fix breaking change check

Signed-off-by: Rajat Gupta <[email protected]>

---------

Signed-off-by: Rajat Gupta <[email protected]>
Signed-off-by: Atri Sharma <[email protected]>
Signed-off-by: Andrew Ross <[email protected]>
Signed-off-by: Craig Perkins <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>
Co-authored-by: Rajat Gupta <[email protected]>
Co-authored-by: Atri Sharma <[email protected]>
Co-authored-by: Andrew Ross <[email protected]>
Co-authored-by: Craig Perkins <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>
Co-authored-by: Anurag Rai <[email protected]>
Signed-off-by: Ankit Jain <[email protected]>
jainankitk pushed a commit to jainankitk/OpenSearch that referenced this pull request Sep 22, 2025
* Add overload for channelFactory

Signed-off-by: Rajat Gupta <[email protected]>

* Fix tests

Signed-off-by: Rajat Gupta <[email protected]>

* Add Changelog entry

Signed-off-by: Rajat Gupta <[email protected]>

* Fix conflicts

Signed-off-by: Rajat Gupta <[email protected]>

* When update operations fail during preparation (e.g., version conflicts), (opensearch-project#18917)

TransportShardBulkAction still triggers refresh even though no actual
  writes occurred. This fix checks if locationToSync is null (indicating
  no writes) and prevents refresh in such cases.

  Fixes opensearch-project#15261

Signed-off-by: Atri Sharma <[email protected]>

* Remove all entries from changelog to be released in 3.2 (opensearch-project#18989)

Signed-off-by: Andrew Ross <[email protected]>

* Add temporal routing processors for time-based document routing (opensearch-project#18966)

Implements TemporalRoutingProcessor for ingest pipelines and
TemporalRoutingSearchProcessor for search pipelines based on RFC opensearch-project#18920.

 Features:
 - Route documents to shards based on timestamp fields
 - Support hour, day, week, and month granularities
 - Optional hash bucketing for better distribution
 - Automatic search routing to relevant time ranges
 - ISO week format support

The processors enable efficient time-based data organization for
log and metrics workloads by co-locating documents from the same
time period on the same shards.

---------

Signed-off-by: Atri Sharma <[email protected]>

* Add CompletionStage variants to methods in the Client Interface and default to ActionListener impl (opensearch-project#18998)

* Add CompletableFuture variables to methods in the Client Interface and default to ActionListener impl

Signed-off-by: Craig Perkins <[email protected]>

* Add to CHANGELOG

Signed-off-by: Craig Perkins <[email protected]>

* Fix typo in CHANGELOG

Signed-off-by: Craig Perkins <[email protected]>

* Switch to CompletionStage

Signed-off-by: Craig Perkins <[email protected]>

* Update CHANGELOG entry

Signed-off-by: Craig Perkins <[email protected]>

---------

Signed-off-by: Craig Perkins <[email protected]>

* Expand fetch phase profiling to support inner hits and top hits aggregation phases (opensearch-project#18936)

---------

Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>

* IllegalArgumentException when scroll ID has a node no longer part of the Cluster (opensearch-project#19031)

---------

Signed-off-by: Anurag Rai <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>

* Add Changelog entry

Signed-off-by: Rajat Gupta <[email protected]>

* Add secondary constructor

Signed-off-by: Rajat Gupta <[email protected]>

* Modify changelog

Signed-off-by: Rajat Gupta <[email protected]>

* Update changelog

Signed-off-by: Rajat Gupta <[email protected]>

* Add another constructor to fix breaking change check

Signed-off-by: Rajat Gupta <[email protected]>

---------

Signed-off-by: Rajat Gupta <[email protected]>
Signed-off-by: Atri Sharma <[email protected]>
Signed-off-by: Andrew Ross <[email protected]>
Signed-off-by: Craig Perkins <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>
Co-authored-by: Rajat Gupta <[email protected]>
Co-authored-by: Atri Sharma <[email protected]>
Co-authored-by: Andrew Ross <[email protected]>
Co-authored-by: Craig Perkins <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>
Co-authored-by: Anurag Rai <[email protected]>
Signed-off-by: Ankit Jain <[email protected]>
asimmahmood1 pushed a commit to jainankitk/OpenSearch that referenced this pull request Sep 23, 2025
* Add overload for channelFactory

Signed-off-by: Rajat Gupta <[email protected]>

* Fix tests

Signed-off-by: Rajat Gupta <[email protected]>

* Add Changelog entry

Signed-off-by: Rajat Gupta <[email protected]>

* Fix conflicts

Signed-off-by: Rajat Gupta <[email protected]>

* When update operations fail during preparation (e.g., version conflicts), (opensearch-project#18917)

TransportShardBulkAction still triggers refresh even though no actual
  writes occurred. This fix checks if locationToSync is null (indicating
  no writes) and prevents refresh in such cases.

  Fixes opensearch-project#15261

Signed-off-by: Atri Sharma <[email protected]>

* Remove all entries from changelog to be released in 3.2 (opensearch-project#18989)

Signed-off-by: Andrew Ross <[email protected]>

* Add temporal routing processors for time-based document routing (opensearch-project#18966)

Implements TemporalRoutingProcessor for ingest pipelines and
TemporalRoutingSearchProcessor for search pipelines based on RFC opensearch-project#18920.

 Features:
 - Route documents to shards based on timestamp fields
 - Support hour, day, week, and month granularities
 - Optional hash bucketing for better distribution
 - Automatic search routing to relevant time ranges
 - ISO week format support

The processors enable efficient time-based data organization for
log and metrics workloads by co-locating documents from the same
time period on the same shards.

---------

Signed-off-by: Atri Sharma <[email protected]>

* Add CompletionStage variants to methods in the Client Interface and default to ActionListener impl (opensearch-project#18998)

* Add CompletableFuture variables to methods in the Client Interface and default to ActionListener impl

Signed-off-by: Craig Perkins <[email protected]>

* Add to CHANGELOG

Signed-off-by: Craig Perkins <[email protected]>

* Fix typo in CHANGELOG

Signed-off-by: Craig Perkins <[email protected]>

* Switch to CompletionStage

Signed-off-by: Craig Perkins <[email protected]>

* Update CHANGELOG entry

Signed-off-by: Craig Perkins <[email protected]>

---------

Signed-off-by: Craig Perkins <[email protected]>

* Expand fetch phase profiling to support inner hits and top hits aggregation phases (opensearch-project#18936)


---------

Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>

* IllegalArgumentException when scroll ID has a node no longer part of the Cluster (opensearch-project#19031)



---------

Signed-off-by: Anurag Rai <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>

* Add Changelog entry

Signed-off-by: Rajat Gupta <[email protected]>

* Add secondary constructor

Signed-off-by: Rajat Gupta <[email protected]>

* Modify changelog

Signed-off-by: Rajat Gupta <[email protected]>

* Update changelog

Signed-off-by: Rajat Gupta <[email protected]>

* Add another constructor to fix breaking change check

Signed-off-by: Rajat Gupta <[email protected]>

---------

Signed-off-by: Rajat Gupta <[email protected]>
Signed-off-by: Atri Sharma <[email protected]>
Signed-off-by: Andrew Ross <[email protected]>
Signed-off-by: Craig Perkins <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>
Co-authored-by: Rajat Gupta <[email protected]>
Co-authored-by: Atri Sharma <[email protected]>
Co-authored-by: Andrew Ross <[email protected]>
Co-authored-by: Craig Perkins <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>
Co-authored-by: Anurag Rai <[email protected]>
vinaykpud pushed a commit to vinaykpud/OpenSearch that referenced this pull request Sep 26, 2025
…gation phases (opensearch-project#18936)

---------

Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>
vinaykpud pushed a commit to vinaykpud/OpenSearch that referenced this pull request Sep 26, 2025
* Add overload for channelFactory

Signed-off-by: Rajat Gupta <[email protected]>

* Fix tests

Signed-off-by: Rajat Gupta <[email protected]>

* Add Changelog entry

Signed-off-by: Rajat Gupta <[email protected]>

* Fix conflicts

Signed-off-by: Rajat Gupta <[email protected]>

* When update operations fail during preparation (e.g., version conflicts), (opensearch-project#18917)

TransportShardBulkAction still triggers refresh even though no actual
  writes occurred. This fix checks if locationToSync is null (indicating
  no writes) and prevents refresh in such cases.

  Fixes opensearch-project#15261

Signed-off-by: Atri Sharma <[email protected]>

* Remove all entries from changelog to be released in 3.2 (opensearch-project#18989)

Signed-off-by: Andrew Ross <[email protected]>

* Add temporal routing processors for time-based document routing (opensearch-project#18966)

Implements TemporalRoutingProcessor for ingest pipelines and
TemporalRoutingSearchProcessor for search pipelines based on RFC opensearch-project#18920.

 Features:
 - Route documents to shards based on timestamp fields
 - Support hour, day, week, and month granularities
 - Optional hash bucketing for better distribution
 - Automatic search routing to relevant time ranges
 - ISO week format support

The processors enable efficient time-based data organization for
log and metrics workloads by co-locating documents from the same
time period on the same shards.

---------

Signed-off-by: Atri Sharma <[email protected]>

* Add CompletionStage variants to methods in the Client Interface and default to ActionListener impl (opensearch-project#18998)

* Add CompletableFuture variables to methods in the Client Interface and default to ActionListener impl

Signed-off-by: Craig Perkins <[email protected]>

* Add to CHANGELOG

Signed-off-by: Craig Perkins <[email protected]>

* Fix typo in CHANGELOG

Signed-off-by: Craig Perkins <[email protected]>

* Switch to CompletionStage

Signed-off-by: Craig Perkins <[email protected]>

* Update CHANGELOG entry

Signed-off-by: Craig Perkins <[email protected]>

---------

Signed-off-by: Craig Perkins <[email protected]>

* Expand fetch phase profiling to support inner hits and top hits aggregation phases (opensearch-project#18936)


---------

Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>

* IllegalArgumentException when scroll ID has a node no longer part of the Cluster (opensearch-project#19031)



---------

Signed-off-by: Anurag Rai <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>

* Add Changelog entry

Signed-off-by: Rajat Gupta <[email protected]>

* Add secondary constructor

Signed-off-by: Rajat Gupta <[email protected]>

* Modify changelog

Signed-off-by: Rajat Gupta <[email protected]>

* Update changelog

Signed-off-by: Rajat Gupta <[email protected]>

* Add another constructor to fix breaking change check

Signed-off-by: Rajat Gupta <[email protected]>

---------

Signed-off-by: Rajat Gupta <[email protected]>
Signed-off-by: Atri Sharma <[email protected]>
Signed-off-by: Andrew Ross <[email protected]>
Signed-off-by: Craig Perkins <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Andre van de Ven <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>
Signed-off-by: Anurag Rai <[email protected]>
Co-authored-by: Rajat Gupta <[email protected]>
Co-authored-by: Atri Sharma <[email protected]>
Co-authored-by: Andrew Ross <[email protected]>
Co-authored-by: Craig Perkins <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>
Co-authored-by: Andre van de Ven <[email protected]>
Co-authored-by: Anurag Rai <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Enhancement or improvement to existing feature or request Search:Query Insights

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expand fetch phase profiling to support inner hits and top hits aggregation phases

2 participants