Skip to content

Conversation

@prudhvigodithi
Copy link
Member

@prudhvigodithi prudhvigodithi commented Aug 2, 2025

This PR targets to support search_after queries from the solution described in this issue #18546.

  • Adjusts the lower and upper bounds based on the sort direction. Lower bound for ASC sort and upper bound for DESC sort. Example:
Data: 1, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100 
Range query: [10-100] with ASC sort → Results: 10, 20, 30, 40, 50, 60, 70, 80, 90, 100
search_after: [30] with ASC sort → New range: [31-100] - excludes 30
                                   Results: 40, 50, 60, 70, 80, 90, 100

Data: 1, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100
Range query: [0-90] with DESC sort → Results: 90, 80, 70, 60, 50, 40, 30, 20, 10, 1
search_after: [30] with DESC sort → New range: [0-29] excludes 30
                                    Results: 20, 10, 1
  • Adds new byte[] encodePoint(Object value, boolean roundUp); in NumericPointEncoder as today search_after value is returned as object and requires converting it into the appropriate field specific byte[].

    • This handles the rounding up and rounding down for field specific value as for search_after with ASC sort we want values greater than the search_after value and for DESC sort we want values less than the search_after.
  • Handles the search_after conversation for dates with date math rounding operations, here is the sample query. Note the existing parseToLong is for date math rounding operations and not for creating search_after bounds.

curl -X POST "localhost:9200/big5/_search" -H 'Content-Type: application/json' -d'
{
  "query": {
    "match_all": {}
  },
  "sort": [
    { "@timestamp": "asc" }
  ],
  "search_after": ["2023-01-01T10:30:00.000Z||/h"]  
}'
  • Coming from this PR https://github.com/opensearch-project/OpenSearch/pull/18763/files where approximation code path is disabled for multi sort, hence search_after should work as expected when dealing with multiple tie breaker search_after values

  • Regarding benchmarks following are the queries we should see improvement from the existing workloads

    • big5: desc_sort_with_after_timestamp, asc_sort_with_after_timestamp
    • http_logs: desc_sort_with_after_timestamp, asc_sort_with_after_timestamp
    • geonames: asc_sort_with_after_population, desc_sort_with_after_geonameid, asc_sort_with_after_geonameid

Description

[Describe what this change achieves]

Related Issues

Resolves #18546

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions
Copy link
Contributor

github-actions bot commented Aug 6, 2025

❌ Gradle check result for 3c6596a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Contributor

github-actions bot commented Aug 6, 2025

✅ Gradle check result for 3c6596a: SUCCESS

Signed-off-by: Prudhvi Godithi <[email protected]>
@github-actions
Copy link
Contributor

github-actions bot commented Aug 6, 2025

❌ Gradle check result for 9137fe5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Contributor

github-actions bot commented Aug 6, 2025

✅ Gradle check result for 9137fe5: SUCCESS

@prudhvigodithi prudhvigodithi requested a review from msfroh August 6, 2025 16:05
@github-actions
Copy link
Contributor

github-actions bot commented Aug 6, 2025

✅ Gradle check result for b46ffcf: SUCCESS

@cwperks
Copy link
Member

cwperks commented Aug 6, 2025

For some reason @kkewwei 's approval didn't count towards merge. Approving the PR while we figure out the codeowners issue.

@prudhvigodithi prudhvigodithi merged commit 6468464 into opensearch-project:main Aug 6, 2025
34 of 35 checks passed
@github-project-automation github-project-automation bot moved this from In Progress to Done in Performance Roadmap Aug 6, 2025
RajatGupta02 pushed a commit to RajatGupta02/OpenSearch that referenced this pull request Aug 18, 2025
…ensearch-project#18896)

* Initial commit for search_after queries

Signed-off-by: Prudhvi Godithi <[email protected]>

* Test with increment and decrement with search_after

Signed-off-by: Prudhvi Godithi <[email protected]>

* search_after queries

Signed-off-by: Prudhvi Godithi <[email protected]>

* Support framework for search_after queries

Signed-off-by: Prudhvi Godithi <[email protected]>

* Fix gradle precommit issues

Signed-off-by: Prudhvi Godithi <[email protected]>

* Add comments

Signed-off-by: Prudhvi Godithi <[email protected]>

* Attempt to fix the test

Signed-off-by: Prudhvi Godithi <[email protected]>

* Attempt to fix the test

Signed-off-by: Prudhvi Godithi <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Prudhvi Godithi <[email protected]>

* Update CHANGELOG.md and fetch upstream

Signed-off-by: Prudhvi Godithi <[email protected]>

* Update tests validating with lucene searchAfter

Signed-off-by: Prudhvi Godithi <[email protected]>

* Update tests validating with lucene searchAfter

Signed-off-by: Prudhvi Godithi <[email protected]>

* Update tests validating with lucene searchAfter

Signed-off-by: Prudhvi Godithi <[email protected]>

* Update code with comments

Signed-off-by: Prudhvi Godithi <[email protected]>

* Add encode tests

Signed-off-by: Prudhvi Godithi <[email protected]>

* Fix conflicts

Signed-off-by: Prudhvi Godithi <[email protected]>

* Fix spotless

Signed-off-by: Prudhvi Godithi <[email protected]>

* Code clean up

Signed-off-by: Prudhvi Godithi <[email protected]>

* Upstream fetch

Signed-off-by: Prudhvi Godithi <[email protected]>

* Upstream Fetch

Signed-off-by: Prudhvi Godithi <[email protected]>

* Add clamps

Signed-off-by: Prudhvi Godithi <[email protected]>

* Upstream Fetch to resolve conflicts

Signed-off-by: Prudhvi Godithi <[email protected]>

---------

Signed-off-by: Prudhvi Godithi <[email protected]>
kh3ra pushed a commit to kh3ra/OpenSearch that referenced this pull request Sep 5, 2025
…ensearch-project#18896)

* Initial commit for search_after queries

Signed-off-by: Prudhvi Godithi <[email protected]>

* Test with increment and decrement with search_after

Signed-off-by: Prudhvi Godithi <[email protected]>

* search_after queries

Signed-off-by: Prudhvi Godithi <[email protected]>

* Support framework for search_after queries

Signed-off-by: Prudhvi Godithi <[email protected]>

* Fix gradle precommit issues

Signed-off-by: Prudhvi Godithi <[email protected]>

* Add comments

Signed-off-by: Prudhvi Godithi <[email protected]>

* Attempt to fix the test

Signed-off-by: Prudhvi Godithi <[email protected]>

* Attempt to fix the test

Signed-off-by: Prudhvi Godithi <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Prudhvi Godithi <[email protected]>

* Update CHANGELOG.md and fetch upstream

Signed-off-by: Prudhvi Godithi <[email protected]>

* Update tests validating with lucene searchAfter

Signed-off-by: Prudhvi Godithi <[email protected]>

* Update tests validating with lucene searchAfter

Signed-off-by: Prudhvi Godithi <[email protected]>

* Update tests validating with lucene searchAfter

Signed-off-by: Prudhvi Godithi <[email protected]>

* Update code with comments

Signed-off-by: Prudhvi Godithi <[email protected]>

* Add encode tests

Signed-off-by: Prudhvi Godithi <[email protected]>

* Fix conflicts

Signed-off-by: Prudhvi Godithi <[email protected]>

* Fix spotless

Signed-off-by: Prudhvi Godithi <[email protected]>

* Code clean up

Signed-off-by: Prudhvi Godithi <[email protected]>

* Upstream fetch

Signed-off-by: Prudhvi Godithi <[email protected]>

* Upstream Fetch

Signed-off-by: Prudhvi Godithi <[email protected]>

* Add clamps

Signed-off-by: Prudhvi Godithi <[email protected]>

* Upstream Fetch to resolve conflicts

Signed-off-by: Prudhvi Godithi <[email protected]>

---------

Signed-off-by: Prudhvi Godithi <[email protected]>
vinaykpud pushed a commit to vinaykpud/OpenSearch that referenced this pull request Sep 26, 2025
…ensearch-project#18896)

* Initial commit for search_after queries

Signed-off-by: Prudhvi Godithi <[email protected]>

* Test with increment and decrement with search_after

Signed-off-by: Prudhvi Godithi <[email protected]>

* search_after queries

Signed-off-by: Prudhvi Godithi <[email protected]>

* Support framework for search_after queries

Signed-off-by: Prudhvi Godithi <[email protected]>

* Fix gradle precommit issues

Signed-off-by: Prudhvi Godithi <[email protected]>

* Add comments

Signed-off-by: Prudhvi Godithi <[email protected]>

* Attempt to fix the test

Signed-off-by: Prudhvi Godithi <[email protected]>

* Attempt to fix the test

Signed-off-by: Prudhvi Godithi <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Prudhvi Godithi <[email protected]>

* Update CHANGELOG.md and fetch upstream

Signed-off-by: Prudhvi Godithi <[email protected]>

* Update tests validating with lucene searchAfter

Signed-off-by: Prudhvi Godithi <[email protected]>

* Update tests validating with lucene searchAfter

Signed-off-by: Prudhvi Godithi <[email protected]>

* Update tests validating with lucene searchAfter

Signed-off-by: Prudhvi Godithi <[email protected]>

* Update code with comments

Signed-off-by: Prudhvi Godithi <[email protected]>

* Add encode tests

Signed-off-by: Prudhvi Godithi <[email protected]>

* Fix conflicts

Signed-off-by: Prudhvi Godithi <[email protected]>

* Fix spotless

Signed-off-by: Prudhvi Godithi <[email protected]>

* Code clean up

Signed-off-by: Prudhvi Godithi <[email protected]>

* Upstream fetch

Signed-off-by: Prudhvi Godithi <[email protected]>

* Upstream Fetch

Signed-off-by: Prudhvi Godithi <[email protected]>

* Add clamps

Signed-off-by: Prudhvi Godithi <[email protected]>

* Upstream Fetch to resolve conflicts

Signed-off-by: Prudhvi Godithi <[email protected]>

---------

Signed-off-by: Prudhvi Godithi <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Enhancement or improvement to existing feature or request lucene Roadmap:Search Project-wide roadmap label Search:Performance v3.2.0

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

[Feature Request] Approximation Framework for queries with search_after

6 participants