Allow format sort values of date fields #70357

dnhatn · 2021-03-13T03:29:46Z

If a search after request targets multiple indices and some of its sort field has type date in one index but date_nanos in other indices, then Elasticsearch won't interpret the search_after parameter correctly in every target index. The sort value of a date field by default is a long of milliseconds since the epoch while a date_nanos field is a long of nanoseconds.

This commit introduces the format parameter in the sort field so a sort value of a date or date_nanos will be formatted using a date format in a search response.

The below example illustrates how to use this new parameter.

{
    "query": {
        "match_all": {}
    },
    "sort": [
        {
            "timestamp": { 
                "order": "asc", 
                "format": "strict_date_optional_time_nanos" 
           }
        }
    ]
}

{
    "query": {
        "match_all": {}
    },
    "sort": [
        {
            "timestamp": { 
                "order": "asc",
                "format": "strict_date_optional_time_nanos" 
            }
        }
    ],
    "search_after": [
        "2015-01-01T12:10:30.123456789Z" // in `strict_date_optional_time_nanos` format 
    ]
}

Closes #69192

elasticmachine · 2021-03-13T19:01:56Z

Pinging @elastic/es-search (Team:Search)

dnhatn · 2021-03-13T20:21:33Z

@jrodewig Would you mind taking a look at the doc changes? Thank you!

jrodewig

LGTM. Thanks @dnhatn!

I left a couple of non-blocking suggestions that you can ignore if wanted.

docs/reference/search/search-your-data/sort-search-results.asciidoc

docs/reference/search/search-your-data/paginate-search-results.asciidoc

…iidoc Co-authored-by: James Rodewig <[email protected]>

….asciidoc Co-authored-by: James Rodewig <[email protected]>

dnhatn · 2021-03-15T15:14:21Z

@jrodewig Thanks for the quick review. I've integrated your suggestions. Both of them are great 👍.

dnhatn · 2021-03-15T15:41:17Z

@elasticmachine update branch

jimczi

I left one comment regarding the docs, LGTM otherwise.

docs/reference/search/search-your-data/paginate-search-results.asciidoc

server/src/main/java/org/elasticsearch/search/sort/FieldSortBuilder.java

jrodewig

Expanded docs LGTM. Thanks @dnhatn.

mayya-sharipova · 2021-03-16T15:54:25Z

server/src/internalClusterTest/java/org/elasticsearch/search/searchafter/SearchAfterIT.java

+
+        resp = client().prepareSearch("test")
+            .addSort(SortBuilders.fieldSort("start_date").setFormat("dd/MM/yyyy"))
+            .addSort(SortBuilders.fieldSort("end_date")) // it's okay as we use the default format, but the output does not format


Isn't there a requirement that searchAfter formats and formats in Sort should match?

Or it is ok because "yyyy-MM-dd" is a default formant for end_date?

Or it is ok because "yyyy-MM-dd" is a default formant for end_date?

This is okay because end_date has the format yyyy-MM-dd. I reworded this comment in 9c537d5.

Isn't there a requirement that searchAfter formats and formats in Sort should match?

The formats (specified by sort) will try to parse the search_after parameters. If the parameters do not conform, then we will fail the request as a bad request.

thanks for the clarifications, Nhat

mayya-sharipova

@dnhatn Thanks Nhat, excellent work! All the changes LGTM!

dnhatn · 2021-03-17T01:26:56Z

Thanks everyone :).

If a search after request targets multiple indices and some of its sort field has type `date` in one index but `date_nanos` in other indices, then Elasticsearch won't interpret the search_after parameter correctly in every target index. The sort value of a date field by default is a long of milliseconds since the epoch while a date_nanos field is a long of nanoseconds. This commit introduces the `format` parameter in the sort field so a sort value of a date or date_nanos will be formatted using a date format in a search response. The below example illustrates how to use this new parameter. ```js { "query": { "match_all": {} }, "sort": [ { "timestamp": { "order": "asc", "format": "strict_date_optional_time_nanos" } } ] } ``` ```js { "query": { "match_all": {} }, "sort": [ { "timestamp": { "order": "asc", "format": "strict_date_optional_time_nanos" } } ], "search_after": [ "2015-01-01T12:10:30.123456789Z" // in `strict_date_optional_time_nanos` format ] } ``` Closes elastic#69192

If a search after request targets multiple indices and some of its sort field has type `date` in one index but `date_nanos` in other indices, then Elasticsearch won't interpret the search_after parameter correctly in every target index. The sort value of a date field by default is a long of milliseconds since the epoch while a date_nanos field is a long of nanoseconds. This commit introduces the `format` parameter in the sort field so a sort value of a date or date_nanos will be formatted using a date format in a search response. The below example illustrates how to use this new parameter. ```js { "query": { "match_all": {} }, "sort": [ { "timestamp": { "order": "asc", "format": "strict_date_optional_time_nanos" } } ] } ``` ```js { "query": { "match_all": {} }, "sort": [ { "timestamp": { "order": "asc", "format": "strict_date_optional_time_nanos" } } ], "search_after": [ "2015-01-01T12:10:30.123456789Z" // in `strict_date_optional_time_nanos` format ] } ``` Closes #69192

Relates #70357

dnhatn mentioned this pull request Mar 13, 2021

Formatted sort values for search_after #69192

Closed

Allow format sort values of date fields

c1d7f75

dnhatn force-pushed the format-sort-values branch from 1093423 to c1d7f75 Compare March 13, 2021 19:00

dnhatn marked this pull request as ready for review March 13, 2021 19:01

dnhatn added the :Search/Search Search-related issues that do not fall into other categories label Mar 13, 2021

elasticmachine added the Team:Search Meta label for search team label Mar 13, 2021

dnhatn added >enhancement v8.0.0 v7.13.0 and removed Team:Search Meta label for search team labels Mar 13, 2021

dnhatn requested review from jimczi and mayya-sharipova March 13, 2021 19:02

jrodewig self-requested a review March 13, 2021 23:00

jrodewig approved these changes Mar 15, 2021

View reviewed changes

docs/reference/search/search-your-data/sort-search-results.asciidoc Outdated Show resolved Hide resolved

docs/reference/search/search-your-data/paginate-search-results.asciidoc Outdated Show resolved Hide resolved

dnhatn and others added 2 commits March 15, 2021 11:11

Update docs/reference/search/search-your-data/sort-search-results.asc…

7bc7e47

…iidoc Co-authored-by: James Rodewig <[email protected]>

Update docs/reference/search/search-your-data/paginate-search-results…

802d860

….asciidoc Co-authored-by: James Rodewig <[email protected]>

Merge branch 'master' into format-sort-values

e126e31

jimczi approved these changes Mar 16, 2021

View reviewed changes

docs/reference/search/search-your-data/paginate-search-results.asciidoc Outdated Show resolved Hide resolved

server/src/main/java/org/elasticsearch/search/sort/FieldSortBuilder.java Outdated Show resolved Hide resolved

dnhatn added 3 commits March 16, 2021 11:06

explain numeric type

ca1b1cd

make it public

3fa2ab6

Merge branch 'master' into format-sort-values

d766039

jrodewig approved these changes Mar 16, 2021

View reviewed changes

mayya-sharipova reviewed Mar 16, 2021

View reviewed changes

wording

9c537d5

dnhatn requested a review from mayya-sharipova March 16, 2021 17:43

mayya-sharipova approved these changes Mar 16, 2021

View reviewed changes

dnhatn merged commit 8b5aa84 into elastic:master Mar 17, 2021

dnhatn deleted the format-sort-values branch March 17, 2021 01:27

dnhatn added the backport pending label Mar 17, 2021

dnhatn added a commit that referenced this pull request Mar 21, 2021

Adjust wire compact version

fa51d3f

Relates #70357

dnhatn mentioned this pull request Mar 21, 2021

Allow format sort values of date fields #70606

Merged

dnhatn removed the backport pending label Mar 21, 2021

majagrubic mentioned this pull request Apr 13, 2021

[Discover] Sort option: Introduce format parameter elastic/kibana#96774

Merged

2 tasks

stevejgordon mentioned this pull request Apr 21, 2021

7.13.0 Meta Ticket elastic/elasticsearch-net#5584

Closed

62 tasks

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Allow format sort values of date fields #70357

Allow format sort values of date fields #70357

Uh oh!

Conversation

dnhatn commented Mar 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Mar 13, 2021

Uh oh!

dnhatn commented Mar 13, 2021

Uh oh!

jrodewig left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dnhatn commented Mar 15, 2021

Uh oh!

dnhatn commented Mar 15, 2021

Uh oh!

jimczi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jrodewig left a comment

Choose a reason for hiding this comment

Uh oh!

mayya-sharipova Mar 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dnhatn Mar 16, 2021

Choose a reason for hiding this comment

Uh oh!

mayya-sharipova Mar 16, 2021

Choose a reason for hiding this comment

Uh oh!

mayya-sharipova left a comment

Choose a reason for hiding this comment

Uh oh!

dnhatn commented Mar 17, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

dnhatn commented Mar 13, 2021 •

edited

Loading

mayya-sharipova Mar 16, 2021 •

edited

Loading