Fix sort on date fields with missing values (#73763)

[hanjongho]

hi @cbuescher, @jimczi
I'm trying to find and share the error in the logic of sort with missing, order method together in the date field.

while using .missing("_last").sort(SortOrder.DESC) in date field (doc does not have a date field),
sort value return -9223372036854775808L by the following logic

can simulate this case with this order

1. Put Index Settings

curl -X PUT "localhost:9200/my-index-000001" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "properties": {
      "date_nanos_test": {
        "type": "date_nanos"
      },
      "date_test": {
        "type": "date"
      }
    }
  }
}
'

2. Bulk 2 documents that has date_nanos, date type each

curl -X PUT "localhost:9200/my-index-000001/_bulk?refresh" -H 'Content-Type: application/json' -d'
{"index":{"_id":"1"}}
{"date_test":"2024-11-01"}
{"index":{"_id":"2"}}
{"date_nanos_test":"2024-11-01T23:50:00.000000001"}
'

3. Query with .missing("_last").sort(SortOrder.DESC) in date_nanos field

curl -X GET "localhost:9200/my-index-000001/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "size": 5,
  "sort": [
    {
      "date_nanos_test": {
        "order": "desc",
        "missing": "_last"
      }
    }
  ]
}
'

3.1 Actual Result

{
  "took" : 7,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [
      {
        "_index" : "my-index-000001",
        "_id" : "2",
        "_score" : null,
        "_source" : {
          "date_nanos_test" : "2024-11-01T23:50:00.000000001"
        },
        "sort" : [
          1730505000000000001
        ]
      },
      {
        "_index" : "my-index-000001",
        "_id" : "1",
        "_score" : null,
        "_source" : {
          "date_test" : "2024-11-01"
        },
        "sort" : [
          0
        ]
      }
    ]
  }
}

this works well, fixed in #74760

4. Query with .missing("_last").sort(SortOrder.DESC) in date field

curl -X GET "localhost:9200/my-index-000001/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "size": 5,
  "sort": [
    {
      "date_test": {
        "order": "desc",
        "missing": "_last"
      }
    }
  ]
}
'

4.1 Actual Result

{
  "took" : 7,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [
      {
        "_index" : "my-index-000001",
        "_id" : "1",
        "_score" : null,
        "_source" : {
          "date_test" : "2024-11-01"
        },
        "sort" : [
          1730419200000
        ]
      },
      {
        "_index" : "my-index-000001",
        "_id" : "2",
        "_score" : null,
        "_source" : {
          "date_nanos_test" : "2024-11-01T23:50:00.000000001"
        },
        "sort" : [
          -9223372036854775808
        ]
      }
    ]
  }
}

Suggest

when I searched for the issues related to this error, I found #73763, which is the same issue about date_nanos.
I checked the PR #74760 (commited by @cbuescher) for the issue. when I added the date type to the logic (PR) and tested it in my local, 0L responded well normally. and when I tested it with order asc, needed Long.MAX_VALUE, elastic search's specification.

please understand if there is something wrong in my explanation. review it when you have time.
thank you 👍

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[hanjongho]

hi @drempapis @smalyshev
could you please review this PR? I’m happy to update it if you have feedback.
thx