dfs_query_then_fetch can cause serialization errors with 6.x nodes

[jtibshirani]

In a mixed 6.x and 7.x cluster, a search that uses dfs_query_then_fetch can cause a transport serialization error:

[2021-07-14T14:48:54,223][DEBUG][o.e.a.s.TransportSearchAction] [v6.8.11-2] [16] Failed to execute query phase
 org.elasticsearch.transport.RemoteTransportException: [v6.8.11-0][127.0.0.1:50319][indices:data/read/search[phase/query/id]]
Caused by: java.lang.IllegalArgumentException: totalTermFreq must be positive, totalTermFreq: -1
      at org.apache.lucene.search.TermStatistics.<init>(TermStatistics.java:70) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
      at org.elasticsearch.search.dfs.AggregatedDfs.<init>(AggregatedDfs.java:37) ~[elasticsearch-6.8.11.jar:6.8.11]
      at org.elasticsearch.search.query.QuerySearchRequest.<init>(QuerySearchRequest.java:48) ~[elasticsearch-6.8.11.jar:6.8.11]
      at org.elasticsearch.transport.RequestHandlerRegistry.newRequest(RequestHandlerRegistry.java:48) ~[elasticsearch-6.8.

This seems related to https://issues.apache.org/jira/browse/LUCENE-8007, which was introduced in Lucene 8 and adds stricter checks to TermStatistics.

I was able to reproduce this with our rolling upgrade tests. A rough example:

1. In the old 6.8 cluster, create a couple documents with a keyword value like "field": "some_value".
2. In a mixed 6.8 and 7.x cluster, perform a search on the field:

   GET /index/_search?search_type=dfs_query_then_fetch
   {
     "query": {
       "match": {
         "field": {
           "query": "some_value"
         }
       }
     }
   }
   

This search can then fail with the same serialization error.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[jpountz]

Oh good catch and sorry for not thinking about this when merging this Lucene change!

[163satish]

Waiting for this fix

[ywelsch]

Fixed by #75735

[nathandh22]

FYI, @ywelsch. *** raised some concerns that the Bug and PR tickets were not mentioned in our Elasticsearch version 7.14.0 release notes, but I have pointed out that while this seems to be true, the code changes do appear in the 7.14.0 codebase.

[ywelsch]

@nathandh22 I've redacted your post, as you shouldn't call out specific customers in public issues.

As the fix (#75735) got merged very late in the 7.14.0 release cycle, I suspect that it wasn't properly picked up by the doc release for 7.14.0 in #75873. @probakowski can you check what went wrong there?