Reporting regression: reports always retrying regardless of outcome

[joelgriffith]

Summary

I've been experiencing a recent regression, where, reports are continuously retrying regardless of status. Upon further investigation it appears that our queries for outstanding jobs were returning jobs of all statuses. This PR fixes the underlying issue (which appears to be that should doesn't necessarily need to match in order to return). I'm not sure why this wasn't an issue prior, perhaps a bugfix that landed in ES or its deps.

Checklist

Use strikethroughs to remove checklist items you don't feel are applicable to this PR.

• [ ] This was checked for cross-browser compatibility, including a check against IE11
• [ ] Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• [ ] Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• [ ] This was checked for keyboard-only and screenreader accessibility

[tsullivan]

We want a query that's equivalent of

jobtype == this.jobtype &&
  (status == "pending" ||
    (status == "processing" && process_expiration <= nowTime))

I'd recommend trying to do this without using constant_score. Can we try:

{
  query: {
    bool: {
      filter: {
        bool: {
          must: { term: { jobtype: this.jobtype } },
          should: [
            { term: { status: 'pending' } },
            {
              bool: {
                must: [
                  { term: { status: 'processing' } },
                  { range: { process_expiration: { lte: nowTime } } }
                ]
              }
            }
          ]
        }
      }
    }
  }
}

[joelgriffith]

Dope! I still needed the minimum_should_match: 1 in order for this to work, just an FYI

[tsullivan]

With further digging, I find that adding minimum_should_match: 1 was perhaps the only change needed.

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[tsullivan]

I believe after this PR goes in, we can revert the changes to reporting tests in #26877

[tsullivan]

LGTM

[tsullivan]

I got word that the change in ES around this was elastic/elasticsearch#35354

[tsullivan]

6.x/6.6.0: #27082