Add allow_partial_search_results flag to search requests with default setting true

[markharwood]

When false, will error if search either timeouts, has partial errors or has missing shards rather
than returning partial search results. A cluster-level setting provides a default for searches
with no flag.

Closes #27435

[markharwood]

Initial commit here just for CI checking that forward-port of 6.x-based #27906 was successful. Once green I'll add a commit that changes the default for allowPartialSearchResults to false (meaning errors will be thrown by default when it looks like partial results would be returned)

[colings86]

I left a couple of comments int eh tests.

[colings86]

Could you explain why this is needed? This test doesn't appear to test partial results?

[markharwood]

That was a necessity from a previously failing trial on 6.x - if I recall correctly it was for searches that rely on the use of field data. Ditto the other test

[colings86]

Could you explain why this is needed? This test doesn't appear to test partial results?

[markharwood]

Reverted the code that defaults searches to strict (error on partial results). Will open a separate issue once this is pushed to consider changing the default to strict.

[markharwood]

CI build failures look to all be related to BWC checks with the 6.3 snapshot that doesn't yet have the new allowPartialSearchResults flag serializing in SearchRequest

[jasontedor]

If I can assist in integrating a change like this with cross-cutting BWC implications, please let me know. I am always happy to help.

[markharwood]

@jasontedor Thanks for the offer of help. I have 2 PRs lined up ready to push (if I can get an OK on this one, @colings86 ?)
This PR is for master and is failing on BWC, but I have a seperate 6.x PR which is passing tests.
I thought the best way to sequence this might be to:

1. push the 6.x PR which should be green but will then cause all master BWC search tests to fail (master will not de/serialize search requests to 6.x)
2. push this master PR which should then hopefully be green

Not sure if there needs to be a gap between 1) and 2) waiting for the 6.x build to make it through the intake tests?

After all this is done I'll open a separate issue to open the debate on whether new cluster default setting is to fail on partial results.

[colings86]

LGTM

[jasontedor]

@markharwood I think that this would help keep the build green every step of the way and let you integrate this at a leisurely pace:

• push a commit to this PR switching all version-dependent serialization logic to Version.V_7_0_0
• push a commit to this PR that disables BWC testing; make a change to the root build.gradle setting ext.bwc_tests_enabled to false in all projects.
• at this point, you should be able to get a completely green build
• merge this PR
• prepare a backport of this change to 6.x, changing the version-dependent serialization logic to Version.V_6_3_0, and setting ext.bwc_tests_enabled to true
• at this point, you should be able to get a completely green build on 6.x, and integrating this change there will have no impact on master
• backport this PR with the above changes to 6.x
• prepare a commit against master that changes the version-dependent serialization logic to Version.V_6_3_0 and reenables ext.bwc_tests_enabled; at this point you can run the BWC tests and they should be green
• integrate the above change of the version logic and reenabling BWC into master