Investigate potential gap in bwc testing for search requests

[cbuescher]

While working on #75745 I accidentally broke the wire serialization of SearchRequest on 7.x and was suprised to find that not all hell broke lose when e.g. running the mixed cluster rest tests. I created an experimental PR that adds a new boolean flag to the SearchRequest serialization on 8.0 #77889, which without a bwc layer should break communication with older 7.x nodes. It appears the only thing that failed was a CCS rolling upgrade test (SearchStatesIT) which I find to be a poot indicator of such a big break.
I'd like to understand better why the mixed cluster tests don't seem to be too sensitive to this kind of change and potentially modify them or add additional test that would surface these situations better.

[cbuescher]

It seems like we seldom need to serialize/deserialize SearchRequest. Changing serialization in e.g. ShardSearchRequest slightly breaks tons of tests in :qa:mixed-cluster as expected, so this might not be such a huge gap but something specific to SearchRequest. Perhaps its really only CCS scenarios in which we send SearchRequest over the wire.

[javanna]

Good point, I created #84481 to expand CCS tests coverage. The two may be related, as I think from your conclusion above CCS is one of the few cases where we serialize the SearchRequest (when ccs_minimize_roundtrips is enabled). I wonder if we should also have some more specific serialization unit tests for the SearchRequest? Would that have caught the wire serialization issue that was introduced?

[cbuescher]

I wonder if we should also have some more specific serialization unit tests for the SearchRequest?

We do have somewhat randomized serialization roundtrip testing in SearchRequestTests, also a variation ("testRandomVersionSerialization") that reads and writes to a stream with a different version setting, but in the later case both versions (for the input and output stream) are the same and also the actual reading/writing logic under test is from the current class under test, so that won't catch all things. I think unit tests are fine but ultimately this is something what mixed cluster were supposed to be for.

Would that have caught the wire serialization issue that was introduced?

I don't think so. The issue with #75745 was that we introduced serialization of a new boolean flag to 7.x but nothing to master, so the PR passed but then tests on master were failing as far as I can recall. As you can see in #77889, if we do brutally stupid changes to SearchRequest, at least something fails, but I was expecting way more to break.