Add support to testclusters for older versions of elasticsearch

[alpar-t]

I ran into trouble while back-porting some of the recent bwc changes, so decided to add this test that would make sure we can start all bwc versions of elasticsearch.
For now it will only run as part of bwcTest so it doesn't add additional time to intake or PRs.

This revealed some changes that were required to be able to have clusters running on older versions. Some of these will not be encountered in master but since they don't add too much complexit, I think it would be better to have them and make back-ports easier.

One added benefit of this test is that we check that testclusters actually started the version and flavor we tell it to.

[elasticmachine]

Pinging @elastic/es-core-infra

[rjernst]

This seems like overkill to me. I don't think we should be referencing 6.x versions in master; it would be yet another thing that has to be updated on major version bumps. And bwc tests should be run when backporting if the change affects bwc. Perhaps we need a label to trigger full bwc tests like we have for packaging? It also seems like some of these deficiencies here could be caught by better testing within build-tools.

[mark-vieira]

I think I need some more context on this. Perhaps we should sync up and discuss.

[alpar-t]

I do think we need a full integration test for things like elasticsearch settings and access patterns for distributions, these are hard to mock and the value of the test is reduced because it doesn't verify assumptions we make when implementing.

The 6.x versions don't need to be updated on new releases. I'm also fine with removing that part,. We never call bwcTest in CI any more, we call the version specific v<version>#bwcTest and we do that for index compatible versions only, so the 6.x series will only be tested on 7.x, but having it there allows one to explicitly test from master, so it's more like a tool that was useful for testing that the back-port will work, rather than doing a dance where it doesn't and then master needs an update which can't be tested there. The alternative of doing it only in 7.x puts us on a path that will make it difficult to port changes back.

[alpar-t]

#45654 is the backport that needs these changes first.

@elasticmachine run elasticsearch-ci/1 adn run elasticsearch-ci/bwc

[mark-vieira]

Perhaps we can split out the smoke test bit and just have the stuff needed to backport the BWC changes to 7.x. We can then separately discuss the test coverage issue w/o holding up those backport changes.

[alpar-t]

Thanks @mark-vieira that's a great idea, I'll do that