Unable to run ElasticSearch integration tests in parallel

[ghost]

This issue relates to a few issues already created, namely #2217 and #2224 - none of which solved the original problem so far.

#2217 was about randomizing ports to avoid collisions between parallel builds when running Allegro+ES but it was not sufficient to prevent issues that ultimately failed the tests (e.g. later issues were about Allegro "unable" to access the /tmp folders it requires for temp data, broadly speaking).

#2224 tackled the problem at Jenkins level, by attempting to throttle concurrent builds. Aside from the fact that it doesn't seem to work at least when builds are (re-)started manually, it seems a bit overkill to limit our build concurrency just for some integration tests to work.

The current situation leaves 3 options open (in order of priority to investigate):

1. Investigate the Allegro framework to see if we didn't miss out an existing option to allow running it concurrently
2. If not, either consider getting rid of Allegro altogether and spinning a Docker container with ES directly (port randomization would then be very relevant)
3. Or, investigate the Allegro source itself and find a way to allow concurrent runs - then propose it in the project and use a fork in the meantime

Will start with n. 1 shortly.

[ghost]

Regarding point 1 - research what Allegro offers in terms of build/test parallelism.

There is a section of their documentation that specifically addresses this...
... but fails to provide clarity on our specific use case.

We already have PopularProperties.TRANSPORT_TCP_PORT and PopularProperties.HTTP_PORT randomized per implementation of #2217 , which did solve one of the issues but not the overall problem.

We do not use withInstallationDirectory, so that section of the docs is either inaccurate (cannot reuse same installation dir when using concurrently) or irrelevant to our case.

Per @t-gauss ' changes, we are using withInResourceLocation pointing to a zip with ES.

Things to try

• Specify a randomized withInstallationDirectory for each running test, or
• Not clean the installation directory (withCleanInstallationDirectoryOnStop(false) - defaults to true when not specified).

These experiments will require:

1. Having two fake pull requests to run concurrent builds
2. Unblocking our pipeline (currently stuck at yet another compliance issue only @t-gauss can waive)

Will wait until tomorrow and get cracking again if all goes well.

[ghost]

This seems to have partially worked, but the tests then complained about timeouts to start ES emanating from Allegro itself.
Looks like the default value was 15".
I changed it to a more generous 1' in #2242.

After merging the latest into those PRs and running them concurrently, I can see everything green again, which I find encouraging.

There is no 100% certainty these cumulated tweaks will hold indefinitely, but it's encouraging enough.

Will have to monitor a while longer but I think I can close this and #2242.