[ci] Add PR pipeline for testing ipv6

[brianseeders]

Overview

This PR is to support running most of our tests in an IPv6-only environment.

The running environment looks like this:

• External IPv6 address, no external IPv4
• NAT64 and DNS64 to that IPv4-only services on the open Internet are still reachable via IPv6
• localhost resolves to the IPv6 loopback address only
• Java/gradle is run with java.net.preferIPv6Addresses=true
• ES test clusters started with "network.host": "_local:ipv6_" by default when java.net.preferIPv6Addresses=true

Stuff in this PR:

• Add an optional PR pipeline for running tests with IPv6, can be enabled by adding test-ipv6 label
• Add a weekly periodic pipeline for running the IPv6 tests on main branches weekly
• Refactor tests all over the place that have "127.0.0.1", "localhost", etc hard-coded
• Refactor tests all over the place that construct URLs using string concat, which fails when the host string is an IPv6 address, e.g. http://::1:80 needs to be http://[::1]:80

Skipped Tests

There are a few sets of tests that were skipped when running with IPv6:

.../azure/classic/AzureDiscoveryClusterFormationTests.java

I'm not really sure what was ultimately going on here. Possibly a problem with an Azure SDK. The responsible team will need to address.

Several yamlRestCompatTest

There are several YAML tests that needed to be updated to parse IPv6 IP addresses correctly. However, these tests check out the YAML that exists on older branches and uses it to run tests, so these tests have to be skipped until the YAML is updated on the old branches.

Any test using HdfsFixture.java

Hadoop doesn't support IPv6, so these tests fail. See comment in the code for more details.

Any test using Krb5kDcContainer.java

There's configuration for this fixture hard-coded to 127.0.0.1, and fixing it is more complicated than just changing that. The responsible team will have to take care of this one.

Running locally

Here is how I run the tests locally (we should add a command or something that does this for convenience):

export JAVA_TOOL_OPTIONS="-Djava.net.preferIPv6Addresses=true -Djava.net.preferIPv4Stack=false"
export _JAVA_OPTIONS="-Djava.net.preferIPv6Addresses=true -Djava.net.preferIPv4Stack=false"

./gradlew -Djava.net.preferIPv6Addresses=true -Djava.net.preferIPv4Stack=false <tasks>

[elasticsearchmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[brianseeders]

@DaveCTurner @rjernst Adding you two as well since I believe you were involved in some of the initial discussions. I'm most curious about your thoughts on how close this gets us to being able to say that standalone Elasticsearch officially supports IPv6.

[DaveCTurner]

I think if we're going to exclude certain things from these tests then we need to document that they're unsupported in an IPv6-only environment. IMO it'd be ok to say that for these things:

• Azure discovery
• HDFS
• Kerberos
• Bracketed IPv6 literals in Azure SDK

I think we should fix HttpClient5SslTests by creating a certificate that works with both 127.0.0.1 and ::1. We have the CA key here from which we can generate more certificates right?

For the YAML test skips, will this skip all of those YAML tests forevermore? Is there a way we could skip them only for branches 9.3 and earlier, so that we start to get this covered from 9.4 onwards?

I'm concerned that we're not running any Docker/testcontainers tests: I think that's too much of a loss of test coverage.