[SPARK-39459][CORE] local*HostName* methods should support IPv6

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to

• Support IPv6-only environment in localHostName, localHostNameForUri and localCanonicalHostName methods
• have no side-effects in IPv4 environment.

Why are the changes needed?

Currently, Apache Spark fails on pure-IPv6 environment (which doesn't have IPv4 address).

BEFORE

$ SPARK_LOCAL_IP=::1 build/sbt "core/testOnly *.DistributedSuite" -Djava.net.preferIPv6Addresses=true
...
Using SPARK_LOCAL_IP=::1
...
[info] *** 45 TESTS FAILED ***
[error] Failed tests:
[error] 	org.apache.spark.DistributedSuite
[error] (core / Test / testOnly) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 9 s, completed Jun 13, 2022, 8:38:03 PM

AFTER

$ SPARK_LOCAL_IP=::1 build/sbt "core/testOnly *.DistributedSuite" -Djava.net.preferIPv6Addresses=true
...
Using SPARK_LOCAL_IP=::1
...
[info] Tests: succeeded 46, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.

Does this PR introduce any user-facing change?

This will help IPv6-only environment users.

How was this patch tested?

Since we don't have IPv6 test CI, this should be tested in IPv6 environment manually with DistributedSuite and Spark-Shell.

DistributedSuite

$ SPARK_LOCAL_IP=::1 build/sbt "core/testOnly *.DistributedSuite" -Djava.net.preferIPv6Addresses=true

SPARK-SHELL

$ SPARK_LOCAL_IP=2600:...:...:c26a bin/spark-shell
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/06/13 20:17:44 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://unknown1498774f7f18.attlocal.net:4040
Spark context available as 'sc' (master = local[*], app id = local-1655176664558).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.4.0-SNAPSHOT
      /_/

Using Scala version 2.12.16 (OpenJDK 64-Bit Server VM, Java 17.0.3)
Type in expressions to have them evaluated.
Type :help for more information.

[dongjoon-hyun]

cc @dbtsai

[LuciferYang]

I found other failed cases, like

• UISeleniumSuite

- effects of unpersist() / persist() should be reflected *** FAILED ***
  java.net.ConnectException: Connection refused (Connection refused)
  at java.net.PlainSocketImpl.socketConnect(Native Method)
  at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
  at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
  at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
  at java.net.Socket.connect(Socket.java:613)
  at java.net.Socket.connect(Socket.java:561)
  at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
  at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
  at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
  ...
- failed stages should not appear to be active *** FAILED ***
  java.net.ConnectException: Connection refused (Connection refused)
  at java.net.PlainSocketImpl.socketConnect(Native Method)
  at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
  at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
  at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
  at java.net.Socket.connect(Socket.java:613)
  at java.net.Socket.connect(Socket.java:561)
  at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
  at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
  at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)

• UISuite

- SPARK-36237: Attach and start handler after application started in UI  *** FAILED ***
  java.net.ConnectException: Connection refused (Connection refused)
  at java.net.PlainSocketImpl.socketConnect(Native Method)
  at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
  at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
  at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
  at java.net.Socket.connect(Socket.java:613)
  at java.net.Socket.connect(Socket.java:561)
  at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
  at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
  at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)

[LuciferYang]

• RocksDBBackendHistoryServerSuite

- application list json *** FAILED ***
  java.net.ConnectException: Connection refused (Connection refused)
  at java.net.PlainSocketImpl.socketConnect(Native Method)
  at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
  at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
  at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
  at java.net.Socket.connect(Socket.java:613)
  at java.net.Socket.connect(Socket.java:561)
  at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
  at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
  at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)

It seems that HTTP related cases will fail?

[dongjoon-hyun]

There are totally independent from this patch. I'm working on them, too, @LuciferYang . :)

[dongjoon-hyun]

Just FYI, scala.io.Source.fromURL seems to be unable to support IPv6 for now. I'm digging that part.

[dongjoon-hyun]

The second PR is here and more will come.

• [SPARK-39464][CORE][TESTS] Use Utils.localCanonicalHostName instead of localhost in tests #36866

[dongjoon-hyun]

All tests passed in the previous commit (before empty commit).

[dongjoon-hyun]

Could you review this when you have a chance please, @HyukjinKwon ?

[HyukjinKwon]

Looks fine. cc @Ngone51 and @jiangxb1987 FYI

[dongjoon-hyun]

Thank you, @HyukjinKwon and @LuciferYang .
Merged to master.