[SPARK-46525][SQL][TESTS][3.5] Fix docker-integration-tests on Apple Silicon

[dongjoon-hyun]

What changes were proposed in this pull request?

This is a merged backport of SPARK-46525 with the original authorship, @yaooqinn .

• [SPARK-46525][DOCKER][TESTS] Fix docker-integration-tests on Apple Sillicon #44509
• [SPARK-46525][DOCKER][TESTS][FOLLOWUP] Fix docker-integration-tests on Apple Silicon for db2 and oracle with third-party docker environments #44612
• [SPARK-46525][BUILD][TESTS][FOLLOWUP] Cleanup http client deps for spotify docker client #45303

com.spotify.docker.client is not going to support Apple Silicons as it has already been archived and the jnr-unixsocket 0.18 it uses is not compatible with Apple Silicons.

If we run our docker IT tests on Apple Silicons, it will fail like

[info] org.apache.spark.sql.jdbc.MariaDBKrbIntegrationSuite *** ABORTED *** (2 seconds, 264 milliseconds)
[info]   com.spotify.docker.client.exceptions.DockerException: java.util.concurrent.ExecutionException: 
com.spotify.docker.client.shaded.javax.ws.rs.ProcessingException:
java.lang.UnsatisfiedLinkError: could not load FFI provider jnr.ffi.provider.jffi.Provider
...
[info]   Cause: java.lang.IllegalStateException: Can't overwrite cause with java.lang.UnsatisfiedLinkError: 
java.lang.UnsatisfiedLinkError: /Users/hzyaoqin/spark/target/tmp/jffi15403099445119552969.dylib:
dlopen(/Users/hzyaoqin/spark/target/tmp/jffi15403099445119552969.dylib, 0x0001): tried: 
'/Users/hzyaoqin/spark/target/tmp/jffi15403099445119552969.dylib' (fat file, but missing compatible architecture (have 'i386,x86_64', need 'arm64')), 
'/System/Volumes/Preboot/Cryptexes/OS/Users/hzyaoqin/spark/target/tmp/jffi15403099445119552969.dylib' (no such file), '/Users/hzyaoqin/spark/target/tmp/jffi15403099445119552969.dylib' (fat file, but missing compatible architecture (have 'i386,x86_64', need 'arm64'))

In this PR, we use its alternative to enable docker-related tests on Apple Chips

    <dependency>
      <groupId>com.github.docker-java</groupId>
      <artifactId>docker-java</artifactId>
      <scope>test</scope>
    </dependency>

Why are the changes needed?

For developers who use Apple Silicons, w/ this patch, they can test JDBC/Docker Integration test locally instead of suffering slowness from GitHub actions.

Does this PR introduce any user-facing change?

No, dev only

How was this patch tested?

Pass the CIs and do the manual test on Apple Silicon.

$ build/sbt -Pdocker-integration-tests 'docker-integration-tests/testOnly org.apache.spark.sql.jdbc.*MariaDB*'
...
[info] All tests passed.
[success] Total time: 157 s (02:37), completed Sep 27, 2024, 2:45:16 PM

$ build/sbt -Pdocker-integration-tests 'docker-integration-tests/testOnly org.apache.spark.sql.jdbc.*MySQL*'
...
[info] All tests passed.
[success] Total time: 109 s (01:49), completed Sep 27, 2024, 2:48:47 PM

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

Hi, @yaooqinn . It would be great if we can have this test coverage in the release branches. Could you review this as the author?

[dongjoon-hyun]

Please note that MsSqlServer suites are currently broken in branch-3.5/3.4. This PR is very helpful during the investigation because we can investigate on Mac environment.

• https://github.com/apache/spark/actions/runs/11069480476/job/30784081196 (branch-3.5)
• https://github.com/apache/spark/actions/runs/11067744916/job/30784071170 (branch-3.4)

[info] *** 3 SUITES ABORTED ***
[error] Error during tests:
[error] 	org.apache.spark.sql.jdbc.v2.MsSqlServerNamespaceSuite
[error] 	org.apache.spark.sql.jdbc.v2.MsSqlServerIntegrationSuite
[error] 	org.apache.spark.sql.jdbc.MsSqlServerIntegrationSuite

[dongjoon-hyun]

Thank you so much, @HyukjinKwon . I'll merge this and continue to my investigation!

[yaooqinn]

Late LGTM

[dongjoon-hyun]

For the record, the daily CIs of branch-3.5/branch-3.4 are recovered with this and SPARK-47949 . Thank you, @yaooqinn .