Skip to content

Conversation

@dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Jun 17, 2022

What changes were proposed in this pull request?

This PR aims to propagate java.net.preferIPv6Addresses=true in SBT tests.

  • This also fixes several hive module (which downloads Hive or Hadoop dependencies) failures.
  • Maven will be handled separately.

Why are the changes needed?

To test IPv6 SBT tests, we need to set consistently with the SBT_OPTS.

java.net.preferIPv6Addresses (default: false)

When dealing with a host which has both IPv4 and IPv6 addresses, and if IPv6 is available on the operating system, the default behavior is to prefer using IPv4 addresses over IPv6 ones. This is to ensure backward compatibility, for example applications that depend on the representation of an IPv4 address (e.g. 192.168.1.1). This property can be set to true to change that preference and use IPv6 addresses over IPv4 ones where possible, or system to preserve the order of the addresses as returned by the operating system.

Does this PR introduce any user-facing change?

This is a test-only change.

How was this patch tested?

Pass the GitHub Action with new test cases.

  1. New test case will be disabled by default or when java.net.preferIPv6Addresses is not true.

  2. With java.net.preferIPv6Addresses=true in IPv6 environment, new test case will pass.

[info] SparkSubmitUtilsSuite:
:: loading settings :: url = jar:file:/Users/dongjoon/Library/Caches/Coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/ivy/ivy/2.5.0/ivy-2.5.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
[info] - SPARK-39501: Resolve maven dependenicy in IPv6 (243 milliseconds)
[info] Run completed in 1 second, 141 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.

@github-actions github-actions bot added the BUILD label Jun 17, 2022
@dongjoon-hyun dongjoon-hyun marked this pull request as draft June 17, 2022 05:46
@dongjoon-hyun dongjoon-hyun changed the title [SPARK-39501][TESTS] Use java.net.preferIPv6Addresses=true in tests [SPARK-39501][TESTS] Use java.net.preferIPv6Addresses=true in tests Jun 17, 2022
@dongjoon-hyun dongjoon-hyun changed the title [SPARK-39501][TESTS] Use java.net.preferIPv6Addresses=true in tests [WIP][SPARK-39501][TESTS] Use java.net.preferIPv6Addresses=true in tests Jun 17, 2022
@dongjoon-hyun dongjoon-hyun changed the title [WIP][SPARK-39501][TESTS] Use java.net.preferIPv6Addresses=true in tests [WIP][SPARK-39501][TESTS] Propagate java.net.preferIPv6Addresses=true in tests Jun 17, 2022
@github-actions github-actions bot added the CORE label Jun 17, 2022
…ests

SBT_OPTS='-Djava.net.preferIPv6Addresses=true' build/sbt "core/testOnly *.SparkSubmitUtilsSuite -- -z IPv6"
@dongjoon-hyun dongjoon-hyun changed the title [WIP][SPARK-39501][TESTS] Propagate java.net.preferIPv6Addresses=true in tests [WIP][SPARK-39501][TESTS] Propagate java.net.preferIPv6Addresses=true in SBT tests Jun 17, 2022
@dongjoon-hyun dongjoon-hyun changed the title [WIP][SPARK-39501][TESTS] Propagate java.net.preferIPv6Addresses=true in SBT tests [SPARK-39501][TESTS] Propagate java.net.preferIPv6Addresses=true in SBT tests Jun 17, 2022
@dongjoon-hyun dongjoon-hyun marked this pull request as ready for review June 17, 2022 07:46
Copy link
Contributor

@LuciferYang LuciferYang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dongjoon-hyun
Copy link
Member Author

Thank you, @LuciferYang .

@dongjoon-hyun
Copy link
Member Author

All tests passed. Could you review this, @HyukjinKwon ?
Screen Shot 2022-06-17 at 2 30 49 AM

@HyukjinKwon
Copy link
Member

Merged to master.

@dongjoon-hyun
Copy link
Member Author

Thank you, @HyukjinKwon !

@dongjoon-hyun dongjoon-hyun deleted the SPARK-39501 branch June 17, 2022 15:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants