Skip to content

[SPARK-27397][Core] Take care of OpenJ9 JVM in Spark #24308

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kiszk wants to merge 3 commits into from
Closed

[SPARK-27397][Core] Take care of OpenJ9 JVM in Spark #24308

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
96bcfd4
initial commit
kiszk Apr 6, 2019
6265481
address review comment
kiszk Apr 6, 2019
08eb67a
address review comment
kiszk Apr 15, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions core/src/main/scala/org/apache/spark/util/SizeEstimator.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,8 +131,9 @@ object SizeEstimator extends Logging {
return System.getProperty(TEST_USE_COMPRESSED_OOPS_KEY).toBoolean
}

// java.vm.info provides compressed ref info for IBM JDKs
if (System.getProperty("java.vendor").contains("IBM")) {
// java.vm.info provides compressed ref info for IBM and OpenJ9 JDKs
if (System.getProperty("java.vendor").contains("IBM") ||
Comment thread
dongjoon-hyun marked this conversation as resolved.
Outdated

@srowen srowen Apr 6, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: do you want to retrieve the value of java.vendor once here and compare it twice? the overall change seems fine, though I don't know if it means these JDKs will always work. We only test against the Oracle/OpenJDK

System.getProperty("java.vendor").contains("OpenJ9")) {
return System.getProperty("java.vm.info").contains("Compressed Ref")
}

Expand Down
5 changes: 4 additions & 1 deletion launcher/src/main/java/org/apache/spark/launcher/CommandBuilderUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ class CommandBuilderUtils {

/** The set of known JVM vendors. */
enum JavaVendor {
Oracle, IBM, OpenJDK, Unknown
Oracle, IBM, OpenJDK, OpenJ9, Unknown
}

/** Returns whether the given string is null or empty. */
Expand Down Expand Up @@ -124,6 +124,9 @@ static JavaVendor getJavaVendor() {
if (vendorString.contains("OpenJDK")) {
return JavaVendor.OpenJDK;
}
if (vendorString.contains("OpenJ9")) {

@attilapiros attilapiros Apr 8, 2019 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know who uses this getJavaVendor method?
As it is package private so only classes from the same package could access this.
But in the entire Spark code there is no usage for it:

$ find . -type f -exec grep "getJavaVendor" \{} \; -print
  static JavaVendor getJavaVendor() {
./launcher/src/main/java/org/apache/spark/launcher/CommandBuilderUtils.java

Could it be a 3rd party tool should use the package name: org.apache.spark.launcher?

@srowen srowen Apr 8, 2019 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm yeah it's unused as far as I can see. It shouldn't be used by third parties anyway. Delete it?

@attilapiros attilapiros Apr 8, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is one more method near this one also seams to me unused: firstNonEmptyValue. Should I create a Minor PR or OK to remove within this PR?

@attilapiros attilapiros Apr 8, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh and there will be the enum as well: getJavaVendor seams to me the only place where used (and mima exclude).

@srowen srowen Apr 8, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, you can delete the unused code here.
It looks like this was added to account for the fact that IBM JDKs don't have a PermGen, but Spark no longer tries to set this now that Java 8 is required and doesn't have a PermGen.

return JavaVendor.OpenJ9;
}
return JavaVendor.Unknown;
}

Expand Down