-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-16581][SPARKR] Make JVM backend calling functions public #14775
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 5 commits
472072a
d772285
5f4b53a
5f29361
d267f2f
0959208
448de0c
d1ec80b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -25,9 +25,23 @@ isInstanceOf <- function(jobj, className) { | |
| callJMethod(cls, "isInstance", jobj) | ||
| } | ||
|
|
||
| # Call a Java method named methodName on the object | ||
| # specified by objId. objId should be a "jobj" returned | ||
| # from the SparkRBackend. | ||
| #' Call Java Methods | ||
| #' | ||
| #' Call a Java method in the JVM running the Spark driver. | ||
| #' | ||
| #' @param objId object to invoke the method on. Should be a "jobj" created by newJObject. | ||
|
||
| #' @param methodName method name to call. | ||
|
||
| #' @param ... parameters to pass to the Java method. | ||
| #' @export | ||
| #' @seealso callJStatic, newJObject | ||
|
||
| #' @examples | ||
|
||
| #' \dontrun{ | ||
| #' sparkR.session() # Need to have a Spark JVM running before calling newJObject | ||
| #' # Create a Java ArrayList and populate it | ||
| #' jarray <- newJObject("java.util.ArrayList") | ||
| #' callJMethod(jarray, "add", 42L) | ||
| #' callJMethod(jarray, "get", 0L) # Will print 42 | ||
| #' } | ||
| callJMethod <- function(objId, methodName, ...) { | ||
| stopifnot(class(objId) == "jobj") | ||
| if (!isValidJobj(objId)) { | ||
|
|
@@ -37,12 +51,42 @@ callJMethod <- function(objId, methodName, ...) { | |
| invokeJava(isStatic = FALSE, objId$id, methodName, ...) | ||
| } | ||
|
|
||
| # Call a static method on a specified className | ||
| #' Call Static Java Methods | ||
| #' | ||
| #' Call a static method in the JVM running the Spark driver. | ||
| #' | ||
| #' @param className class containing the static method to invoke. | ||
|
||
| #' @param methodName name of static method to invoke. | ||
| #' @param ... parameters to pass to the Java method. | ||
| #' @export | ||
| #' @seealso callJMethod, newJObject | ||
|
||
| #' @examples | ||
| #' \dontrun{ | ||
|
||
| #' sparkR.session() # Need to have a Spark JVM running before calling callJStatic | ||
| #' callJStatic("java.lang.System", "currentTimeMillis") | ||
| #' callJStatic("java.lang.System", "getProperty", "java.home") | ||
| #' } | ||
| callJStatic <- function(className, methodName, ...) { | ||
| invokeJava(isStatic = TRUE, className, methodName, ...) | ||
| } | ||
|
|
||
| # Create a new object of the specified class name | ||
| #' Create Java Objects | ||
| #' | ||
| #' Create a new Java object in the JVM running the Spark driver. | ||
| #' | ||
| #' @param className name of the class to create | ||
|
||
| #' @param ... arguments to be passed to the constructor | ||
| #' @export | ||
| #' @seealso callJMethod, callJStatic | ||
|
||
| #' @examples | ||
| #' \dontrun{ | ||
| #' sparkR.session() # Need to have a Spark JVM running before calling newJObject | ||
| #' # Create a Java ArrayList and populate it | ||
| #' jarray <- newJObject("java.util.ArrayList") | ||
| #' callJMethod(jarray, "add", 42L) | ||
| #' callJMethod(jarray, "get", 0L) # Will print 42 | ||
| #' } | ||
| #' @note newJObject since 2.0.1 | ||
| newJObject <- function(className, ...) { | ||
| invokeJava(isStatic = TRUE, className, methodName = "<init>", ...) | ||
| } | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -82,7 +82,20 @@ getClassName.jobj <- function(x) { | |
| callJMethod(cls, "getName") | ||
| } | ||
|
|
||
| cleanup.jobj <- function(jobj) { | ||
| #' Garbage collect Java Objects | ||
| #' | ||
| #' Garbage collect an object allocated on Spark driver JVM heap. | ||
| #' | ||
| #' cleanup.jobj is a low level method that lets developers manually garbage collect objects | ||
| #' allocated using newJObject. This is only to be used for advanced use cases as objects allocated | ||
| #' on the JVM heap are automatically garbage collected when the corresponding R reference goes out | ||
| #' of scope. | ||
| #' | ||
| #' @param x the Java object that should be garbage collected. | ||
| #' @note cleanup.jobj since 2.0.1 | ||
|
||
| #' @export | ||
| cleanup.jobj <- function(x) { | ||
|
||
| jobj <- x | ||
| if (isValidJobj(jobj)) { | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I guess since this is back to an internal method we dont need to worry about it. But I guess I can add the check if we still want it ?
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we are good for now if we are not exporting |
||
| objId <- jobj$id | ||
| # If we don't know anything about this jobj, ignore it | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| # | ||
| # Licensed to the Apache Software Foundation (ASF) under one or more | ||
| # contributor license agreements. See the NOTICE file distributed with | ||
| # this work for additional information regarding copyright ownership. | ||
| # The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| # (the "License"); you may not use this file except in compliance with | ||
| # the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| # | ||
|
|
||
| context("JVM API") | ||
|
|
||
| sparkSession <- sparkR.session(enableHiveSupport = FALSE) | ||
|
|
||
| test_that("Create and call methods on object", { | ||
| jarr <- newJObject("java.util.ArrayList") | ||
| # Add an element to the array | ||
| callJMethod(jarr, "add", 1L) | ||
| # Check if get returns the same element | ||
| expect_equal(callJMethod(jarr, "get", 0L), 1L) | ||
| }) | ||
|
|
||
| test_that("Call static methods", { | ||
| # Convert a boolean to a string | ||
| strTrue <- callJStatic("java.lang.String", "valueOf", TRUE) | ||
| expect_equal(strTrue, "true") | ||
| }) | ||
|
|
||
| test_that("Manually garbage collect objects", { | ||
| jarr <- newJObject("java.util.ArrayList") | ||
| cleanup.jobj(jarr) | ||
| # Using a jobj after GC should throw an error | ||
| expect_error(print(jarr), "Error in invokeJava.*") | ||
| }) | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. add
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done |
||
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am a little confused about the discussion at https://issues.apache.org/jira/browse/SPARK-16611:
It seems that systemML still uses automatic GC to cleanup jobjs as it is said that "use it’s finalize method to register the cleanup.jobj ". But a jobj already has cleanup.jobj registered as its finalizer.
If my understanding is correct, I don't see strong reason for exposing cleanup.jobj. If active cleanup is demanded, a call to "gc" can be made?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@aloknsingh
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @sun-rui
Thanks for pointing this out. I look at the code and you are right in mentioning that the jobj has the cleanup.jobj registered.
Initially, we were under the assumption that we have to register the created obj, cleanup table, otherwise the jvm (running scala) won't cleanup our obj, as there is the ref in the SparkR.
In summary,
thanks,
Alok
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @sun-rui and @aloknsingh - I'll update the PR today