Skip to content

[SPARK-19828][R] Support array type in from_json in R #17178

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HyukjinKwon wants to merge 5 commits into from
Closed

[SPARK-19828][R] Support array type in from_json in R #17178

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
10feb9d
Support array in from_json in R
HyukjinKwon Mar 6, 2017
f13ad18
Add asArray option for from_json
HyukjinKwon Mar 10, 2017
5c2450d
Remove unrelated changes
HyukjinKwon Mar 10, 2017
d01f952
Clean up and name it asJsonArray
HyukjinKwon Mar 14, 2017
8ec7cd0
Minor comments
HyukjinKwon Mar 14, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions R/pkg/R/functions.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2437,6 +2437,7 @@ setMethod("date_format", signature(y = "Column", x = "character"),
#'
#' @param x Column containing the JSON string.
#' @param schema a structType object to use as the schema to use when parsing the JSON string.
#' @param asJsonArray indicating if input string is JSON array or object.

@felixcheung felixcheung Mar 14, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe clarifying
JSON array or object. -> JSON array of objects or a single object.

#' @param ... additional named properties to control how the json is parsed, accepts the same
#' options as the JSON data source.
#'
Expand All @@ -2452,11 +2453,18 @@ setMethod("date_format", signature(y = "Column", x = "character"),
#'}
#' @note from_json since 2.2.0
setMethod("from_json", signature(x = "Column", schema = "structType"),
function(x, schema, ...) {
function(x, schema, asJsonArray = FALSE, ...) {

@HyukjinKwon HyukjinKwon Mar 14, 2017

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@felixcheung, I am still not fully sure of this name, asJsonArray. I am okay if you have a better one.

if (asJsonArray) {
jschema <- callJStatic("org.apache.spark.sql.types.DataTypes",
"createArrayType",

@felixcheung felixcheung Mar 12, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need a wrapper, actually? can't we call newJObject?

@HyukjinKwon HyukjinKwon Mar 13, 2017

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, sure. Let me try.

@HyukjinKwon HyukjinKwon Mar 14, 2017

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems we'd better use

DataTypes.createArrayType

per

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/types/ArrayType.scala

Lines 49 to 56 in 04ee8cf

* The data type for collections of multiple values.
* Internally these are represented as columns that contain a ``scala.collection.Seq``.
*
* Please use `DataTypes.createArrayType()` to create a specific instance.
*
* An [[ArrayType]] object comprises two fields, `elementType: [[DataType]]` and
* `containsNull: Boolean`. The field of `elementType` is used to specify the type of
* array elements. The field of `containsNull` is used to specify if the array has `null` values.

Let me remove that new wrapper and use the original one.

schema$jobj)
} else {
jschema <- schema$jobj
}
options <- varargsToStrEnv(...)
jc <- callJStatic("org.apache.spark.sql.functions",
"from_json",
x@jc, schema$jobj, options)
x@jc, jschema, options)
column(jc)
})

Expand Down
12 changes: 12 additions & 0 deletions R/pkg/inst/tests/testthat/test_sparkSQL.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1364,6 +1364,18 @@ test_that("column functions", {
# check for unparseable
df <- as.DataFrame(list(list("a" = "")))
expect_equal(collect(select(df, from_json(df$a, schema)))[[1]][[1]], NA)

# check if array type in string is correctly supported.
jsonArr <- "[{\"name\":\"Bob\"}, {\"name\":\"Alice\"}]"
df <- as.DataFrame(list(list("people" = jsonArr)))
schema <- structType(structField("name", "string"))
arr <- collect(select(df, alias(from_json(df$people, schema, asJsonArray = TRUE), "arrcol")))
expect_equal(ncol(arr), 1)
expect_equal(nrow(arr), 1)
expect_is(arr[[1]][[1]], "list")
expect_equal(length(arr$arrcol[[1]]), 2)
expect_equal(arr$arrcol[[1]][[1]]$name, "Bob")
expect_equal(arr$arrcol[[1]][[2]]$name, "Alice")
})

test_that("column binary mathfunctions", {
Expand Down
2 changes: 1 addition & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/api/r/SQLUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ private[sql] object SQLUtils extends Logging {
new JavaSparkContext(spark.sparkContext)
}

def createStructType(fields : Seq[StructField]): StructType = {
def createStructType(fields: Seq[StructField]): StructType = {
StructType(fields)
}

Expand Down