[SPARK-31290][R] Add back the deprecated R APIs #28058

huaxingao · 2020-03-28T00:00:09Z

What changes were proposed in this pull request?

Add back the deprecated R APIs removed by #22843 and #22815.

These APIs are

sparkR.init
sparkRSQL.init
sparkRHive.init
registerTempTable
createExternalTable
dropTempTable

No need to port the function such as

createExternalTable <- function(x, ...) {
  dispatchFunc("createExternalTable(tableName, path = NULL, source = NULL, ...)", x, ...)
}

because this was for the backward compatibility when SQLContext exists before assuming from #9192, but seems we don't need it anymore since SparkR replaced SQLContext with Spark Session at #13635.

Why are the changes needed?

Amend Spark's Semantic Versioning Policy

Does this PR introduce any user-facing change?

Yes
The removed R APIs are put back.

How was this patch tested?

Add back the removed tests

huaxingao · 2020-03-28T00:01:26Z

cc @HyukjinKwon @felixcheung

SparkQA · 2020-03-28T00:40:42Z

Test build #120517 has finished for PR 28058 at commit 69e2953.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

huaxingao · 2020-03-28T01:03:00Z

Do we need to add back saveAsParquetFile? Seems it is not added back in scala? Also, do we need to add back sparkR.init, sparkRSQL.init, sparkRHive.init?

gatorsmile · 2020-03-28T04:08:09Z

Also cc @mengxr @falaki Could you take a look?

falaki · 2020-03-28T04:18:46Z

If we have added these back to Scala, adding them to R seems good.

dongjoon-hyun · 2020-03-29T02:02:08Z

cc @marmbrus

R/pkg/R/DataFrame.R

HyukjinKwon

LGTM except one question https://github.com/apache/spark/pull/28058/files#r399743329

felixcheung · 2020-03-29T23:51:03Z

They don’t have 1:1 to Scala. Some R APIs are older and long since removed even in Scala before 3.0. I’m fine with this change if this is what the community has agreed on.

rxin · 2020-03-29T23:53:35Z

If they have already been removed prior to 3.0 and nobody has said anything, I don't think we should add those back in.

HyukjinKwon · 2020-03-30T00:27:15Z

Just to clarify things, what the community agreed on is the rubric added at https://spark.apache.org/versioning-policy.html.

Several APIs listed here, jsonFile, parquetFile and saveAsParquetFile were removed in Scala and PySpark before Spark 3.0 at SPARK-12600. I skimmed user mailing list by keyword search and nobody complained.

One last thing is, SparkR dropped RDD support. So I don't think it also makes sense to add jsonRDD back.

HyukjinKwon · 2020-03-30T00:30:58Z

@huaxingao, seems like we roughly agreed upon excluding jsonFile, parquetFile and saveAsParquetFile at SPARK-12600. jsonRDD seems not making sense to add back since SparkR dropped RDD support officially. Can we remove these four in this PR?

SparkQA · 2020-03-30T01:46:00Z

Test build #120563 has finished for PR 28058 at commit 754f938.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon

@huaxingao, can you also add the reasons why we didn't port:

createExternalTable <- function(x, ...) {
  dispatchFunc("createExternalTable(tableName, path = NULL, source = NULL, ...)", x, ...)
}

in the PR description?

Apparently, this was for the backward compatibility when SQLContext exists before assuming from #9192 but seems we don't need it anymore since SparkR replaced SQLContext with Spark Session at #13635

R/pkg/R/catalog.R

HyukjinKwon · 2020-03-30T03:08:09Z

It seems more complicated than I thought. Let me try to clarify the current status here:

`jsonFile` and `parquetFile`

Scala side, jsonFile, parquetFile were removed at [SPARK-12600][SQL] Remove deprecated methods in Spark SQL #10559 and recovered at [SPARK-15914][SQL] Add deprecated method back to SQLContext for backward source code compatibility #13637 in the same Spark 2.0.
PySpark side, they were removed at [SPARK-12600][SQL] Remove deprecated methods in Spark SQL #10559 and were not recovered afterwards.
SparkR side, they were never removed

`jsonRDD`

SparkR made RDD API as private APIs as of SPARK-7230; RDD related APIs are supposed to be removed as they are uesless.

`saveAsParquetFile`

Scala side saveAsParquetFile was removed at [SPARK-12600][SQL] Remove deprecated methods in Spark SQL #10559
PySpark side, it was also removed at [SPARK-12600][SQL] Remove deprecated methods in Spark SQL #10559.
SparkR side, it was never removed.

I skimmed user-mailing list with keyword search, and I found no complaints. This implies, no complaints found about:

jsonFile, parquetFile and jsonRDD missing in PySpark from Spark 2.0
saveAsParquetFile in Scala and PySpark from Spark 2.0

So, I guess it's fine to don't add jsonFile, parquetFile, jsonRDD and saveAsParquetFile back.

TL;DR: from my rough investigation, I think it's fine to don't add jsonFile, parquetFile, jsonRDD and saveAsParquetFile because:

people don't quite care about jsonFile, parquetFile, jsonRDD and saveAsParquetFile.
jsonFile and parquetFile don't exist in PySpark.
jsonRDD is useless as RDD APIs became private in SparkR.
saveAsParquetFile does not exist in Scala and PySpark sides.

WDYT @falaki, @mengxr, @felixcheung?

SparkQA · 2020-03-30T04:16:07Z

Test build #120567 has finished for PR 28058 at commit ce5969b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-04-01T01:36:19Z

I discussed offline with @marmbrus. Looks we're fine to exclude these four above.

Merged to master and branch-3.0.

### What changes were proposed in this pull request? Add back the deprecated R APIs removed by #22843 and #22815. These APIs are - `sparkR.init` - `sparkRSQL.init` - `sparkRHive.init` - `registerTempTable` - `createExternalTable` - `dropTempTable` No need to port the function such as ```r createExternalTable <- function(x, ...) { dispatchFunc("createExternalTable(tableName, path = NULL, source = NULL, ...)", x, ...) } ``` because this was for the backward compatibility when SQLContext exists before assuming from #9192, but seems we don't need it anymore since SparkR replaced SQLContext with Spark Session at #13635. ### Why are the changes needed? Amend Spark's Semantic Versioning Policy ### Does this PR introduce any user-facing change? Yes The removed R APIs are put back. ### How was this patch tested? Add back the removed tests Closes #28058 from huaxingao/r. Authored-by: Huaxin Gao <[email protected]> Signed-off-by: HyukjinKwon <[email protected]> (cherry picked from commit fd0b228) Signed-off-by: HyukjinKwon <[email protected]>

huaxingao · 2020-04-01T01:41:24Z

Thank you all for the help!

### What changes were proposed in this pull request? Add back the deprecated R APIs removed by apache#22843 and apache#22815. These APIs are - `sparkR.init` - `sparkRSQL.init` - `sparkRHive.init` - `registerTempTable` - `createExternalTable` - `dropTempTable` No need to port the function such as ```r createExternalTable <- function(x, ...) { dispatchFunc("createExternalTable(tableName, path = NULL, source = NULL, ...)", x, ...) } ``` because this was for the backward compatibility when SQLContext exists before assuming from apache#9192, but seems we don't need it anymore since SparkR replaced SQLContext with Spark Session at apache#13635. ### Why are the changes needed? Amend Spark's Semantic Versioning Policy ### Does this PR introduce any user-facing change? Yes The removed R APIs are put back. ### How was this patch tested? Add back the removed tests Closes apache#28058 from huaxingao/r. Authored-by: Huaxin Gao <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>

[SPARK-31290][SQL][R] Add back the deprecated R APIs

69e2953

dongjoon-hyun changed the title ~~[SPARK-31290][SQL][R] Add back the deprecated R APIs~~ [SPARK-31290][R] Add back the deprecated R APIs Mar 29, 2020

dongjoon-hyun added the SPARKR label Mar 29, 2020

HyukjinKwon reviewed Mar 29, 2020

View reviewed changes

R/pkg/R/DataFrame.R Outdated Show resolved Hide resolved

HyukjinKwon approved these changes Mar 29, 2020

View reviewed changes

remove jsonFile, parquetFile, saveAsParquetFile and jsonRDD

754f938

HyukjinKwon approved these changes Mar 30, 2020

View reviewed changes

This comment has been minimized.

Sign in to view

HyukjinKwon reviewed Mar 30, 2020

View reviewed changes

R/pkg/R/catalog.R Show resolved Hide resolved

mark Drop Temporary Table depreacated

ce5969b

HyukjinKwon closed this in fd0b228 Apr 1, 2020

huaxingao deleted the r branch April 1, 2020 01:41

[SPARK-31290][R] Add back the deprecated R APIs #28058

[SPARK-31290][R] Add back the deprecated R APIs #28058

Uh oh!

Conversation

huaxingao commented Mar 28, 2020 • edited by HyukjinKwon Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

huaxingao commented Mar 28, 2020

Uh oh!

SparkQA commented Mar 28, 2020

Uh oh!

huaxingao commented Mar 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gatorsmile commented Mar 28, 2020

Uh oh!

falaki commented Mar 28, 2020

Uh oh!

dongjoon-hyun commented Mar 29, 2020

Uh oh!

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

felixcheung commented Mar 29, 2020 via email

Uh oh!

rxin commented Mar 29, 2020

Uh oh!

HyukjinKwon commented Mar 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented Mar 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Mar 30, 2020

Uh oh!

This comment has been minimized.

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

HyukjinKwon commented Mar 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

jsonFile and parquetFile

jsonRDD

saveAsParquetFile

Uh oh!

SparkQA commented Mar 30, 2020

Uh oh!

HyukjinKwon commented Apr 1, 2020

Uh oh!

huaxingao commented Apr 1, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

huaxingao commented Mar 28, 2020 •

edited by HyukjinKwon

Loading

huaxingao commented Mar 28, 2020 •

edited

Loading

HyukjinKwon commented Mar 30, 2020 •

edited

Loading

HyukjinKwon commented Mar 30, 2020 •

edited

Loading

HyukjinKwon commented Mar 30, 2020 •

edited

Loading

`jsonFile` and `parquetFile`

`jsonRDD`

`saveAsParquetFile`