[SPARK-26090][CORE][SQL][ML] Resolve most miscellaneous deprecation and build warnings for Spark 3 #23065

srowen · 2018-11-17T02:52:36Z

What changes were proposed in this pull request?

The build has a lot of deprecation warnings. Some are new in Scala 2.12 and Java 11. We've fixed some, but I wanted to take a pass at fixing lots of easy miscellaneous ones here.

They're too numerous and small to list here; see the pull request. Some highlights:

@BeanInfo is deprecated in 2.12, and BeanInfo classes are pretty ancient in Java. Instead, case classes can explicitly declare getters
Eta expansion of zero-arg methods; foo() becomes () => foo() in many cases
Floating-point Range is inexact and deprecated, like 0.0 to 100.0 by 1.0
finalize() is finally deprecated (just needs to be suppressed)
StageInfo.attempId was deprecated and easiest to remove here

I'm not now going to touch some chunks of deprecation warnings:

Parquet deprecations
Hive deprecations (particularly serde2 classes)
Deprecations in generated code (mostly Thriftserver CLI)
ProcessingTime deprecations (we may need to revive this class as internal)
many MLlib deprecations because they concern methods that may be removed anyway
a few Kinesis deprecations I couldn't figure out
Mesos get/setRole, which I don't know well
Kafka/ZK deprecations (e.g. poll())
Kinesis
a few other ones that will probably resolve by deleting a deprecated method

How was this patch tested?

Existing tests, including manual testing with the 2.11 build and Java 11.

srowen · 2018-11-17T02:52:55Z

common/unsafe/pom.xml

LevenshteinDistance moved here from commons lang

srowen · 2018-11-17T02:53:21Z

core/src/main/scala/org/apache/spark/scheduler/StageInfo.scala

Funny, MiMa didn't warn about this, but will go in release notes

srowen · 2018-11-17T02:53:47Z

external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaContinuousTest.scala

Complains about existential types if you access a method in an anonymous inner class

srowen · 2018-11-17T02:54:01Z

external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala

Should be an exact workalike invocation here, just works around a deprecation

srowen · 2018-11-17T02:54:34Z

...l/kinesis-asl/src/main/java/org/apache/spark/examples/streaming/JavaKinesisWordCountASL.java

Most of the Kinesis changes are of this form, to use the new client builder, with the same argument going in as far as I can tell. The rest are Java 8 cleanups.

srowen · 2018-11-17T02:55:02Z

...sl/src/test/java/org/apache/spark/streaming/kinesis/JavaKinesisInputDStreamBuilderSuite.java

These assertions were wrong in two ways: == and assert

srowen · 2018-11-17T02:55:23Z

mllib/src/main/scala/org/apache/spark/ml/feature/LabeledPoint.scala

These are added to explicitly add what the companion BeanInfo class was implicitly adding

srowen · 2018-11-17T02:56:17Z

mllib/src/test/scala/org/apache/spark/ml/feature/QuantileDiscretizerSuite.scala

Interestingly, avoiding double Ranges actually fixed the code here. You can see the bucketing before didn't quite make sense. Now it's even. It's because of...

scala> (0.0 to 1.0 by 1.0 / 10).toList <console>:12: warning: method to in trait FractionalProxy is deprecated (since 2.12.6): use BigDecimal range instead (0.0 to 1.0 by 1.0 / 10).toList ^ res5: List[Double] = List(0.0, 0.1, 0.2, 0.30000000000000004, 0.4, 0.5, 0.6, 0.7, 0.7999999999999999, 0.8999999999999999, 0.9999999999999999) scala> (0 to 10).map(_.toDouble / 10).toList res6: List[Double] = List(0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0)

srowen · 2018-11-17T02:56:56Z

mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala

The filter here was nonsensical, comparing a value to type. Removed it as it's always true

srowen · 2018-11-17T02:58:16Z

...re/src/test/scala/org/apache/spark/sql/execution/streaming/ProcessingTimeExecutorSuite.scala

Yes, this is the recommended way to get a concurrent Set in the JDK

SparkQA · 2018-11-17T06:38:11Z

Test build #98948 has finished for PR 23065 at commit e2e375b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-11-17T15:52:29Z

Test build #4429 has finished for PR 23065 at commit e2e375b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-11-18T05:44:47Z

Test build #98972 has finished for PR 23065 at commit 4665696.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2018-11-18T15:01:15Z

Hm, Kinesis tests are failing. They pass locally when I set my credentials and enable the tests. They also seem to pass on Jenkins, like on this recent PR: https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4432/testReport/org.apache.spark.streaming.kinesis/

Clearly it's something to do with my changes, but not sure what, as they seem to work locally. If I can't figure it out after another guess I'll back out the kinesis changes.

SparkQA · 2018-11-18T21:25:59Z

Test build #98983 has finished for PR 23065 at commit 0cfcd90.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gengliangwang · 2018-11-19T12:07:10Z

Hi @srowen ,
Could you review and merge srowen#4 ?
I see a lot of warnings as well. We should fix them.

srowen · 2018-11-19T14:44:49Z

@gengliangwang that will fail in Scala 2.11; can't do it unfortunately.

gengliangwang · 2018-11-19T14:47:57Z

@srowen you are right. It will failed in the SQL module.

srowen · 2018-11-19T15:17:00Z

Merged to master

gengliangwang

LGTM.

…nd build warnings for Spark 3 ## What changes were proposed in this pull request? The build has a lot of deprecation warnings. Some are new in Scala 2.12 and Java 11. We've fixed some, but I wanted to take a pass at fixing lots of easy miscellaneous ones here. They're too numerous and small to list here; see the pull request. Some highlights: - `BeanInfo` is deprecated in 2.12, and BeanInfo classes are pretty ancient in Java. Instead, case classes can explicitly declare getters - Eta expansion of zero-arg methods; foo() becomes () => foo() in many cases - Floating-point Range is inexact and deprecated, like 0.0 to 100.0 by 1.0 - finalize() is finally deprecated (just needs to be suppressed) - StageInfo.attempId was deprecated and easiest to remove here I'm not now going to touch some chunks of deprecation warnings: - Parquet deprecations - Hive deprecations (particularly serde2 classes) - Deprecations in generated code (mostly Thriftserver CLI) - ProcessingTime deprecations (we may need to revive this class as internal) - many MLlib deprecations because they concern methods that may be removed anyway - a few Kinesis deprecations I couldn't figure out - Mesos get/setRole, which I don't know well - Kafka/ZK deprecations (e.g. poll()) - Kinesis - a few other ones that will probably resolve by deleting a deprecated method ## How was this patch tested? Existing tests, including manual testing with the 2.11 build and Java 11. Closes apache#23065 from srowen/SPARK-26090. Authored-by: Sean Owen <[email protected]> Signed-off-by: Sean Owen <[email protected]>

srowen commented Nov 17, 2018

View reviewed changes

srowen added 2 commits November 17, 2018 13:01

Resolve most miscellaneous deprecations and some build warnings

d2e6699

Increase timeouts

4665696

srowen force-pushed the SPARK-26090 branch from e2e375b to 4665696 Compare November 18, 2018 01:35

Revert Kinesis changes

0cfcd90

gengliangwang reviewed Nov 19, 2018

View reviewed changes

asfgit closed this in 32365f8 Nov 20, 2018

srowen deleted the SPARK-26090 branch November 26, 2018 13:15

[SPARK-26090][CORE][SQL][ML] Resolve most miscellaneous deprecation and build warnings for Spark 3 #23065

[SPARK-26090][CORE][SQL][ML] Resolve most miscellaneous deprecation and build warnings for Spark 3 #23065

Uh oh!

Conversation

srowen commented Nov 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 17, 2018

Uh oh!

SparkQA commented Nov 17, 2018

Uh oh!

SparkQA commented Nov 18, 2018

Uh oh!

srowen commented Nov 18, 2018

Uh oh!

SparkQA commented Nov 18, 2018

Uh oh!

gengliangwang commented Nov 19, 2018

Uh oh!

srowen commented Nov 19, 2018

Uh oh!

gengliangwang commented Nov 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

srowen commented Nov 19, 2018

Uh oh!

gengliangwang left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

srowen commented Nov 17, 2018 •

edited

Loading

gengliangwang commented Nov 19, 2018 •

edited

Loading