[SPARK-26132][BUILD][CORE] Remove support for Scala 2.11 in Spark 3.0.0

[srowen]

What changes were proposed in this pull request?

Remove Scala 2.11 support in build files and docs, and in various parts of code that accommodated 2.11. See some targeted comments below.

How was this patch tested?

Existing tests.

[srowen]

@felixcheung is it OK to refer to _2.12 artifacts here? I don't think this one actually exists, but is it just an example?

[vanzin]

I think there's even a separate discussion about even using this as an example, since that package is now in Spark since 2.4.

[srowen]

@felixcheung was the conclusion that we can make this a dummy package? I just want to avoid showing _2.11 usage here.

[felixcheung]

yes, dummy name is completely fine with me.

[srowen]

@gengliangwang this was the update I was talking about to the .cmd script. You can follow up with this change, uncommented, if you like, separately from this PR.

[srowen]

@vanzin @cloud-fan you may want to look at this. It's getting a little hairy in this script.

I recall that the goal was to use this script to create older Spark releases, so it needs logic for older versions. But looking at it, I don't think it actually creates quite the same release as older versions anyway. Is it OK to clean house here and assume only Spark 3 will be built from this script? I already deleted some really old logic here (Spark < 2.2)

[vanzin]

I prefer keeping the script in master working against all currently supported versions. I find it pretty hard to keep things in sync across different branches, especially if you need to fix things in the middle of an RC cycle. Having master be the source of truth for this makes some of that pain goes away, with the cost of some added logic here.

I think the main problem now is that for 3.0 the default Scala version is different than for 2.4, which is the only added complication I can think of here...

[srowen]

Just a question though, if you're releasing 2.3 wouldn't you use the release script as of 2.3?

I think the script works for Spark 1.6 and 2.x, but not earlier versions, already. What about drawing the lines at major releases, so that this can simplify further?

Right now I think it's basically trying to support 2.2+, which are the non-EOL releases, which seems reasonable?

[vanzin]

if you're releasing 2.3 wouldn't you use the release script as of 2.3?

I started like that when I RM'ed 2.3.1, but the scripts were kinda broken that I had to fix a bunch of stuff in master. At that point, it didn't make sense to use the scripts in the 2.3 branch anymore, and keeping them in sync was kinda wasted effort.

basically trying to support 2.2+, which seems reasonable?

That's my view. The release scripts in master should be usable for all non-EOL releases.

[srowen]

OK, the issue is that the release script for 2.3.0 that works might only appear in 2.3.1 because of some last-minute hacks during the release process. I can see being careful not to drop 2.3.0 logic immediately after 2.3.0. Maybe at 2.4.0, but that's aggressive.

At least, sure, we can forget Spark < 2.3 in Spark 3.0.0's build script. Maybe later we decide to go further. I'll work on making it that way.

[srowen]

@maryannxue I think you helped work on this part ... if we're only on Scala 2.12 now can we simplify this further?

[SparkQA]

Test build #99086 has finished for PR 23098 at commit 0420c2d.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #99087 has finished for PR 23098 at commit 1c1f000.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gengliangwang]

I am not familiar with .cmd script. Should we keep the quote here, "2.12"?

[HyukjinKwon]

Nope, the string became as is including quotes if it's quoted on Windows ... haha odd (to me).

[srowen]

Oh I see, so we shouldn't add quotes to values like SPARK_ENV_CMD above, but use them in if conditions, call, etc?

[HyukjinKwon]

Yea, I think we shouldn't quote if I remember this correctly. Let me test and get back to you today or tomorrow. I'll have to fly to Korea for my one week vacation from tomorrow :D.

[HyukjinKwon]

but at least that's what I remember when I tested https://github.com/apache/spark/blob/master/bin/spark-sql2.cmd#L23 line.

[HyukjinKwon]

For call, it's a bit different (it can be quoted IIRC)(https://ss64.com/nt/call.html)

[HyukjinKwon]

Here, I ran some of simple commands:

C:\>set A=aa

C:\>ECHO %A%
aa

C:\>set A="aa"

C:\>ECHO %A%
"aa"

C:\>call "python.exe"
Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> exit(0)

C:\>call python.exe
Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> exit(0)

[SparkQA]

Test build #99382 has finished for PR 23098 at commit 2b8d770.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Note I'm holding on to this PR for a while as I understand it might be disruptive to downstream builds to remove 2.11 support just now. Will look at merging it in weeks. Right now it's an FYI.

[SparkQA]

Test build #99667 has finished for PR 23098 at commit 96f9c41.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Still holding on to this. I intend to merge it before Spark 3, but have gotten feedback that it would be helpful to hold off for a while.

[SparkQA]

Test build #100974 has finished for PR 23098 at commit 87fc1f1.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dbtsai]

LGTM for removing Scala 2.11.

[srowen]

Rebased to keep this up to date; not merging just yet

[SparkQA]

Test build #102713 has finished for PR 23098 at commit ffbad3e.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

@srowen @dbtsai Have we decided to remove Scala 2.11 support in 3.0? If not, shall we start a vote?

[srowen]

I think there is support for it, and don't know if we need a vote. Part of the reason I delay is that it's not a hurry, and in a way I wouldn't mind keeping Spark 3 pretty nearly compatible with 2.11, even if we won't support it. Leaving the 2.11 build maybe lets us avoid unnecessary incompatibilities with 2.11. I was planning to check on support again for this in a month or so.

[SparkQA]

Test build #103686 has started for PR 23098 at commit d3adda6.

[skonto]

@srowen +100 for removing 2.11, let's move forward, 2.11 is old and Spark 3.0.0 gives us an opportunity to get rid of it. I believe we should focus on 2.12 and how to support 2.13.

[SparkQA]

Test build #4655 has finished for PR 23098 at commit d3adda6.

• This patch fails due to an unknown error code, -9.
• This patch does not merge cleanly.
• This patch adds no public classes.

[srowen]

Going to merge this if the next set of test passes. Last call for comments.

[SparkQA]

Test build #103850 has finished for PR 23098 at commit b212c27.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #103861 has finished for PR 23098 at commit 7372853.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Merged to master

[jzhuge]

@srowen So awesome to merge this. Does it mean we can revert [SPARK-27250][TEST-MAVEN][BUILD] Scala 2.11 maven compile should target Java 1.8 now?

[srowen]

@jzhuge no that change is still valid. We still need to target Java 1.8 now, and will need to continue to do so even when Java 11 support works

[kiszk]

@sowen I realized that this PR causes an error when we execute do-release-docker.sh for branch-2.3

...
Checked out revision 35358.
Copying release tarballs
cp: cannot stat 'pyspark-*': No such file or directory

For Spark 2.4, PUBLISH_SCALA_2_11 = 1, PUBLISH_SCALA_2_10=1
For Spark 2.3, PUBLISH_SCALA_2_11 = 1, PUBLISH_SCALA_2_10=0
Since This change does not pass "withpip,withr", i think the script does not build pyspark-* and SparkR_*.

Should we use the original code in the following if then ... fi:

  if [[ $PUBLISH_SCALA_2_11 = 1 ]]; then
    ...
  fi

cc @dongjoon-hyun

[dongjoon-hyun]

Hi, @kiszk . You should not use master branch release script.
For branch-2.4, I'm using the release script in branch-2.4 as you see my VOTE email.

cc @gatorsmile , @vanzin

[dongjoon-hyun]

BTW, that's the reason why we made 2.4.3 urgently after 2.4.2.

[dongjoon-hyun]

FYI, I'm blocked by recent Jekyll release issue. I'll make a PR soon against master and branch-2.4. I'll ping you on there, too.

[kiszk]

@dongjoon-hyun Thank you for your suggestion. Got it for 2.4.3.
I will use the script in branch-2.3.

[kiszk]

I will add it to https://spark.apache.org/release-process.html

[dongjoon-hyun]

BTW, I didn't test branch-2.3 release script before. As you know, I've been only testing on branch-2.4. Please test it before releasing.

[kiszk]

Branch-2.3 is pretty old. I realized there is no script do-release-docker.sh. I will check the old page or adopt do-release-docker.sh for release 2.3.

[dongjoon-hyun]

Although this is Apache Spark 2.3.x EOL release, you need to commit to branch-2.3 whatever you do. Otherwise, it will not be reproducible for the others.

[kiszk]

Yes, you are right. If backporting of do-release-docker.sh works well, I will commit it to branch-2.3.