Spark: Moves Distribution and Ordering Utils to Spark3 Module #2820
Conversation
|
@aokolnychyi + @rdblue I would like to have access to these Utilities for doing some work with the Sort Rewrite Action. If anyone has another Idea on how to get access to this functionality I would be glad to hear it. Ideally I would like to keep Scala and this code out of the Spark3 module, but I can't have an Action use code from within the Extensions module. |
RussellSpitzer
force-pushed
the
MoveSortToolsToSpark3
branch
from
954dfeb to
44dec68
Compare
| scala.srcDirs = ['src/main/scala', 'src/main/java'] | ||
| java.srcDirs = [] | ||
| } | ||
| // Compile test source against Spark 3.1 and main classes compiled against Spark 3.0 |
There was a problem hiding this comment.
Do we need to do anything with the spark31 sourceSet?
There was a problem hiding this comment.
Nope, we still use the "main" source set
build.gradle
Outdated
|
|
||
| sourceSets { | ||
| main { | ||
| // Use Scala Compiler for everything so we can do cross compilation within the module |
There was a problem hiding this comment.
I don't think I understand why we can't use the default scala plugin settings. This is disabling Java compilation by removing srcDirs. Should we just remove the Java plugin?
|
I tried to convert some of these classes to Java today and didn't get anywhere. Some of the traits we need to implement just don't exist for some reason and I think it's probably better just to do it this way. |
RussellSpitzer
force-pushed
the
MoveSortToolsToSpark3
branch
from
44dec68 to
58924c8
Compare
|
@rdblue Are we good to go with this? |
To enable core Spark3 code to use distribution and ordering information when performing writes or other operations. One example is to allow Sort Strategy to transform SortOrder information so it can be used when rewriting data files.
RussellSpitzer
force-pushed
the
MoveSortToolsToSpark3
branch
from
58924c8 to
5daa960
Compare
|
Looks good to me. The other build settings weren't needed after all? |
Yep I'm not sure what I was thinking :) |
3 participants
To enable core Spark3 code to use distribution and ordering information when
performing writes or other operations. One example is to allow Sort Strategy
to transform SortOrder information so it can be used when rewriting data files.