Snap 293 #310
Snap 293 #310
Conversation
Now user need to implement isValidJob and runSnappyJob methods for all jobs added contextClassLoader for driver and client thread. it wil enable it to read the classes from the snappy store repository.
it handles executor class loader per jar file. still need to implement clientfirst classloader
it supports both childFirst and parentFirst jarLoading added basic test coverage
added full JUnit coverage for DynamicURLClassLoader
with Junit and DUnit coverage
SnappySQLJob can be used for same purpose
|
|
||
| } | ||
|
|
||
|
|
There was a problem hiding this comment.
This override can also be added for IsolatedClientLoader.classLoader. It is used for any external JDBC URLs apart from our gemxd drivers (e.g. if user wishes to use it to load data into gemxd). If it is not added there, we need docs on how user should go about doing it (perhaps JDBC drivers have to be in SPARK_DIST_CLASSPATH while other parts will be fine with dynamic install-jar).
There was a problem hiding this comment.
I will remove the following line
private val overwriteFiles = env.conf.getBoolean("spark.files.overwrite", false)
as it is not needed after the changes in the job server. The Job server always create a new jar file where we have added our own suffix in the name so now executor knows that the jar belongs to the same job and it can reload it. executors can also copy this jar as it is a new jar for them.
|
@rishitesh Please review the Java job API changes in this. |
| public abstract class JavaSnappyStreamingJob implements SparkJobBase { | ||
|
|
||
| abstract public Object runJavaJob(JavaSnappyStreamingContext snc, Config jobConfig); | ||
| abstract public Object runSnappyJob(JavaSnappyStreamingContext snc, Config jobConfig); |
There was a problem hiding this comment.
Why this rename? I suppose the idea was to emphasize that the code needs to use the Java wrapper JavaSnappyStreamingContext (and not the regular SnappyStreamingContext). @rishitesh your thoughts?
There was a problem hiding this comment.
It was more to do with type resolution in case of scala generic type. The API proposed is looking cleaner if its running all the tests. As use only have to deal with two methods. Only input types will change.
| else new File(tempDir, jarName.format(System.currentTimeMillis())) | ||
| TestUtils.createJar(files1 ++ files2, jarFile) | ||
| } | ||
|
|
There was a problem hiding this comment.
Since this is being used only be tests, move this to io.snappydata.util.TestUtils
|
Haven't looked at the tests yet but looked through pretty much rest of the code. Please ensure we have tests for: We also need docs for how user should install/replace/drop jars as well as JDBC drivers which is common case users try to load import data into snappydata. If c) is not possible, then we need to document and test that too (SPARK_DIST_CLASSPATH) |
|
Added another pull request after incorporating the review comments and 2.0 merge. For some open issues with PR open the Jira https://jira.snappydata.io/browse/SNAP-999?filter=-1 |
Changes proposed in this pull request
Now only two methods are exposed to the user runSnappyJob and isValidJob with a single return Type SnappyValidation so user has a small learning curve and limited entry points.
Patch testing
dunit , junit , clean precheckin with store
ReleaseNotes.txt changes
No. but will update the related *md files for same.
Other PRs
TIBCOSoftware/snappy-spark-jobserver#1
TIBCOSoftware/snappy-spark#39
TIBCOSoftware/snappy-store#88