Add ExecutorPlugin methods called on Task Start/End #713
Conversation
fsamuel-bs
changed the title
fsamuel-bs
changed the title
| // Static value modified by testing plugins to ensure plugins are called correctly. | ||
| public static int numOnTaskStart = 0; | ||
| public static int numOnTaskSucceeded = 0; | ||
| public static int numOnTaskFailed = 0; |
There was a problem hiding this comment.
Apologies, not following. Why is non-static insufficient to ensure that?
There was a problem hiding this comment.
We could, I was following the pattern of https://github.com/palantir/spark/blob/master/core/src/test/java/org/apache/spark/ExecutorPluginSuite.java#L35. I think this is a preference thing, and I particularly would rather have these be variables in each plugin. Will refactor such this is the case.
There was a problem hiding this comment.
Actually, we need a static variable because we don't have access to the instances of the plugin being instantiated here (plugins are instantiated on the spark executors). The way upstream does now is a bit more verbose, but still relies on static variables... this is a bit simpler, so will leave like this.
core/src/test/scala/org/apache/spark/scheduler/TaskContextSuite.scala
Outdated
Show resolved
Hide resolved
rshkv
left a comment
rshkv
left a comment
There was a problem hiding this comment.
Very cool! I left some minor comments but can merge as-is.
fsamuel-bs
force-pushed
the
ssouza/onTask-executorPlugin
branch
from
a19df5e to
42d60c2
Compare
Upstream SPARK-33088 ticket and apache#29977
Done.
What changes were proposed in this pull request?
Add a set of endpoints on
ExecutorPluginwhich are called on the lifecycle of aTask. Main usage of such endpoints is for tracing purposes, i.e. to propagate the inbound tracing information fromTaskContext#getLocalPropertiesto a tracing framework of choice (in our case, https://github.com/palantir/tracing-java/).How was this patch tested?
Unit tests on
ExecutorPluginTaskSuite.Please review http://spark.apache.org/contributing.html before opening a pull request.