Skip to content

[SPARK-20078] [MESOS] Mesos executor configurability for task name and labels #17404

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kalvinnchau wants to merge 6 commits into from
Closed

[SPARK-20078] [MESOS] Mesos executor configurability for task name and labels #17404

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
84656e3
add configurable mesos.task.name and mesos.task.labels
Mar 23, 2017
bc57ba0
add test to verify incorrect labels are not added
Mar 23, 2017
7bf49f8
fix typo
Mar 23, 2017
79492f5
make task name sc.appName - not configurable, remove mesos-labels to …
Mar 24, 2017
5f3816a
remove task.name and task.labels from README
Mar 24, 2017
dc5974c
use string literal to assert task name, and rename test for correctness
Mar 24, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion ...n/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -403,7 +403,8 @@ private[spark] class MesosCoarseGrainedSchedulerBackend(
.setTaskId(TaskID.newBuilder().setValue(taskId.toString).build())
.setSlaveId(offer.getSlaveId)
.setCommand(createCommand(offer, taskCPUs + extraCoresPerExecutor, taskId))
.setName("Task " + taskId)
.setName(s"${sc.appName} $taskId")

taskBuilder.addAllResources(resourcesToUse.asJava)
taskBuilder.setContainer(MesosSchedulerBackendUtil.containerInfo(sc.conf))

Expand Down
11 changes: 11 additions & 0 deletions ...la/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackendSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -464,6 +464,17 @@ class MesosCoarseGrainedSchedulerBackendSuite extends SparkFunSuite
assert(!uris.asScala.head.getCache)
}

test("mesos sets configurable task name") {
setBackend()

val offers = List(Resources(backend.executorMemory(sc), 1))
offerResources(offers)
val launchedTasks = verifyTaskLaunched(driver, "o1")

// Add " 0" to the taskName to match the executor number that is appended
assert(launchedTasks.head.getName == s"${sc.appName} 0")

@mgummelt mgummelt Mar 24, 2017

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use the actual literal string here, rather than a parameterized string, so we can directly assert the task name we want.

@kalvinnchau kalvinnchau Mar 24, 2017

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done!

}

test("mesos supports spark.mesos.network.name") {
setBackend(Map(
"spark.mesos.network.name" -> "test-network-name"
Expand Down