Skip to content

[SPARK-20078] [MESOS] Mesos executor configurability for task name and labels #17404

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kalvinnchau wants to merge 6 commits into from
Closed

[SPARK-20078] [MESOS] Mesos executor configurability for task name and labels #17404

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
84656e3
add configurable mesos.task.name and mesos.task.labels
Mar 23, 2017
bc57ba0
add test to verify incorrect labels are not added
Mar 23, 2017
7bf49f8
fix typo
Mar 23, 2017
79492f5
make task name sc.appName - not configurable, remove mesos-labels to …
Mar 24, 2017
5f3816a
remove task.name and task.labels from README
Mar 24, 2017
dc5974c
use string literal to assert task name, and rename test for correctness
Mar 24, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion ...n/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -403,7 +403,8 @@ private[spark] class MesosCoarseGrainedSchedulerBackend(
.setTaskId(TaskID.newBuilder().setValue(taskId.toString).build())
.setSlaveId(offer.getSlaveId)
.setCommand(createCommand(offer, taskCPUs + extraCoresPerExecutor, taskId))
.setName("Task " + taskId)
.setName(s"${sc.appName} $taskId")

taskBuilder.addAllResources(resourcesToUse.asJava)
taskBuilder.setContainer(MesosSchedulerBackendUtil.containerInfo(sc.conf))

Expand Down
11 changes: 11 additions & 0 deletions ...la/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackendSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -464,6 +464,17 @@ class MesosCoarseGrainedSchedulerBackendSuite extends SparkFunSuite
assert(!uris.asScala.head.getCache)
}

test("mesos sets task name to spark.app.name") {
setBackend()

val offers = List(Resources(backend.executorMemory(sc), 1))
offerResources(offers)
val launchedTasks = verifyTaskLaunched(driver, "o1")

// Add " 0" to the taskName to match the executor number that is appended
assert(launchedTasks.head.getName == "test-mesos-dynamic-alloc 0")
Copy link
Copy Markdown

@mgummelt mgummelt Mar 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, as a side note, we need to update this task name, since it's not general to all tests here. But we can do that later.

}

test("mesos supports spark.mesos.network.name") {
setBackend(Map(
"spark.mesos.network.name" -> "test-network-name"
Expand Down