[SPARK-22292][Mesos] Added spark.mem.max support for Mesos #19510

windkit · 2017-10-17T06:02:39Z

What changes were proposed in this pull request?

To limit the amount of resources a spark job accept from Mesos, currently we can only use spark.cores.max to limit in terms of cpu cores.
However, when we have big memory executors, it would consume all the resources.

This PR added spark.mem.max option for Mesos

How was this patch tested?

Added Unit Test

skonto · 2017-10-18T11:14:09Z

...scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackendSuite.scala

remove space

skonto · 2017-10-18T11:35:31Z

@windkit there is an open issue here: https://issues.apache.org/jira/browse/SPARK-22133
Could you please add documentation of the new property along with the old ones.

skonto · 2017-10-18T11:49:27Z

...main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala

Can we defend against minimum values and fail fast? For example default executor memory is 1.4MB. We could calculate the value returned by MesosSchedulerUtils.executorMemory. I don't think these values calculated in canLaunchTask ever change. Same applies for cpus.

@skonto
For cpus, I think we can compare with minCoresPerExecutor
For mem, calling the MesosSchedulerUtils.executorMemory to get the minimum requirement.

Then at here, we parse the option, check the minimum and if it is too small, throw exception?

Exception I think would be ok, the idea if something is never going to work let the user know, especially for the novice user. In general, the minimum would be a warning if we dont want an exception thrown.

Should I add the check with this PR or a separate one?

Do we need to have a check similar to

spark/resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala

Line 73 in 06df34d

// Maximum number of cores to acquire

but for memory, so that we know we'll "land" on the maximum?

windkit · 2017-10-20T07:02:09Z

@skonto sure I can add those in, can you point to me where the documentation source code is?

skonto · 2017-10-20T11:23:12Z

@windkit https://github.com/apache/spark/blob/e2fea8cd6058a807ff4841b496ea345ff0553044/docs/running-on-mesos.md

skonto · 2017-10-20T11:23:36Z

@susanxhuynh pls review.

susanxhuynh · 2017-10-26T21:28:11Z

@windkit Trying to understand the need for this config ... could you accomplish the same thing by setting spark.cores.max, spark.executor.cores, and spark.executor.memory? Could you give an example scenario where this is needed?

susanxhuynh · 2017-10-26T20:58:30Z

docs/running-on-mesos.md

Typo: 'park'

susanxhuynh · 2017-10-26T20:58:44Z

docs/running-on-mesos.md

Typo: 'park'

susanxhuynh · 2017-10-26T20:58:52Z

docs/running-on-mesos.md

Typo: 'park'

susanxhuynh · 2017-10-26T21:37:27Z

docs/running-on-mesos.md

Maybe add "across the cluster (not from each machine)". And, something about there is no maximum if this property is not set.

susanxhuynh · 2017-10-26T21:51:13Z

docs/running-on-mesos.md

My suggestion: "Duration for which unused resources are considered declined, when maximum number of cores spark.cores.max has been reached."
@ArtRand Is this the documentation you had in mind in https://issues.apache.org/jira/browse/SPARK-22133 ? Is this enough information for a non-Mesos expert to set this?

windkit · 2017-11-01T00:38:12Z

@susanxhuynh Thanks for reviewing.
I want to use both spark.mem.max and spark.cores.max to limit resource one task can use within the cluster.
Now I am setting up a common cluster for several users, they are allowed to configure spark.executor.cores and spark.executor.memory according to their need. I then need a limit for both cpu cores and memory.

## What changes were proposed in this pull request? Documentation about Mesos Reject Offer Configurations ## Related PR #19510 for `spark.mem.max` Author: Li, YanKit | Wilson | RIT <[email protected]> Closes #19555 from windkit/spark_22133.

skonto · 2018-05-19T18:20:32Z

@windkit Could you update the PR? @susanxhuynh I think this is fine to merge, are you ok?

AmplabJenkins · 2018-06-09T00:18:29Z

Can one of the admins verify this patch?

HyukjinKwon · 2018-07-16T02:18:13Z

gentle ping @windkit

Closes apache#17422 Closes apache#17619 Closes apache#18034 Closes apache#18229 Closes apache#18268 Closes apache#17973 Closes apache#18125 Closes apache#18918 Closes apache#19274 Closes apache#19456 Closes apache#19510 Closes apache#19420 Closes apache#20090 Closes apache#20177 Closes apache#20304 Closes apache#20319 Closes apache#20543 Closes apache#20437 Closes apache#21261 Closes apache#21726 Closes apache#14653 Closes apache#13143 Closes apache#17894 Closes apache#19758 Closes apache#12951 Closes apache#17092 Closes apache#21240 Closes apache#16910 Closes apache#12904 Closes apache#21731 Closes apache#21095 Added: Closes apache#19233 Closes apache#20100 Closes apache#21453 Closes apache#21455 Closes apache#18477 Added: Closes apache#21812 Closes apache#21787 Author: hyukjinkwon <[email protected]> Closes apache#21781 from HyukjinKwon/closing-prs.

skonto reviewed Oct 18, 2017

View reviewed changes

...scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackendSuite.scala Outdated

Copy link

Contributor

skonto Oct 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove space

skonto reviewed Oct 18, 2017

View reviewed changes

windkit mentioned this pull request Oct 23, 2017

[SPARK-22133][DOCS] Documentation for Mesos Reject Offer Configurations #19555

Closed

susanxhuynh reviewed Oct 26, 2017

View reviewed changes

windkit added 4 commits November 1, 2017 00:30

[SPARK-22133][DOCS] Documentation for Mesos Reject Offer Configurations

5f3b35b

[SPARK-22292][Mesos] Added spark.mem.max support for Mesos

9714694

Remove Space, to be squashed

dc4f441

Added documentation about spark.mem.max

e82f828

windkit force-pushed the mem_max branch from 960f3e9 to e82f828 Compare November 1, 2017 00:34

HyukjinKwon mentioned this pull request Jul 16, 2018

[INFRA] Close stale PR #21781

Closed

asfgit closed this in 1a4fda8 Jul 19, 2018

[SPARK-22292][Mesos] Added spark.mem.max support for Mesos #19510

[SPARK-22292][Mesos] Added spark.mem.max support for Mesos #19510

Uh oh!

Conversation

windkit commented Oct 17, 2017

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

skonto commented Oct 18, 2017

Uh oh!

skonto Oct 18, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

skonto Oct 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

windkit commented Oct 20, 2017

Uh oh!

skonto commented Oct 20, 2017

Uh oh!

skonto commented Oct 20, 2017

Uh oh!

susanxhuynh commented Oct 26, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

windkit commented Nov 1, 2017

Uh oh!

skonto commented May 19, 2018

Uh oh!

AmplabJenkins commented Jun 9, 2018

Uh oh!

HyukjinKwon commented Jul 16, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

skonto Oct 18, 2017 •

edited

Loading

skonto Oct 20, 2017 •

edited

Loading