[SPARK-22256][MESOS] - Introduce spark.mesos.driver.memoryOverhead #21006
Conversation
|
Jenkins, ok to test |
|
Test build #89091 has finished for PR 21006 at commit
|
|
ok to test |
|
Test build #92795 has finished for PR 21006 at commit
|
|
jenkins ok to test |
| .filter(_.getName.equals("mem")) | ||
| .map(_.getScalar.getValue) | ||
| .head | ||
| assert(1384.0 === taskMem) |
There was a problem hiding this comment.
Can we also test the 10% case as well?
|
Besides the test comment, everything else LGTM |
|
Test build #93914 has finished for PR 21006 at commit
|
|
Can one of the admins verify this patch? |
|
We just had to build a custom version of 2.4.4 to pull in this change, are there any plans to merge it into an upcoming release? |
|
I created this pull request pmackles#1 to update the branch for this pull request with changes on branch https://github.com/dmcwhorter/spark/tree/dmcwhorter-SPARK-22256. The updates do the following:
My hope is this update will make this pull request ready to merge and include in the next spark release. |
|
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
### What changes were proposed in this pull request? This is a simple change to support allocating a specified amount of overhead memory for the driver's mesos container. This is already supported for executors. ### Why are the changes needed? This is needed to keep the driver process from exceeding memory limits and being killed off when running on mesos. ### Does this PR introduce _any_ user-facing change? Yes, it adds a `spark.mesos.driver.memoryOverhead` configuration option. Documentation changes for this option are included in the PR. ### How was this patch tested? Test cases covering allocation of driver memory overhead are included in the changes. ### Other notes This is a second attempt to get this change reviewed, accepted and merged. The original pull request was closed as stale back in January: #21006. For this pull request, I took the original change by pmackles, rebased it onto the current master branch, and added a test case that was requested in the original code review. I'm happy to make any further edits or do anything needed so that this can be included in a future spark release. I keep having to build custom spark distributions so that we can use spark within our mesos clusters. Closes #30739 from dmcwhorter/dmcwhorter-SPARK-22256. Lead-authored-by: David McWhorter <[email protected]> Co-authored-by: Paul Mackles <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
8 participants
When running spark driver in a container such as when using the Mesos dispatcher service, we need to apply the same rules as for executors in order to avoid the JVM going over the allotted limit and then killed.
Tested manually on spark 2.3 branch