[SPARK-7877][MESOS] Allow configuration of framework timeout #14936

philipphoffmann · 2016-09-02T11:59:26Z

What changes were proposed in this pull request?

This commit introduces a setting for configuring the Mesos framework failover timeout (spark.mesos.failoverTimeout). The default timeout is 10 seconds.

Before, the timeout was set to 0 (which causes all tasks to be killed by mesos) or Integer.MAX which kindof leaves them hanging forever, depending on how the driver was launched.

How was this patch tested?

unit test

SparkQA · 2016-09-02T12:22:48Z

Test build #64852 has finished for PR 14936 at commit c15f207.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

philipphoffmann · 2016-09-24T19:11:03Z

rebase to master

SparkQA · 2016-09-24T19:31:50Z

Test build #65873 has finished for PR 14936 at commit 95cce37.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tnachen · 2016-10-07T17:13:02Z

mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala

Fix the parameters indent

tnachen · 2016-10-07T17:13:34Z

mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala

This new flag needs to get added to the documentation as well

philipphoffmann · 2016-10-10T08:28:06Z

fixed indent
added documentation

any objections on the default timeout?

SparkQA · 2016-10-10T08:38:42Z

Test build #66638 has finished for PR 14936 at commit 6441f87.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-10-10T08:48:22Z

Test build #66639 has finished for PR 14936 at commit ee10ad3.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

tnachen · 2016-10-10T21:55:39Z

Hmm it is currently Integer.MAX because we assume the cluster scheduler to be long living, and without setting it be a long value Mesos will automatically terminate that framework when it's disconnected. I think currently all Spark jobs don't have it specified so when it disconnects it's simply removed.

I think we should keep the same semantics and don't impose a default value for everything, we have it left to be 0 in coarse grain scheduler, and default to Int.MAX in cluster scheduler. But user can always override it no matter what.

philipphoffmann · 2016-10-15T11:33:47Z

Alright, I changed the implementation to keep the existing defaults.

SparkQA · 2016-10-15T11:57:15Z

Test build #67010 has finished for PR 14936 at commit dec052c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tnachen · 2016-12-06T23:11:35Z

@philipphoffmann Sorry for the long delay, one last ask. Can you add a simple unit test to verify it works?

philipphoffmann · 2016-12-25T13:57:16Z

will do ...

HyukjinKwon · 2017-02-09T14:06:46Z

(@philipphoffmann gentle ping)

philipphoffmann · 2017-02-09T14:08:00Z

@HyukjinKwon sry, should find some time tomorrow ...

philipphoffmann · 2017-02-10T11:20:27Z

rebase to master; added tests for all three schedulers

SparkQA · 2017-02-10T11:24:23Z

Test build #72702 has finished for PR 14936 at commit ca7fdad.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-02-10T11:30:18Z

Test build #72704 has finished for PR 14936 at commit 7383817.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-02-10T12:00:47Z

Test build #72707 has finished for PR 14936 at commit 2602f7f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

This commit introduces a setting for configuring the Mesos framework failover timeout (`spark.mesos.failoverTimeout`).

philipphoffmann · 2017-03-14T14:52:55Z

rebase to master

SparkQA · 2017-03-14T15:09:58Z

Test build #74529 has finished for PR 14936 at commit bd9ace4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-02-21T04:53:38Z

Test build #87573 has finished for PR 14936 at commit bd9ace4.

This patch fails PySpark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

github-actions · 2020-01-18T00:08:16Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

### What changes were proposed in this pull request? This pr aims to upgrade Arrow from 14.0.2 to 15.0.0, this version fixes the compatibility issue with Netty 4.1.104.Final(GH-39265). Additionally, since the `arrow-vector` module uses `eclipse-collections` to replace `netty-common` as a compile-level dependency, Apache Spark has added a dependency on `eclipse-collections` after upgrading to use Arrow 15.0.0. ### Why are the changes needed? The new version brings the following major changes: Bug Fixes GH-34610 - [Java] Fix valueCount and field name when loading/transferring NullVector GH-38242 - [Java] Fix incorrect internal struct accounting for DenseUnionVector#getBufferSizeFor GH-38254 - [Java] Add reusable buffer getters to char/binary vectors GH-38366 - [Java] Fix Murmur hash on buffers less than 4 bytes GH-38387 - [Java] Fix JDK8 compilation issue with TestAllTypes GH-38614 - [Java] Add VarBinary and VarCharWriter helper methods to more writers GH-38725 - [Java] decompression in Lz4CompressionCodec.java does not set writer index New Features and Improvements GH-38511 - [Java] Add getTransferPair(Field, BufferAllocator, CallBack) for StructVector and MapVector GH-14936 - [Java] Remove netty dependency from arrow-vector GH-38990 - [Java] Upgrade to flatc version 23.5.26 GH-39265 - [Java] Make it run well with the netty newest version 4.1.104 The full release notes as follows: - https://arrow.apache.org/release/15.0.0.html ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GitHub Actions ### Was this patch authored or co-authored using generative AI tooling? No Closes #44797 from LuciferYang/SPARK-46718. Authored-by: yangjie01 <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

philipphoffmann force-pushed the mesos-failover-timeout branch from c15f207 to 95cce37 Compare September 24, 2016 19:10

tnachen reviewed Oct 7, 2016

View reviewed changes

mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala Outdated

Copy link

Contributor

tnachen Oct 7, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix the parameters indent

tnachen reviewed Oct 7, 2016

View reviewed changes

philipphoffmann force-pushed the mesos-failover-timeout branch from 95cce37 to 6441f87 Compare October 10, 2016 08:27

philipphoffmann force-pushed the mesos-failover-timeout branch from 6441f87 to ee10ad3 Compare October 10, 2016 08:41

philipphoffmann force-pushed the mesos-failover-timeout branch from ee10ad3 to dec052c Compare October 15, 2016 11:33

philipphoffmann force-pushed the mesos-failover-timeout branch from dec052c to ca7fdad Compare February 10, 2017 11:19

philipphoffmann force-pushed the mesos-failover-timeout branch from ca7fdad to 7383817 Compare February 10, 2017 11:23

philipphoffmann force-pushed the mesos-failover-timeout branch from 7383817 to 2602f7f Compare February 10, 2017 11:36

[SPARK-7877][MESOS] Allow configuration of framework timeout

bd9ace4

This commit introduces a setting for configuring the Mesos framework failover timeout (`spark.mesos.failoverTimeout`).

philipphoffmann force-pushed the mesos-failover-timeout branch from 2602f7f to bd9ace4 Compare March 14, 2017 14:52

dongjoon-hyun added the MESOS label Jun 14, 2019

github-actions bot added the Stale label Jan 18, 2020

github-actions bot closed this Jan 19, 2020

LuciferYang mentioned this pull request Jan 23, 2024

[SPARK-46718][BUILD] Upgrade Arrow to 15.0.0 #44797

Closed

[SPARK-7877][MESOS] Allow configuration of framework timeout #14936

[SPARK-7877][MESOS] Allow configuration of framework timeout #14936

Uh oh!

Conversation

philipphoffmann commented Sep 2, 2016

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Sep 2, 2016

Uh oh!

philipphoffmann commented Sep 24, 2016

Uh oh!

SparkQA commented Sep 24, 2016

Uh oh!

tnachen Oct 7, 2016

Choose a reason for hiding this comment

Uh oh!

tnachen Oct 7, 2016

Choose a reason for hiding this comment

Uh oh!

philipphoffmann commented Oct 10, 2016

Uh oh!

SparkQA commented Oct 10, 2016

Uh oh!

SparkQA commented Oct 10, 2016

Uh oh!

tnachen commented Oct 10, 2016

Uh oh!

philipphoffmann commented Oct 15, 2016

Uh oh!

SparkQA commented Oct 15, 2016

Uh oh!

tnachen commented Dec 6, 2016

Uh oh!

philipphoffmann commented Dec 25, 2016

Uh oh!

HyukjinKwon commented Feb 9, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

philipphoffmann commented Feb 9, 2017

Uh oh!

philipphoffmann commented Feb 10, 2017

Uh oh!

SparkQA commented Feb 10, 2017

Uh oh!

SparkQA commented Feb 10, 2017

Uh oh!

SparkQA commented Feb 10, 2017

Uh oh!

philipphoffmann commented Mar 14, 2017

Uh oh!

SparkQA commented Mar 14, 2017

Uh oh!

SparkQA commented Feb 21, 2018

Uh oh!

github-actions bot commented Jan 18, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

HyukjinKwon commented Feb 9, 2017 •

edited

Loading