[SPARK-12832][MESOS] Fix dispatcher does not have a constraints config

[Astralidea]

before offer resouce, first meet meetsConstraints to filter some offer.

[dragos]

There's a typo there. I think the comment isn't necessary though, the name of the variable is clear enough.

[dragos]

ok to test

[SparkQA]

Test build #49460 has finished for PR 10768 at commit 6e8a028.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[Astralidea]

@dragos A question, should I also change MesosClusterDispatcher.scala code for cmd argument in?

[SparkQA]

Test build #49516 has finished for PR 10768 at commit ef698a9.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #49517 has finished for PR 10768 at commit f4d586f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dragos]

@Astralidea I don't understand your question, sorry.

[Astralidea]

@dragos
I start a mesos framework use
./start-mesos-dispatcher.sh --master mesos://zk://xxx:2181,yyy:2181,l-zzz/mesos
should modify code for give a argument for this mian method to start dispacher like
./start-mesos-dispatcher.sh --master mesos://zk://xxx:2181,yyy:2181,l-zzz/mesos --constraints colo:cn2
??
to do this, it also need to modify MesosClusterDispatcher.scala to parse the command line argument and passed the configiration to spark.mesos.constraints

[dragos]

Thanks for clarifying. I think this is a bit confusing. You want to respect the Mesos constraints that belong to each particular job that is submitted to the dispatcher. Those configuration options should come from a submission request, not from the Spark config options that are used to launch the dispatcher (probably found inside schedulerProperties)

Right now, the code would only launch drivers on one particular set of constraints, defined when the dispatcher is launched. I believe the better solution is to allow each Spark job to define its Mesos constraints independently, when submitting.

[BrickXu]

@dragos great idea!

[Astralidea]

@dragos Thank you for your answers. so the code may does not need to change other.
I will make a test when I change conf file spark-defaults.conf. Wheather it will take effect on deploy a driver.That is .when submit a job.will it read spark-defaults.conf every time.

[dragos]

This is the problem: This setting is read only once, when the dispatcher is started. What you want is to pick up the constraints set on the submitted job. Have a look at how other job-specific settings are treated in the code (there's a submitProperties variable, or something similar).

[dragos]

@Astralidea The code needs to change, but probably the change is minor. See my comment

[andrewor14]

FYI #10949 is another patch for the same issue.

[andrewor14]

Also @Astralidea please change the title to include [MESOS] instead of [CORE]

[Astralidea]

@dragos Sorry about busy these days, In your comment, The feature is great. Because if I change the configuration I did not redeploy mesos-dispacher, but In my System it is enough to use I didn't change config everyday, it is merely stable. and I did not figure out how to reload configure every time. If I have time to do this I will push another patch.

[dragos]

@Astralidea I think we should focus on getting #10949 in, which implements exactly this behavior.

[Astralidea]

@dragos ok, I hope this issue could fix in next version. only read once spark.mesos.constraints is enough.