On Kubernetes, set pod anti-affinity at the host level for pods of type 'ray'

[virtualluke]

What do these changes do?

Added 'type: ray' to the deployments' metadata and keyed off that type for pod anti-affinity on hosts. This will cause kubernetes to not schedule more than one pod of type 'ray' onto the same host.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12228/
Test FAILed.

[ericl]

How about using the soft form instead?preferredDuringSchedulingIgnoredDuringExecution

[virtualluke]

Seems like during execution and scheduling you would always want node anti-affinity, not just during execution.

[ericl]

It looks like the current pr already ignores during execution. I think it's better to use a soft instead of hard constraint as default is all.

…

On Mon, Feb 25, 2019, 10:38 AM Luke ***@***.***> wrote: Seems like during execution and scheduling you would always want node anti-affinity, not just during execution. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4131 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAA6SuxBjNEOYvIlXEzCh4oqNmR5sDJ6ks5vRC2SgaJpZM4bJl6s> .

[virtualluke]

I definitely prefer the harder (requiredDuringSchedulingIgnoreDuringExecution) form over the softer (preferredDuringSchedulingIgnoredDuringExecution). I am looking at cluster stability and want a hard rule about scheduling of ray pods. If others want it as a cluster scheduling suggestion (which is the softer form) than I am ok with that, just will use the hard version on our cluster.

[ericl]

Ok, I think either is fine if you think there's a strong benefit, so this LGTM.

[AmplabJenkins]

Can one of the admins verify this patch?

[robertnishihara]

I just tried this out and it works for me.

One issue I ran into when running this out of the box (pre-existing I think, and probably unrelated to this PR) is that kubectl create -f ray/kubernetes/submit.yaml didn't succeed because it required too much memory on the workers. We could increase the amount of memory requested by the pods, but then it will be harder to run out of the box (e.g., on minikube the pods don't get scheduled when more resources are requested.)

[robertnishihara]

Thanks @virtualluke!