[ZEPPELIN-4799] Use spark resource configuration

[Reamer]

What is this PR for?

With this PR, we use spark configuration values for K8s Pod resources. A memory limit is not set because of a potential OOM-Killer.

https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits

If you set a memory limit of 4GiB for that Container, the kubelet (and container runtime ) enforce the limit. The runtime prevents the container from using more than the configured resource limit. For example: when a process in the container tries to consume more than the allowed amount of memory, the system kernel terminates the process that attempted the allocation, with an out of memory (OOM) error.

@zjffdu Are using a YARN cluster to schedule your Interpreters? Maybe we should change the location of the calculation class.

What type of PR is it?

• Improvement

What is the Jira issue?

• https://issues.apache.org/jira/browse/ZEPPELIN-4799

How should this be tested?

• Travis-CI: https://travis-ci.org/github/Reamer/zeppelin/builds/683269227

Questions:

• Maybe a higher default memory overhead? Edit: No, because we doesn't need it in the past.
• Does the licenses files need update? No
• Is there breaking changes for older versions? No
• Does this needs documentation? No

[Leemoonsoo]

LGTM

[Leemoonsoo]

nit - How about adding some short comments here about why limits.memory is not configured? maybe a link to this pull request description? So later, other people can see this as a intentional when read the code.

[Reamer]

Done.

[zjffdu]

@zjffdu Are using a YARN cluster to schedule your Interpreters? Maybe we should change the location of the calculation class.

@Reamer I don't get it , what do you mean ?

[Reamer]

@zjffdu Are using a YARN cluster to schedule your Interpreters? Maybe we should change the location of the calculation class.

@Reamer I don't get it , what do you mean ?

The spark documentation mentions also YARN clusters for spark.driver.memoryOverhead.

This option is currently supported on YARN and Kubernetes.

I don't have a YARN cluster to execute spark jobs, and I don't even know how YARN really works, but I think that this calculation is also suitable for YARN. If that is not the case, forget my question.

[zjffdu]

@Reamer For yarn, this calculation is done by spark. e.g. if user specify spark.driver.memory as 1g, actually spark will ask for one container of (1g + 384m).
For k8s, I suspect maybe spark already done this kind of calculation.

[Reamer]

For k8s, I suspect maybe spark already done this kind of calculation.

For technical reasons, Spark can only limit resources in K8s, if you are working in cluster mode. If you run Spark in client mode, as Zeppelin does, you should set --driver-memory (already implemented).

[Reamer]

I assume that no higher memory overhead is needed, because in the past we did not need that for YARN either.

[Leemoonsoo]

Thank @Reamer for the improvement! I'm merging this to master and branch-0.9.