[SPARK-1680] [Docs] Explain environment variables for running on YARN in cluster mode

[weineran]

JIRA 1680 added a property called spark.yarn.appMasterEnv. This PR draws users' attention to this special case by adding an explanation in configuration.html#environment-variables

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

CC @sryza @vanzin @tgravescs for a check

[tgravescs]

you should file a separate jira for this. The original went in and is closed.

I'm fine with the text, it might be nice to add something about preferring the use of spark.executorEnv.[EnvironmentVariableName] over the spark-env.sh file.

[weineran]

It looks like the original JIRA status is "RESOLVED" but it was never "CLOSED" and so it is currently showing this pull request, which is probably a good thing. Since these documentation changes relate to the earlier code changes, I think it makes sense to put them on the same JIRA if possible. Otherwise, I'd be tempted to call this doc change "trivial" and skip creating a new JIRA.

Can you assist with the text relating to spark.executorEnv.[EnvironmentVariableName]? I don't have a deep understanding of when to use the property vs. when to use the .sh file.

[srowen]

@weineran agree with leaving it as is; my full logic: since that JIRA is soo old, and this change is still logically separable (i.e. one makes sense without the other), it's reasonable to make a new JIRA. However it's also trivial (i.e. diff ~= description of change) so in that sense, doesn't really matter.

[srowen]

Nit -- this needs back-ticks rather than <code> right? I think it's OK.

[weineran]

Ah well that could be. Does the <code> tag only get used in tables? Let me know if I should switch to back-ticks.

Just realized I should throw some back-ticks around cluster too.

[srowen]

Yeah as I recall <code> is required in tables since it occurs within the <table> tag and ticks aren't parsed there. But otherwise use back ticks.

[srowen]

Merged to master