ZEPPELIN-1701: User impersonation not work with SPARK_HOME set.

[prabhjyotsingh]

What is this PR for?

User impersonation of front end user does not work with SPARK_HOME set.

What type of PR is it?

[Bug Fix | Hot Fix]

Todos

• - Fix log file permission issue

What is the Jira issue?

• ZEPPELIN-1701

How should this be tested?

set SPARK_HOME zeppelin-env.sh and then try to run

%spark
System.getProperty("user.name")

Screenshots (if appropriate)

Questions:

• Does the licenses files need update? N/A
• Is there breaking changes for older versions? N/A
• Does this needs documentation? Yes

[prabhjyotsingh]

CC: @zjffdu can you help review this.

[zjffdu]

It still doesn't work for me.
This is the error I get, it seems permission issue because it use another user to run the interpreter process.

DEBUG [2016-11-23 16:13:06,100] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) - java.io.FileNotFoundException: /Users/jzhang/github/zeppelin/logs/zeppelin-interpreter-spark-jzhang-jzhangMBPr.local.log (Permission denied)

[prabhjyotsingh]

can you try

chmod 777 /Users/jzhang/github/zeppelin/logs/zeppelin-interpreter-spark-jzhang-jzhangMBPr.local.log

[zjffdu]

I can do that, but it doesn't make sense to force user to do that.

[prabhjyotsingh]

I assumed this log file should have group permission to log into this file. What do you recommend ?

[zjffdu]

I found another crtitical issue is that all the enviroment variables defined in zeppelin-env.sh (like HADOOP_CONF_DIR) won't be passed to the interpreter process because you use ssh. This will cause interpreter process fail to start.

[prabhjyotsingh]

I think that should be fine;

...
    ZEPPELIN_INTP_CLASSPATH+=":${HADOOP_CONF_DIR}"
...
    CLASSPATH+=":${ZEPPELIN_INTP_CLASSPATH}"
...
   SPARK_SUBMIT_RUN=` echo ${SPARK_SUBMIT} --class ${ZEPPELIN_SERVER} --driver-class-path \"${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH}\" --driver-java-options \"${JAVA_INTP_OPTS}\" ${SPARK_SUBMIT_OPTIONS} ${SPARK_APP_JAR} ${PORT}`
...

https://github.com/apache/zeppelin/blob/master/bin/interpreter.sh#L171
https://github.com/apache/zeppelin/blob/master/bin/interpreter.sh#L186
https://github.com/apache/zeppelin/pull/1672/files#diff-8d3bd42a5de342ebad38e1d113d9f75cR189

[zjffdu]

Actually it doesn't work for spark interpreter if you try that. https://github.com/apache/zeppelin/blob/master/bin/interpreter.sh#L171 is for pig. And it is not safe if we add other enviroment variables that is not in classpath. Could we run zeppelin-env.sh after ssh ?

[prabhjyotsingh]

I see your point, let me check that.

[prabhjyotsingh]

@zjffdu Here is what I've done

• https://github.com/apache/zeppelin/pull/1672/files#diff-8d3bd42a5de342ebad38e1d113d9f75cR195
  source "${ZEPPELIN_CONF_DIR}'/zeppelin-env.sh;
  This should take care of zeppelin-env variables

• https://github.com/apache/zeppelin/pull/1672/files#diff-8d3bd42a5de342ebad38e1d113d9f75cR93
  Append user name in log file, this has an assumption that the parent directory has right permission for this guy to write (typically have group permission to write and have the impersonating user in that group).

Can you review this and/or suggest more changes.

[felixcheung]

as an aside, I'd update the JIRA/PR - it might not be obvious what "when SPARK_HOME is set" means. If external Spark is broken then I think it is a much much serious problem :)

[prabhjyotsingh]

This is ready for review, have checked this in few flavours of linux, and it was working as expected.

I'll add more documentation in a while.

[zjffdu]

@prabhjyotsingh It works now, but log file permission issue is still here which make the spark interpreter fail to run unless user manually set permission

[prabhjyotsingh]

@zjffdu did you try it again, as this is working for me, or if you can let me know your environment I can try on the same.

[r-kamath]

LGTM 👍

[Leemoonsoo]

#1704 changes the filename processenduser.md to userimpersonation.md. Could you resolve conflict?

[prabhjyotsingh]

@Leemoonsoo resolved conflict. Thank you for the quick fix.

[felixcheung]

this seem tricky so maybe I'm missing something, we are appending ' here?

[felixcheung]

and this follows immediately with source in L195, so it will be 'source - no space between ' and source?

[prabhjyotsingh]

Yes the ' starts here (L193) and ends at L206. I can add a space between ' and source but since it was working I didn't do it.

[prabhjyotsingh]

Merging this if no more discussion.

[zjffdu]

@prabhjyotsingh I can not run pyspark interpreter. Can you try that in your enviroment ?

[zjffdu]

Please ignore it，I can ran it successfully now.

[prabhjyotsingh]

Thank you for the verification.