Skip to content

[ZEPPELIN-1320] Run zeppelin interpreter process as web front end user #1322

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
prabhjyotsingh wants to merge 12 commits into from
Closed

[ZEPPELIN-1320] Run zeppelin interpreter process as web front end user #1322

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
6aafac4
user should have option to run these interpreters as different user.
prabhjyotsingh Aug 11, 2016
0ab2fe0
add userName as parameter in createOrGetIntepreter
prabhjyotsingh Aug 12, 2016
2758b7f
passing user and isUserImpersonate to RemoteInterpreterManagedProcess
prabhjyotsingh Aug 16, 2016
862f937
Merge remote-tracking branch 'origin/master' into ZEPPELIN-1320
prabhjyotsingh Aug 16, 2016
a2fea0c
fix indentation
prabhjyotsingh Aug 16, 2016
590032e
change nulls to "anonymous" in tests
prabhjyotsingh Aug 16, 2016
787a108
fix CI
prabhjyotsingh Aug 18, 2016
965c8a7
fix CI, null check before deleting paragraph
prabhjyotsingh Aug 18, 2016
8c324cd
CI fix: build stall for InterpreterRestApiTest
prabhjyotsingh Aug 18, 2016
e0a0ccf
Merge remote-tracking branch 'origin/master' into ZEPPELIN-1320
prabhjyotsingh Aug 18, 2016
55ccce3
revert shiro.ini and InterpreterGroup
prabhjyotsingh Aug 19, 2016
cc6c12e
use fromMessage.principal instead of anonymous
prabhjyotsingh Aug 19, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions bin/interpreter.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ function usage() {
echo "usage) $0 -p <port> -d <interpreter dir to load> -l <local interpreter repo dir to load>"
}

while getopts "hp:d:l:v" o; do
while getopts "hp:d:l:v:u" o; do
case ${o} in
h)
usage
Expand All @@ -42,6 +42,9 @@ while getopts "hp:d:l:v" o; do
. "${bin}/common.sh"
getZeppelinVersion
;;
u)
ZEPPELIN_SSH_COMMAND="ssh ${OPTARG}@localhost "
;;
esac

@zjffdu zjffdu Aug 19, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This requires the login user must exist in the os account and be able to ssh to localhost. I am not sure whether this is a good way, but just feel the approach is a little strange compared to the impersonation implementation in hadoop.

@prabhjyotsingh prabhjyotsingh Aug 19, 2016

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zjffdu yes, I agree, its not as implementation in hadoop, would you recommend something else ?

done

Expand Down Expand Up @@ -155,10 +158,11 @@ addJarInDirForIntp "${LOCAL_INTERPRETER_REPO}"

CLASSPATH+=":${ZEPPELIN_INTP_CLASSPATH}"


if [[ -n "${SPARK_SUBMIT}" ]]; then
${SPARK_SUBMIT} --class ${ZEPPELIN_SERVER} --driver-class-path "${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH}" --driver-java-options "${JAVA_INTP_OPTS}" ${SPARK_SUBMIT_OPTIONS} ${SPARK_APP_JAR} ${PORT} &
${ZEPPELIN_SSH_COMMAND} ${SPARK_SUBMIT} --class ${ZEPPELIN_SERVER} --driver-class-path "${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH}" --driver-java-options "${JAVA_INTP_OPTS}" ${SPARK_SUBMIT_OPTIONS} ${SPARK_APP_JAR} ${PORT} &
else
${ZEPPELIN_RUNNER} ${JAVA_INTP_OPTS} ${ZEPPELIN_INTP_MEM} -cp ${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH} ${ZEPPELIN_SERVER} ${PORT} &
${ZEPPELIN_SSH_COMMAND} ${ZEPPELIN_RUNNER} ${JAVA_INTP_OPTS} ${ZEPPELIN_INTP_MEM} -cp ${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH} ${ZEPPELIN_SERVER} ${PORT} &
fi

pid=$!
Expand Down
12 changes: 6 additions & 6 deletions conf/shiro.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,10 @@
[users]
# List of users with their password allowed to access Zeppelin.
# To use a different strategy (LDAP / Database / ...) check the shiro doc at http://shiro.apache.org/configuration.html#Configuration-INISections
admin = password1, admin
user1 = password2, role1, role2
user2 = password3, role3
user3 = password4, role2
admin = admin, admin
user1 = user1, role1, role2
user2 = user2, role3
user3 = user3, role2

# Sample LDAP configuration, for user Authentication, currently tested for single Realm
[main]
Expand Down Expand Up @@ -75,5 +75,5 @@ admin = *
#/api/interpreter/** = authc, roles[admin]
#/api/configurations/** = authc, roles[admin]
#/api/credential/** = authc, roles[admin]
/** = anon
#/** = authc
#/** = anon
/** = authc
3 changes: 3 additions & 0 deletions conf/zeppelin-env.sh.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@
# export ZEPPELIN_INTERPRETER_LOCALREPO # Local repository for interpreter's additional dependency loading
# export ZEPPELIN_NOTEBOOK_STORAGE # Refers to pluggable notebook storage class, can have two classes simultaneously with a sync between them (e.g. local and remote).

# export ZEPPELIN_INTERPRETER_RUNAS_USER= # set this to true if user wants to run interpreter as different user, and if shiro authentication is enabled then it will run as end web user.
# export ZEPPELIN_INTERPRETER_USER= # user to be run as if shiro authentication is not enbaled and you want to run it as different user than the server.

#### Spark interpreter configuration ####

## Use provided spark installation ##
Expand Down
4 changes: 4 additions & 0 deletions ...src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterManagedProcess.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,10 @@ public void start() {
cmdLine.addArgument("-p", false);
cmdLine.addArgument(Integer.toString(port), false);
cmdLine.addArgument("-l", false);


cmdLine.addArgument("-u", false);

cmdLine.addArgument(localRepoDir, false);

executor = new DefaultExecutor();
Expand Down
24 changes: 17 additions & 7 deletions zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,19 +17,19 @@

package org.apache.zeppelin.conf;

import java.net.URL;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

import org.apache.commons.configuration.ConfigurationException;
import org.apache.commons.configuration.XMLConfiguration;
import org.apache.commons.configuration.tree.ConfigurationNode;
import org.apache.zeppelin.notebook.repo.VFSNotebookRepo;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.net.URL;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

/**
* Zeppelin configuration.
*
Expand Down Expand Up @@ -431,6 +431,14 @@ public boolean getUseJdbcAlias() {
return getBoolean(ConfVars.ZEPPELIN_USE_JDBC_ALIAS);
}

public boolean getInterpreterRunasUser() {
return getBoolean(ConfVars.ZEPPELIN_INTERPRETER_RUNAS_USER);
}

public String getZeppelinInterpreterUser() {
return getString(ConfVars.ZEPPELIN_INTERPRETER_USER);
}

public Map<String, String> dumpConfigurations(ZeppelinConfiguration conf,
ConfigurationKeyPredicate predicate) {
Map<String, String> configurations = new HashMap<>();
Expand Down Expand Up @@ -557,7 +565,9 @@ public static enum ConfVars {
ZEPPELIN_ANONYMOUS_ALLOWED("zeppelin.anonymous.allowed", true),
ZEPPELIN_CREDENTIALS_PERSIST("zeppelin.credentials.persist", true),
ZEPPELIN_WEBSOCKET_MAX_TEXT_MESSAGE_SIZE("zeppelin.websocket.max.text.message.size", "1024000"),
ZEPPELIN_USE_JDBC_ALIAS("zeppelin.use.jdbc.alias", true);
ZEPPELIN_USE_JDBC_ALIAS("zeppelin.use.jdbc.alias", true),
ZEPPELIN_INTERPRETER_RUNAS_USER("zeppelin.interpreter.runas.user", null),
ZEPPELIN_INTERPRETER_USER("zeppelin.interpreter.user", null);


private String varName;
Expand Down
70 changes: 26 additions & 44 deletions zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,50 +17,13 @@

package org.apache.zeppelin.interpreter;

import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Type;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLClassLoader;
import java.nio.file.DirectoryStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.Comparator;
import java.util.HashMap;
import java.util.Iterator;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.Set;

import com.google.common.base.Preconditions;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import com.google.gson.reflect.TypeToken;
import org.apache.commons.io.FileUtils;
import org.apache.commons.lang.ArrayUtils;
import org.apache.commons.lang.NullArgumentException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.sonatype.aether.RepositoryException;
import org.sonatype.aether.repository.Authentication;
import org.sonatype.aether.repository.RemoteRepository;

import org.apache.zeppelin.conf.ZeppelinConfiguration;
import org.apache.zeppelin.conf.ZeppelinConfiguration.ConfVars;
import org.apache.zeppelin.dep.Dependency;
Expand All @@ -76,6 +39,24 @@
import org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcessListener;
import org.apache.zeppelin.scheduler.Job;
import org.apache.zeppelin.scheduler.Job.Status;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.sonatype.aether.RepositoryException;
import org.sonatype.aether.repository.Authentication;
import org.sonatype.aether.repository.RemoteRepository;

import java.io.*;
import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Type;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLClassLoader;
import java.nio.file.DirectoryStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.*;

/**
* Manage interpreters.
Expand Down Expand Up @@ -1094,14 +1075,15 @@ private String getInterpreterClassFromInterpreterSetting(InterpreterSetting sett
return null;
}

private Interpreter getInterpreter(String noteId, InterpreterSetting setting, String name) {
private Interpreter getInterpreter(String noteId, InterpreterSetting setting, String name,
String userName) {

@bzz bzz Aug 12, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably a nitpick, but horizontal alignment is controversial idea and generally is discouraged by the styleguide as it creates a "blast radius" of re-formatting in case of future changes i.e renaming a function.

Preconditions.checkNotNull(noteId, "noteId should be not null");
Preconditions.checkNotNull(setting, "setting should be not null");
Preconditions.checkNotNull(name, "name should be not null");

String className;
if (null != (className = getInterpreterClassFromInterpreterSetting(setting, name))) {
List<Interpreter> interpreterGroup = createOrGetInterpreterList(noteId, setting);
List<Interpreter> interpreterGroup = createOrGetInterpreterList(noteId + userName, setting);
for (Interpreter interpreter : interpreterGroup) {
if (className.equals(interpreter.getClassName())) {
return interpreter;
Expand All @@ -1111,7 +1093,7 @@ private Interpreter getInterpreter(String noteId, InterpreterSetting setting, St
return null;
}

public Interpreter getInterpreter(String noteId, String replName) {
public Interpreter getInterpreter(String noteId, String replName, String userName) {
List<InterpreterSetting> settings = getInterpreterSettings(noteId);
InterpreterSetting setting;
Interpreter interpreter;
Expand All @@ -1124,7 +1106,7 @@ public Interpreter getInterpreter(String noteId, String replName) {
// get default settings (first available)
// TODO(jl): Fix it in case of returning null
InterpreterSetting defaultSettings = getDefaultInterpreterSetting(settings);
return createOrGetInterpreterList(noteId, defaultSettings).get(0);
return createOrGetInterpreterList(noteId + userName, defaultSettings).get(0);
}

String[] replNameSplit = replName.split("\\.");
Expand All @@ -1137,7 +1119,7 @@ public Interpreter getInterpreter(String noteId, String replName) {
setting = getInterpreterSettingByGroup(settings, group);

if (null != setting) {
interpreter = getInterpreter(noteId, setting, name);
interpreter = getInterpreter(noteId, setting, name, userName);

if (null != interpreter) {
return interpreter;
Expand All @@ -1152,7 +1134,7 @@ public Interpreter getInterpreter(String noteId, String replName) {
// TODO(jl): Handle with noteId to support defaultInterpreter per note.
setting = getDefaultInterpreterSetting(settings);

interpreter = getInterpreter(noteId, setting, replName);
interpreter = getInterpreter(noteId, setting, replName, userName);

if (null != interpreter) {
return interpreter;
Expand All @@ -1163,7 +1145,7 @@ public Interpreter getInterpreter(String noteId, String replName) {
setting = getInterpreterSettingByGroup(settings, replName);

if (null != setting) {
List<Interpreter> interpreters = createOrGetInterpreterList(noteId, setting);
List<Interpreter> interpreters = createOrGetInterpreterList(noteId + userName, setting);
if (null != interpreters) {
return interpreters.get(0);
}
Expand Down
8 changes: 5 additions & 3 deletions zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -447,7 +447,7 @@ public void runAll() {
p.setAuthenticationInfo(authenticationInfo);

p.setListener(jobListenerFactory.getParagraphJobListener(this));
Interpreter intp = factory.getInterpreter(getId(), p.getRequiredReplName());
Interpreter intp = factory.getInterpreter(getId(), p.getRequiredReplName(), null);

intp.getScheduler().submit(p);
}
Expand All @@ -463,11 +463,13 @@ public void run(String paragraphId) {
Paragraph p = getParagraph(paragraphId);
p.setListener(jobListenerFactory.getParagraphJobListener(this));
String requiredReplName = p.getRequiredReplName();
Interpreter intp = factory.getInterpreter(getId(), requiredReplName);
Interpreter intp = factory.getInterpreter(getId(), requiredReplName,
getParagraph(paragraphId).getAuthenticationInfo().getUser());

if (intp == null) {
// TODO(jongyoul): Make "%jdbc" configurable from JdbcInterpreter
if (conf.getUseJdbcAlias() && null != (intp = factory.getInterpreter(getId(), "jdbc"))) {
if (conf.getUseJdbcAlias() && null != (intp = factory.getInterpreter(getId(), "jdbc",
getParagraph(paragraphId).getAuthenticationInfo().getUser()))) {
String pText = p.getText().replaceFirst(requiredReplName, "jdbc(" + requiredReplName + ")");
logger.debug("New paragraph: {}", pText);
p.setEffectiveText(pText);
Expand Down
3 changes: 2 additions & 1 deletion zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,8 @@ public static String getScriptBody(String text) {
}

public Interpreter getRepl(String name) {
return factory.getInterpreter(note.getId(), name);
return factory.getInterpreter(note.getId(), name,
getAuthenticationInfo().getUser());
}

public Interpreter getCurrentRepl() {
Expand Down
2 changes: 1 addition & 1 deletion zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/InterpreterFactoryTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,6 @@ public void testInterpreterAliases() throws IOException, RepositoryException {
add(setting2.getId());
}});

assertEquals("className1", factory.getInterpreter("note", "test-group1").getClassName());
assertEquals("className1", factory.getInterpreter("note", "test-group1", null).getClassName());
}
}
34 changes: 17 additions & 17 deletions zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/NoteInterpreterLoaderTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,21 +74,21 @@ public void testGetInterpreter() throws IOException {
factory.setInterpreters("note", factory.getDefaultInterpreterSettingList());

// when there're no interpreter selection directive
assertEquals("org.apache.zeppelin.interpreter.mock.MockInterpreter1", factory.getInterpreter("note", null).getClassName());
assertEquals("org.apache.zeppelin.interpreter.mock.MockInterpreter1", factory.getInterpreter("note", "").getClassName());
assertEquals("org.apache.zeppelin.interpreter.mock.MockInterpreter1", factory.getInterpreter("note", " ").getClassName());
assertEquals("org.apache.zeppelin.interpreter.mock.MockInterpreter1", factory.getInterpreter("note", null, null).getClassName());
assertEquals("org.apache.zeppelin.interpreter.mock.MockInterpreter1", factory.getInterpreter("note", "", null).getClassName());
assertEquals("org.apache.zeppelin.interpreter.mock.MockInterpreter1", factory.getInterpreter("note", " ", null).getClassName());

// when group name is omitted
assertEquals("org.apache.zeppelin.interpreter.mock.MockInterpreter11", factory.getInterpreter("note", "mock11").getClassName());
assertEquals("org.apache.zeppelin.interpreter.mock.MockInterpreter11", factory.getInterpreter("note", "mock11", null).getClassName());

// when 'name' is ommitted
assertEquals("org.apache.zeppelin.interpreter.mock.MockInterpreter1", factory.getInterpreter("note", "group1").getClassName());
assertEquals("org.apache.zeppelin.interpreter.mock.MockInterpreter2", factory.getInterpreter("note", "group2").getClassName());
assertEquals("org.apache.zeppelin.interpreter.mock.MockInterpreter1", factory.getInterpreter("note", "group1", null).getClassName());
assertEquals("org.apache.zeppelin.interpreter.mock.MockInterpreter2", factory.getInterpreter("note", "group2", null).getClassName());

// when nothing is ommitted
assertEquals("org.apache.zeppelin.interpreter.mock.MockInterpreter1", factory.getInterpreter("note", "group1.mock1").getClassName());
assertEquals("org.apache.zeppelin.interpreter.mock.MockInterpreter11", factory.getInterpreter("note", "group1.mock11").getClassName());
assertEquals("org.apache.zeppelin.interpreter.mock.MockInterpreter2", factory.getInterpreter("note", "group2.mock2").getClassName());
assertEquals("org.apache.zeppelin.interpreter.mock.MockInterpreter1", factory.getInterpreter("note", "group1.mock1", null).getClassName());
assertEquals("org.apache.zeppelin.interpreter.mock.MockInterpreter11", factory.getInterpreter("note", "group1.mock11", null).getClassName());
assertEquals("org.apache.zeppelin.interpreter.mock.MockInterpreter2", factory.getInterpreter("note", "group2.mock2", null).getClassName());

factory.closeNote("note");
}
Expand All @@ -105,12 +105,12 @@ public void testNoteSession() throws IOException {
assertNull(factory.getInterpreterSettings("noteA").get(0).getInterpreterGroup("shared_process").get("noteA"));
assertNull(factory.getInterpreterSettings("noteB").get(0).getInterpreterGroup("shared_process").get("noteB"));

factory.getInterpreter("noteA", null).open();
factory.getInterpreter("noteB", null).open();
factory.getInterpreter("noteA", null, null).open();
factory.getInterpreter("noteB", null, null).open();

assertTrue(
factory.getInterpreter("noteA", null).getInterpreterGroup().getId().equals(
factory.getInterpreter("noteB", null).getInterpreterGroup().getId()));
factory.getInterpreter("noteA", null, null).getInterpreterGroup().getId().equals(
factory.getInterpreter("noteB", null, null).getInterpreterGroup().getId()));

// interpreters are created after accessing it
assertNotNull(factory.getInterpreterSettings("noteA").get(0).getInterpreterGroup("shared_process").get("noteA"));
Expand Down Expand Up @@ -138,13 +138,13 @@ public void testNotePerInterpreterProcess() throws IOException {
assertNull(factory.getInterpreterSettings("noteA").get(0).getInterpreterGroup("noteA").get("noteA"));
assertNull(factory.getInterpreterSettings("noteB").get(0).getInterpreterGroup("noteB").get("noteB"));

factory.getInterpreter("noteA", null).open();
factory.getInterpreter("noteB", null).open();
factory.getInterpreter("noteA", null, null).open();
factory.getInterpreter("noteB", null, null).open();

// per note interpreter process
assertFalse(
factory.getInterpreter("noteA", null).getInterpreterGroup().getId().equals(
factory.getInterpreter("noteB", null).getInterpreterGroup().getId()));
factory.getInterpreter("noteA", null, null).getInterpreterGroup().getId().equals(
factory.getInterpreter("noteB", null, null).getInterpreterGroup().getId()));

// interpreters are created after accessing it
assertNotNull(factory.getInterpreterSettings("noteA").get(0).getInterpreterGroup("noteA").get("noteA"));
Expand Down
Loading