[ZEPPELIN-1486] Allow configuring whether shown values will be trimmed #1458

mispecto · 2016-09-26T05:30:34Z

What is this PR for?

This patch improves Livy interpreter behavior: it allows configuring whether shown values will be trimmed or not.

What type of PR is it?

Improvement

Todos

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-1486

How should this be tested?

Set property zeppelin.livy.spark.sql.truncateResult to true
Run SQL query using Livy SQL interpreter
See whether long values are trimmed or not

Screenshots (if appropriate)

Questions:

Does the licenses files need update? - No
Is there breaking changes for older versions? - No
Does this needs documentation? - Yes. Documenting zeppelin.livy.spark.sql.truncateResult option.

AhyoungRyu · 2016-09-27T06:17:32Z

Hi @spektom. Thanks for your contribution :)

You said

Yes. Documenting zeppelin.livy.spark.sql.truncateResult option.

But it seems there is no docs file change. The below code needs to be added here as well.

<tr>
  <td>zeppelin.livy.spark.sql.truncateResult</td>
  <td>false</td>
  <td>Truncate long strings in SQL display</td>
</tr>

mispecto · 2016-09-27T08:16:58Z

@AhyoungRyu Added, thanks for pointing it out to me.

r-kamath · 2016-10-04T10:07:36Z

@spektom thanks for the improvement. Can you also updateconf/zeppelin-env.sh.template and conf/zeppelin-site.xml.template with env name/property name and description.

mispecto · 2016-10-04T10:34:34Z

@r-kamath I've added documentation to the zeppelin-env.sh.template file, but I couldn't find any other option related to Livy interpreter in zeppelin-site.xml.template, so I've left this file untouched.

jongyoul · 2016-10-05T06:15:37Z

conf/zeppelin-env.cmd.template

 REM set ZEPPELIN_SPARK_CONCURRENTSQL   REM Execute multiple SQL concurrently if set true. false by default.
 REM set ZEPPELIN_SPARK_IMPORTIMPLICIT  REM Import implicits, UDF collection, and sql if set true. true by default.
 REM set ZEPPELIN_SPARK_MAXRESULT       REM Max number of Spark SQL result to display. 1000 by default.
+REM set ZEPPELIN_SPARK_TRUNCATERESULT  REM Truncate long strings in SQL display. false by default.


It looks like Livy only feature for now. How about removing env here and .sh?

Agree to remove it from env.sh and site.xml, env.sh and site.xml is for zeppelin server rather than interpreter.

jongyoul · 2016-10-05T06:17:34Z

@spektom I agree with you not to add zeppelin-env.* because it depends on Live only.

zjffdu · 2016-10-08T10:03:22Z

livy/src/main/resources/interpreter-setting.json

+      "zeppelin.livy.spark.sql.truncateResult": {
+        "envName": "ZEPPELIN_LIVY_TRUNCATERESULT",
+        "propertyName": "zeppelin.livy.spark.sql.truncateResult",
+        "defaultValue": "false",


Will it make more sense to set true by default ? As it is truncated in spark by default.

zjffdu · 2016-10-08T10:06:27Z

livy/src/main/java/org/apache/zeppelin/livy/LivySparkSQLInterpreter.java

-              property.get("zeppelin.livy.spark.sql.maxResult") + ")",
+              property.get("zeppelin.livy.spark.sql.maxResult") + "," +
+              property.get("zeppelin.livy.spark.sql.truncateResult") + ")",
          interpreterContext, userSessionMap);


What error will be displayed in frontend if I set invalid value for zeppelin.livy.spark.sql.truncateResult ?

There will be an exception thrown. I don't think every property must be checked for validity in an explicit way.

What kind of exception do you see in frontend ? If user set an invalid value like 'F' and if the exception in frontend doesn't imply that, then it might be better to verify the property in method open(), otherwise it is hard to figure out what is wrong.

I don't see whether zeppelin.livy.spark.sql.maxResult or any other property is validated in any place. What's special about zeppelin.livy.spark.sql.truncateResult? Moreover, in my opinion, if user doesn't read documentation, and sets something completely wrong - it is his own problem.

@spektom I hate to be the bearer of bad news but Zeppelin is being adopted by many Enterprises and unforuantely in most cases I agree that it's the users problem. But if you are going to be committing to software that is going to be used by large enterprises then I would expect validation to occur. I am actually going to raise these issues with my Hadoop vendor.

zjffdu · 2016-10-08T10:06:59Z

docs/interpreter/livy.md

+  </tr>
+  <tr>
+    <td>zeppelin.livy.spark.sql.truncateResult</td>
+    <td>false</td>


true by default ?

mispecto · 2016-10-14T15:11:46Z

Are you trying to say that people working for enterprise companies require
special input validation in addition to providing a documentation saying
that an option accepts booleans? :)
I don't think every possible input must be validated, after all these are
Software engineers who will be setting up Zeppelin...
On Fri, Oct 14, 2016 at 18:01 gss2002 [email protected] wrote:

@gss2002 commented on this pull request.

In
livy/src/main/java/org/apache/zeppelin/livy/LivySparkSQLInterpreter.java
#1458:
@@ -80,7 +80,8 @@ public InterpreterResult interpret(String line, InterpreterContext interpreterCo
line.replaceAll(""", "\"")
.replaceAll("\n", " ")
+ "").show(" +
         property.get("zeppelin.livy.spark.sql.maxResult") + ")",
         property.get("zeppelin.livy.spark.sql.maxResult") + "," +
         property.get("zeppelin.livy.spark.sql.truncateResult") + ")",
   interpreterContext, userSessionMap);
@spektom https://github.com/spektom I hate to be the bearer of bad news
but Zeppelin is being adopted by many Enterprises and unforuantely in most
cases I agree that it's the users problem. But if you are going to be
committing to software that is going to be used by large enterprises then I
would expect validation to occur. I am actually going to raise these issues
with my Hadoop vendor.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1458, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAJpO8d402flYpiA3zkU_0nYTNaePF3Pks5qz5ksgaJpZM4KGJwn
.

gss2002 · 2016-10-14T15:20:52Z

Well we will just agree to disagree

gss2002 · 2016-10-14T15:41:13Z

Also input validation is secure java coding best practice regardless.. http://www.oracle.com/technetwork/java/seccodeguide-139067.html#5

close #83 close #86 close #125 close #133 close #139 close #146 close #193 close #203 close #246 close #262 close #264 close #273 close #291 close #299 close #320 close #347 close #389 close #413 close #423 close #543 close #560 close #658 close #670 close #728 close #765 close #777 close #782 close #783 close #812 close #822 close #841 close #843 close #878 close #884 close #918 close #989 close #1076 close #1135 close #1187 close #1231 close #1304 close #1316 close #1361 close #1385 close #1390 close #1414 close #1422 close #1425 close #1447 close #1458 close #1466 close #1485 close #1492 close #1495 close #1497 close #1536 close #1545 close #1561 close #1577 close #1600 close #1603 close #1678 close #1695 close #1739 close #1748 close #1765 close #1767 close #1776 close #1783 close #1799

mispecto force-pushed the ZEPPELIN-1486 branch from 589acda to e007395 Compare September 26, 2016 06:14

mispecto force-pushed the ZEPPELIN-1486 branch from e007395 to 3321e18 Compare September 27, 2016 08:14

mispecto force-pushed the ZEPPELIN-1486 branch from 3321e18 to 062835f Compare October 4, 2016 10:33

jongyoul reviewed Oct 5, 2016

View reviewed changes

mispecto force-pushed the ZEPPELIN-1486 branch from 062835f to e4c9abd Compare October 5, 2016 06:19

zjffdu reviewed Oct 8, 2016

View reviewed changes

mispecto force-pushed the ZEPPELIN-1486 branch from e4c9abd to 8634387 Compare October 9, 2016 04:52

[ZEPPELIN-1486] Allow configuring whether shown values will be trimmed

303a8bc

mispecto force-pushed the ZEPPELIN-1486 branch from 8634387 to 303a8bc Compare October 14, 2016 06:36

asfgit closed this in c38a0a0 May 9, 2018

[ZEPPELIN-1486] Allow configuring whether shown values will be trimmed #1458

[ZEPPELIN-1486] Allow configuring whether shown values will be trimmed #1458

Uh oh!

Conversation

mispecto commented Sep 26, 2016

What is this PR for?

What type of PR is it?

Todos

What is the Jira issue?

How should this be tested?

Screenshots (if appropriate)

Questions:

Uh oh!

AhyoungRyu commented Sep 27, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mispecto commented Sep 27, 2016

Uh oh!

r-kamath commented Oct 4, 2016

Uh oh!

mispecto commented Oct 4, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jongyoul commented Oct 5, 2016

Uh oh!

zjffdu Oct 8, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mispecto commented Oct 14, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

@gss2002 commented on this pull request.

Uh oh!

gss2002 commented Oct 14, 2016

Uh oh!

gss2002 commented Oct 14, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

AhyoungRyu commented Sep 27, 2016 •

edited

Loading

zjffdu Oct 8, 2016 •

edited

Loading

mispecto commented Oct 14, 2016 •

edited

Loading