[SPARK-6392][SQL]Minor fix ClassNotFound exception when use spark cli to add jar #5079

jeanlyn · 2015-03-18T02:22:19Z

When we use spark cli to add jar dynamic,we will get the java.lang.ClassNotFoundException when we use the class of jar to create udf.For example:

spark-sql> add jar /home/jeanlyn/hello.jar;
spark-sql>create temporary function hello as 'hello';
spark-sql>select hello(name) from person;
Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1, localhost): java.lang.ClassNotFoundException: hello

we can use the spark physical plan to fix this problem

AmplabJenkins · 2015-03-18T02:27:11Z

Can one of the admins verify this patch?

marmbrus · 2015-03-18T02:37:06Z

ok to test

marmbrus · 2015-03-18T02:37:44Z

Would it be possible to add a test case in the CLI suite? /cc @liancheng

SparkQA · 2015-03-18T02:43:17Z

Test build #28764 has started for PR 5079 at commit ca95849.

This patch merges cleanly.

SparkQA · 2015-03-18T04:03:34Z

Test build #28764 has finished for PR 5079 at commit ca95849.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-03-18T04:03:37Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28764/
Test PASSed.

liancheng · 2015-03-18T07:45:51Z

@marmbrus We can merge this PR first, I'm trying to write a test case for this PR.

SparkQA · 2015-03-18T10:43:12Z

Test build #28792 has started for PR 5079 at commit ca78d72.

This patch merges cleanly.

jeanlyn · 2015-03-18T11:16:08Z

Updated, @liancheng @marmbrus I had tried to add a test for this patch,could you take a look for the test?Thanks!

SparkQA · 2015-03-18T12:02:18Z

Test build #28792 has finished for PR 5079 at commit ca78d72.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-03-18T12:02:21Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28792/
Test PASSed.

liancheng · 2015-03-18T13:19:19Z

Hm, the problem is that we should also add the source code of hello.jar (probably after renaming it to TestUdf.jar). But I'm not quite sure where to put it for now. That's why I said we can merge the fix first, and later add the test in a proper way.

In Hive, there is separate Maven module itest/test-serde, which results in a similar TestSerDe.jar solely for testing purposes. I wonder is there a simpler way to handle this. Adding a module seems to be an overkill.

/cc @marmbrus @yhuai @pwendell

jeanlyn · 2015-03-18T13:28:15Z

Thanks @liancheng for explain.You are right,it need consider more about it.So,should i remove the test?

liancheng · 2015-03-18T13:32:12Z

Your test case itself makes sense. Let's wait for more comments first :)

jeanlyn · 2015-03-18T13:41:14Z

Ok.

yhuai · 2015-03-18T16:14:32Z

Will it be better if we put the jar in sql/hive-thriftserver/src/test/resources/jar (not in data)? Also, what is in that jar? A hello world function?

SparkQA · 2015-03-19T01:37:36Z

Test build #28849 has started for PR 5079 at commit 2de3945.

This patch merges cleanly.

jeanlyn · 2015-03-19T01:41:03Z

@yhuai ,There is a simple function

    public String evaluate(String str) {
        try {
            return "hello " + str;
        } catch (Exception e) {
            return null;
        }
    }

chenghao-intel · 2015-03-19T02:06:06Z

Same issue with #4586 ? Actually we are quite headache with the class loading/unloading problem. @adrian-wang can you review this also?

jeanlyn · 2015-03-19T02:18:02Z

@chenghao-intel I am not clear what problem #4586 try to fix.If #4586 try to fix the problem as I mentioned.I think we can reuse the SparkContext.addJar is enough to fix the class loading/unloading problem.Right?

adrian-wang · 2015-03-19T02:54:18Z

Actually I tried this method before and it would work in unit test but not ok with shell.
I'll double check this in the afternoon.

SparkQA · 2015-03-19T03:00:24Z

Test build #28849 has finished for PR 5079 at commit 2de3945.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-03-19T03:00:28Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28849/
Test PASSed.

jeanlyn · 2015-03-19T03:41:20Z

@adrian-wang You mean not work in spark-shell ?

adrian-wang · 2015-03-19T04:18:02Z

@jeanlyn have you tried clean assembly and run it in spark-sql?

adrian-wang · 2015-03-19T05:18:14Z

I double checked your code with latest master, the problem still persists.
Considered this is just a dup of #4586 maybe we should close this one.

Thanks!

jeanlyn · 2015-03-19T05:51:42Z

@adrian-wang ,I had tested in spark-sql ,and get result correctly with my test case. Can you provide your test case?By the way,when i debug this issue i found in the thrifter-server mode,it also reuse the SparkContext.addJar.

adrian-wang · 2015-03-19T05:56:16Z

@jeanlyn you can just try execute mapjoin_addjar.q in spark-sql.
Note you need to find a valid jar position.

chenghao-intel · 2015-03-19T06:28:24Z

The tests in 2 PRs are different, this PR is about the UDF jar, but #4586 is the SerDe jar. They may be loaded by difference class loader.

@jeanlyn can you paste the full code for the UDF function?

jeanlyn · 2015-03-19T06:58:32Z

@chenghao-intel my full code is

import org.apache.hadoop.hive.ql.exec.UDF;

public class hello extends UDF {
    public String evaluate(String str) {
        try {
            return "hello " + str;
        } catch (Exception e) {
            return null;
        }
    }
}

@adrian-wang ,I also test the mapjoin_addjar.q in spark-sql.
I got the exception when CREATE TABLE

15/03/19 14:41:36 ERROR DDLTask: java.lang.NoSuchFieldError: CHAR
at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:310)
    at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:277)
    at org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult$lzycompute(NativeCommand.scala:35)
    at org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult(NativeCommand.scala:35)

But it seems that not the load jar problem.Because when i not run the

add jar ${system:maven.local.repository}/org/apache/hive/hcatalog/hive-hcatalog-core/${system:hive.version}/hive-hcatalog-core-${system:hive.version}.jar;

I got the follow exception when i create table

15/03/19 14:54:51 ERROR DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException: Cannot validate serde: org.apache.hive.hcatalog.data.JsonSerDe
    at org.apache.hadoop.hive.ql.exec.DDLTask.validateSerDe(DDLTask.java:3423)
    at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3553)
    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:252)

adrian-wang · 2015-03-19T07:01:52Z

@jeanlyn we are not getting same thing. Even our .q file differs. I don't have CHAR in my .q file.

jeanlyn · 2015-03-19T07:30:25Z

I also don't have CHAR in mapjoin_addjar.q. I only find one mapjoin_addjar.q,and the path of my file is
sql/hive/src/test/resources/ql/src/test/queries/clientpositive/mapjoin_addjar.q

set hive.auto.convert.join=true;
set hive.auto.convert.join.use.nonstaged=false;

add jar ${system:maven.local.repository}/org/apache/hive/hcatalog/hive-hcatalog-core/${system:hive.version}/hive-hcatalog-core-${system:hive.version}.jar;

CREATE TABLE t1 (a string, b string)
ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
;
LOAD DATA LOCAL INPATH "../../data/files/sample.json" INTO TABLE t1;
select * from src join t1 on src.key =t1.a;
drop table t1;
set hive.auto.convert.join=false;

May be we can discuss this offline?

jeanlyn · 2015-03-23T16:38:24Z

After communicated with @adrian-wang offline. I realized this PR still leave some class loader problem.So i close this one.

use spark physical plan when add jar

ca95849

add test

ca78d72

rename udf

2de3945

jeanlyn closed this Mar 23, 2015

jeanlyn deleted the SPARK-6392 branch July 3, 2015 05:52

[SPARK-6392][SQL]Minor fix ClassNotFound exception when use spark cli to add jar #5079

[SPARK-6392][SQL]Minor fix ClassNotFound exception when use spark cli to add jar #5079

Uh oh!

Conversation

jeanlyn commented Mar 18, 2015

Uh oh!

AmplabJenkins commented Mar 18, 2015

Uh oh!

marmbrus commented Mar 18, 2015

Uh oh!

marmbrus commented Mar 18, 2015

Uh oh!

SparkQA commented Mar 18, 2015

Uh oh!

SparkQA commented Mar 18, 2015

Uh oh!

AmplabJenkins commented Mar 18, 2015

Uh oh!

liancheng commented Mar 18, 2015

Uh oh!

SparkQA commented Mar 18, 2015

Uh oh!

jeanlyn commented Mar 18, 2015

Uh oh!

SparkQA commented Mar 18, 2015

Uh oh!

AmplabJenkins commented Mar 18, 2015

Uh oh!

liancheng commented Mar 18, 2015

Uh oh!

jeanlyn commented Mar 18, 2015

Uh oh!

liancheng commented Mar 18, 2015

Uh oh!

jeanlyn commented Mar 18, 2015

Uh oh!

yhuai commented Mar 18, 2015

Uh oh!

SparkQA commented Mar 19, 2015

Uh oh!

jeanlyn commented Mar 19, 2015

Uh oh!

chenghao-intel commented Mar 19, 2015

Uh oh!

jeanlyn commented Mar 19, 2015

Uh oh!

adrian-wang commented Mar 19, 2015

Uh oh!

SparkQA commented Mar 19, 2015

Uh oh!

AmplabJenkins commented Mar 19, 2015

Uh oh!

jeanlyn commented Mar 19, 2015

Uh oh!

adrian-wang commented Mar 19, 2015

Uh oh!

adrian-wang commented Mar 19, 2015

Uh oh!

jeanlyn commented Mar 19, 2015

Uh oh!

adrian-wang commented Mar 19, 2015

Uh oh!

chenghao-intel commented Mar 19, 2015

Uh oh!

jeanlyn commented Mar 19, 2015

Uh oh!

adrian-wang commented Mar 19, 2015

Uh oh!

jeanlyn commented Mar 19, 2015

Uh oh!

jeanlyn commented Mar 23, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants