[SPARK-14828][SQL] Start SparkSession in REPL instead of SQLContext #12589

andrewor14 · 2016-04-22T00:10:15Z

What changes were proposed in this pull request?

Spark context available as 'sc' (master = local[*], app id = local-1461283768192).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.0.0-SNAPSHOT
      /_/

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_51)
Type in expressions to have them evaluated.
Type :help for more information.

scala> sql("SHOW TABLES").collect()
16/04/21 17:09:39 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/04/21 17:09:39 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
res0: Array[org.apache.spark.sql.Row] = Array([src,false])

scala> sql("SHOW TABLES").collect()
res1: Array[org.apache.spark.sql.Row] = Array([src,false])

scala> spark.createDataFrame(Seq((1, 1), (2, 2), (3, 3)))
res2: org.apache.spark.sql.DataFrame = [_1: int, _2: int]

Hive things are loaded lazily.

How was this patch tested?

Manual.

rxin · 2016-04-22T00:11:39Z

I think there was a suggestion from @marmbrus to just name this "spark". Would it cause any problems because it conflicts with the package name?

andrewor14 · 2016-04-22T00:12:07Z

OK, I like spark better too. Let's try it out.

marmbrus · 2016-04-22T00:45:32Z

Yeah, if there are problems we don't have to do that, but val df = spark.read.json(...) is pretty nice

andrewor14 · 2016-04-22T01:40:28Z

I don't see any problems off the top of my head. The only collision is when people do import org.apache.spark, but who does that?

SparkQA · 2016-04-22T01:51:24Z

Test build #56614 has finished for PR 12589 at commit e7dee4f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-22T02:09:35Z

Test build #56616 has finished for PR 12589 at commit 45783de.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-04-22T20:35:26Z

LGTM - let's see if we run into issues in the future with this.

yhuai · 2016-04-22T20:45:10Z

repl/scala-2.10/src/main/scala/org/apache/spark/repl/SparkILoop.scala

-    }
-    sqlContext
-  }
+  def createSparkSession(): SparkSession = Main.createSparkSession()


Does repl/scala-2.10/src/main/scala/org/apache/spark/repl/Main.scala have createSparkSession?

hm, is there a way to do this without duplicating code?

Not very sure. How about we still duplicate the code for now?

SparkQA · 2016-04-22T20:56:54Z

Test build #56720 has finished for PR 12589 at commit 8642cd7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-25T19:34:38Z

Test build #56905 has finished for PR 12589 at commit e69d7cf.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

## What changes were proposed in this pull request? This removes the class `HiveContext` itself along with all code usages associated with it. The bulk of the work was already done in #12485. This is mainly just code cleanup and actually removing the class. Note: A couple of things will break after this patch. These will be fixed separately. - the python HiveContext - all the documentation / comments referencing HiveContext - there will be no more HiveContext in the REPL (fixed by #12589) ## How was this patch tested? No change in functionality. Author: Andrew Or <[email protected]> Closes #12585 from andrewor14/delete-hive-context.

SparkQA · 2016-04-25T22:04:37Z

Test build #2872 has finished for PR 12589 at commit 601df7c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class GaussianMixtureModel(JavaModel, JavaMLWritable, JavaMLReadable):
- class GaussianMixture(JavaEstimator, HasFeaturesCol, HasPredictionCol, HasMaxIter, HasTol, HasSeed,

SparkQA · 2016-04-25T22:04:50Z

Test build #2873 has finished for PR 12589 at commit 601df7c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class GaussianMixtureModel(JavaModel, JavaMLWritable, JavaMLReadable):
- class GaussianMixture(JavaEstimator, HasFeaturesCol, HasPredictionCol, HasMaxIter, HasTol, HasSeed,

SparkQA · 2016-04-25T22:05:35Z

Test build #2874 has finished for PR 12589 at commit 601df7c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class GaussianMixtureModel(JavaModel, JavaMLWritable, JavaMLReadable):
- class GaussianMixture(JavaEstimator, HasFeaturesCol, HasPredictionCol, HasMaxIter, HasTol, HasSeed,

SparkQA · 2016-04-25T22:25:31Z

Test build #56919 has finished for PR 12589 at commit 601df7c.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class GaussianMixtureModel(JavaModel, JavaMLWritable, JavaMLReadable):
- class GaussianMixture(JavaEstimator, HasFeaturesCol, HasPredictionCol, HasMaxIter, HasTol, HasSeed,

rxin · 2016-04-25T22:29:54Z

Thanks - merging in master.

felixcheung · 2016-04-25T23:07:04Z

should this go to pyspark and sparkR shell as well?
or rather, should SparkSession be in Python and R first?

andrewor14 · 2016-04-26T01:57:15Z

it'll be there later

Initialize sparkSession in REPL instead of sqlContext

e7dee4f

sparkSession -> spark

45783de

andrewor14 mentioned this pull request Apr 22, 2016

[SPARK-14721][SQL] Remove HiveContext (part 2) #12585

Closed

Andrew Or added 2 commits April 21, 2016 19:21

Merge branch 'master' of github.com:apache/spark into spark-session-repl

b9b6349

Fix in-memory case

8642cd7

yhuai reviewed Apr 22, 2016
View reviewed changes

Andrew Or added 2 commits April 25, 2016 10:48

Merge branch 'master' of github.com:apache/spark into spark-session-repl

91e1da5

Just do-plicate it (check)

e69d7cf

Merge branch 'master' of github.com:apache/spark into spark-session-repl

601df7c

asfgit closed this in 34336b6 Apr 25, 2016

andrewor14 deleted the spark-session-repl branch April 25, 2016 22:32

[SPARK-14828][SQL] Start SparkSession in REPL instead of SQLContext #12589

[SPARK-14828][SQL] Start SparkSession in REPL instead of SQLContext #12589

Uh oh!

Conversation

andrewor14 commented Apr 22, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

rxin commented Apr 22, 2016

Uh oh!

andrewor14 commented Apr 22, 2016

Uh oh!

marmbrus commented Apr 22, 2016

Uh oh!

andrewor14 commented Apr 22, 2016

Uh oh!

SparkQA commented Apr 22, 2016

Uh oh!

SparkQA commented Apr 22, 2016

Uh oh!

rxin commented Apr 22, 2016

Uh oh!

yhuai Apr 22, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewor14 Apr 23, 2016

Choose a reason for hiding this comment

Uh oh!

yhuai Apr 23, 2016

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 22, 2016

Uh oh!

SparkQA commented Apr 25, 2016

Uh oh!

SparkQA commented Apr 25, 2016

Uh oh!

SparkQA commented Apr 25, 2016

Uh oh!

SparkQA commented Apr 25, 2016

Uh oh!

SparkQA commented Apr 25, 2016

Uh oh!

rxin commented Apr 25, 2016

Uh oh!

felixcheung commented Apr 25, 2016

Uh oh!

andrewor14 commented Apr 26, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

andrewor14 commented Apr 22, 2016 •

edited

Loading

yhuai Apr 22, 2016 •

edited

Loading