Skip to content

[SPARK-25003][PYSPARK] Use SessionExtensions in Pyspark #21990

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
RussellSpitzer wants to merge 6 commits into from
Closed

[SPARK-25003][PYSPARK] Use SessionExtensions in Pyspark #21990

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
84c2513
[SPARK-25003][PYSPARK] Use SessionExtensions in Pyspark
RussellSpitzer Aug 3, 2018
def4f3e
[SPARK-25003][PYSPARK]: Add Tests for spark.sql.extensions in Pyspark
RussellSpitzer Aug 16, 2018
67d9772
SPARK-25003: Add helper methods to create new Extensions from Conf
RussellSpitzer Sep 19, 2018
4ddaff8
SPARK-25003: Address Reviewer Comments
RussellSpitzer Oct 1, 2018
d9b2a55
SPARK-25003: More Refactoring
RussellSpitzer Oct 16, 2018
3629c78
SPARK-25003: Refactor ApplyExtensions Function
RussellSpitzer Oct 17, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion python/pyspark/sql/session.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -218,7 +218,9 @@ def __init__(self, sparkContext, jsparkSession=None):
.sparkContext().isStopped():
jsparkSession = self._jvm.SparkSession.getDefaultSession().get()
else:
jsparkSession = self._jvm.SparkSession(self._jsc.sc())
jsparkSession = self._jvm.SparkSession.builder() \
.sparkContext(self._jsc.sc()) \
.getOrCreate()
Copy link
Member

@HyukjinKwon HyukjinKwon Aug 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@RussellSpitzer, mind checking the logic getOrCreate inside Scala side and deduplicate them here while we are here? Some logics for instance setting default session, etc. are duplicated Here in Python side and there in Scala side.

It would be nicer if we have some tests as well. spark.sql.extensions are static configuration, right? in that case, we could add a test, for example, please refer #21007. I added a test with static configuration before there.

Copy link
Member Author

@RussellSpitzer RussellSpitzer Aug 15, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah let me add in the test, and then I'll clear out all the python duplication of Scala code. I can make it more of a wrapper and less of a reimplementer.

self._jsparkSession = jsparkSession
self._jwrapped = self._jsparkSession.sqlContext()
self._wrapped = SQLContext(self._sc, self, self._jwrapped)
Expand Down