Skip to content

[SPARK-41212][CONNECT][PYTHON] Implement DataFrame.isEmpty #38734

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zhengruifeng wants to merge 2 commits into from
+17 −0
Closed

[SPARK-41212][CONNECT][PYTHON] Implement DataFrame.isEmpty #38734

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c6712bf
init
zhengruifeng Nov 21, 2022
34accba
remove cache
zhengruifeng Nov 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions python/pyspark/sql/connect/dataframe.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,18 @@ def withPlan(cls, plan: plan.LogicalPlan, session: "RemoteSparkSession") -> "Dat
new_frame._plan = plan
return new_frame

def isEmpty(self) -> bool:
"""Returns ``True`` if this :class:`DataFrame` is empty.
Copy link
Contributor

@amaliujia amaliujia Nov 22, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit:

When I read this, I actually not sure if it means the size of DataFrame or it means DataFrame is None. I guess it is the size. Not sure if there is a better way to clarify this.

Copy link
Contributor Author

@zhengruifeng zhengruifeng Nov 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this doc was copied from pyspark and scala api. I think the dataframe itself is expected to be not None, otherwise, this method can not be called.


.. versionadded:: 3.4.0

Returns
-------
bool
Whether it's empty DataFrame or not.
"""
return len(self.take(1)) == 0
Copy link
Contributor

@cloud-fan cloud-fan Nov 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes me think that maybe we should have some shared code between pyspark and python connect client, to share some API definitions (api doc, function signature) and functions with default implementations. cc @HyukjinKwon


def select(self, *cols: "ExpressionOrString") -> "DataFrame":
return DataFrame.withPlan(plan.Project(self._plan, *cols), session=self._session)

Expand Down
5 changes: 5 additions & 0 deletions python/pyspark/sql/tests/connect/test_connect_basic.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -319,6 +319,11 @@ def test_empty_dataset(self):
self.assertEqual(1, len(pdf.columns)) # one column
self.assertEqual("X", pdf.columns[0])

def test_is_empty(self):
# SPARK-41212: Test is empty
self.assertFalse(self.connect.sql("SELECT 1 AS X").isEmpty())
self.assertTrue(self.connect.sql("SELECT 1 AS X LIMIT 0").isEmpty())

def test_session(self):
self.assertEqual(self.connect, self.connect.sql("SELECT 1").sparkSession())

Expand Down