Skip to content

[SPARK-41856][CONNECT][TESTS] Enable test_create_nan_decimal_dataframe, test_freqItems, test_input_files, test_to_pandas_required_pandas_not_found #39358

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
techaddict wants to merge 6 commits into from
Closed

[SPARK-41856][CONNECT][TESTS] Enable test_create_nan_decimal_dataframe, test_freqItems, test_input_files, test_to_pandas_required_pandas_not_found #39358

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b2d0d72
[SPARK-41856][CONNECT][TESTS] Enable test_create_nan_decimal_datafram…
techaddict Jan 3, 2023
1111153
address comments
techaddict Jan 3, 2023
70b402f
createDataFrame doesn't support numOfPartitions
techaddict Jan 3, 2023
2cae6db
test_dataframe requires more work
techaddict Jan 3, 2023
bbbdaaa
Merge remote-tracking branch 'upstream/master' into SPARK-41856
techaddict Jan 3, 2023
53fe2e2
test_create_nan_decimal_dataframe needs more work, will address in ne…
techaddict Jan 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 0 additions & 12 deletions python/pyspark/sql/tests/connect/test_parity_dataframe.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,10 +85,6 @@ def test_extended_hint_types(self):
def test_fillna(self):
super().test_fillna()

@unittest.skip("Fails in Spark Connect, should enable.")
def test_freqItems(self):
super().test_freqItems()

@unittest.skip("Fails in Spark Connect, should enable.")
def test_generic_hints(self):
super().test_generic_hints()
Expand All @@ -97,10 +93,6 @@ def test_generic_hints(self):
def test_help_command(self):
super().test_help_command()

@unittest.skip("Fails in Spark Connect, should enable.")
def test_input_files(self):
super().test_input_files()

@unittest.skip("Fails in Spark Connect, should enable.")
def test_invalid_join_method(self):
super().test_invalid_join_method()
Expand Down Expand Up @@ -193,10 +185,6 @@ def test_to_pandas_from_null_dataframe(self):
def test_to_pandas_on_cross_join(self):
super().test_to_pandas_on_cross_join()

@unittest.skip("Fails in Spark Connect, should enable.")
def test_to_pandas_required_pandas_not_found(self):
super().test_to_pandas_required_pandas_not_found()

@unittest.skip("Fails in Spark Connect, should enable.")
def test_to_pandas_with_duplicated_column_names(self):
super().test_to_pandas_with_duplicated_column_names()
Expand Down
2 changes: 1 addition & 1 deletion python/pyspark/sql/tests/test_dataframe.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ def test_duplicated_column_names(self):

def test_freqItems(self):
vals = [Row(a=1, b=-2.0) if i % 2 == 0 else Row(a=i, b=i * 1.0) for i in range(100)]
df = self.sc.parallelize(vals).toDF()
df = self.spark.createDataFrame(vals)
items = df.stat.freqItems(("a", "b"), 0.4).collect()[0]
self.assertTrue(1 in items[0])
self.assertTrue(-2.0 in items[1])
Expand Down