Fix Frame.abs to support bool type and disallow non-numeric types. #1980
Conversation
| } | ||
| ) | ||
| kdf = ks.from_pandas(pdf) | ||
| self.assert_eq(kdf.A.abs(), pdf.A.abs()) | ||
| self.assert_eq(kdf.B.abs(), pdf.B.abs()) | ||
| self.assert_eq(kdf.E.abs(), pdf.E.abs()) | ||
| # pandas' bug? | ||
| # self.assert_eq(kdf[["B", "C", "E"]].abs(), pdf[["B", "C", "E"]].abs()) |
There was a problem hiding this comment.
The pandas result is:
>>> pdf = pd.DataFrame(
... {
... "A": [1, -2, 3, -4, 5],
... "B": [1.0, -2, 3, -4, 5],
... "C": [-6.0, -7, -8, -9, 10],
... "D": ["a", "b", "c", "d", "e"],
... "E": [True, False, False, True, True],
... }
... )
>>> pdf[["B", "C", "E"]].abs()
B C E
0 1 6 1
1 2 7 0
2 3 8 0
3 4 9 1
4 5 10 1whereas calculating separately:
>>> pdf[["B", "C"]].abs()
B C
0 1.0 6.0
1 2.0 7.0
2 3.0 8.0
3 4.0 9.0
4 5.0 10.0
>>> pdf[["E"]].abs()
E
0 True
1 False
2 False
3 True
4 TrueI believe this is a pandas bug.
The current Koalas behavior:
>>> ks.from_pandas(pdf)[["B", "C", "E"]].abs()
B C E
0 1.0 6.0 True
1 2.0 7.0 False
2 3.0 8.0 False
3 4.0 9.0 True
4 5.0 10.0 True
There was a problem hiding this comment.
Seems pandas bug to me, too. Let me report this to pandas dev.
Codecov Report
@@ Coverage Diff @@
## master #1980 +/- ##
=======================================
Coverage 94.57% 94.57%
=======================================
Files 50 50
Lines 10946 10952 +6
=======================================
+ Hits 10352 10358 +6
Misses 594 594
Continue to review full report at Codecov.
|
| return kser.spark.transform(F.abs) | ||
| else: | ||
| raise TypeError( | ||
| "bad operand type for abs(): {}".format(kser.spark.data_type.simpleString()) |
There was a problem hiding this comment.
Hm, should we show pandas' dtype instead of Spark data type?
There was a problem hiding this comment.
In that case, shall we address it separately. The same error message will appear in the other places.
|
Thanks! merging. |
Addressing #1980 (comment) to add pandas dtypes.
Addressing databricks/koalas#1980 (comment) to add pandas dtypes.
Fixes
Frame.absto support bool type and disallow non-numeric types.