Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix fillna to handle "ffill" and "bfill" properly. #1654

Merged
merged 1 commit into from
Jul 16, 2020

Conversation

ueshin
Copy link
Collaborator

@ueshin ueshin commented Jul 15, 2020

fillna with method="ffill" or method="bfill" uses Spark's Window functions internally, so we should create a new Spark DataFrame to apply the function properly.

Fixes #1651:

>>> from databricks.koalas import Series
>>> import numpy as np
>>> a = ["Bob", "John", None]
>>> b = ["Bob", "John", np.nan]
>>> s1 = Series(a)
>>> s2 = Series(b)
>>> r1 = s1.fillna(method="ffill")
>>> r1
0     Bob
1    John
2    John
Name: 0, dtype: object
>>> r1[2]
'John'
>>> r2 = s2.fillna(method="ffill")
>>> r2
0     Bob
1    John
2    John
Name: 0, dtype: object
>>> r2[2]
'John'

@ueshin
Copy link
Collaborator Author

ueshin commented Jul 16, 2020

Thanks! merging.

@ueshin ueshin merged commit 1a5d3d1 into databricks:master Jul 16, 2020
@ueshin ueshin deleted the fillna branch July 16, 2020 04:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fillna doesn't get the correct result when arg "method" is "ffill".
2 participants