Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read numpy nans/datetimes #741

Merged
merged 10 commits into from
Sep 25, 2019
Merged

Read numpy nans/datetimes #741

merged 10 commits into from
Sep 25, 2019

Conversation

sc1f
Copy link
Contributor

@sc1f sc1f commented Sep 24, 2019

This PR improves the Python API's ability to parse Pandas DataFrames, ensuring compatibility with numpy.nan and numpy.datetime64.

  • Columns that begin with numpy.nan will now have their types inferred as float, as numpy.nan is a float type. If the column contains a value (i.e. [nan, nan, 1]) later on, and the value is not of float or integer type, then the column will be promoted to a string, all numpy.nans will be converted to None, and subsequent writes will be converted to string.
  • numpy.nan serializes to None when exported from Perspective.
  • Timestamp generation has been moved to the date parser, and numpy.datetime64 correctly returns Unix timestamps for use within the C++ engine.

python/perspective/perspective/table/_accessor.py Outdated Show resolved Hide resolved
python/perspective/perspective/table/_date_validator.py Outdated Show resolved Hide resolved
python/perspective/perspective/tests/core/test_layout.py Outdated Show resolved Hide resolved
python/perspective/perspective/src/fill.cpp Outdated Show resolved Hide resolved
python/perspective/perspective/src/fill.cpp Outdated Show resolved Hide resolved
python/perspective/perspective/src/fill.cpp Outdated Show resolved Hide resolved
python/perspective/perspective/table/_date_validator.py Outdated Show resolved Hide resolved
python/perspective/perspective/table/_date_validator.py Outdated Show resolved Hide resolved
python/perspective/perspective/src/fill.cpp Outdated Show resolved Hide resolved
@timkpaine timkpaine changed the title Correctly parse dataframes, streamline datetime processing Read numpy nans Sep 24, 2019
@timkpaine timkpaine changed the title Read numpy nans Read numpy nans/datetimes Sep 24, 2019
@codecov-io
Copy link

codecov-io commented Sep 24, 2019

Codecov Report

Merging #741 into master will increase coverage by 1.06%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #741      +/-   ##
==========================================
+ Coverage    92.3%   93.36%   +1.06%     
==========================================
  Files          61       61              
  Lines        2703     2970     +267     
==========================================
+ Hits         2495     2773     +278     
+ Misses        208      197      -11
Impacted Files Coverage Δ
perspective/table/_date_validator.py 86.84% <0%> (-0.66%) ⬇️
perspective/tests/node/test_node.py 31.25% <0%> (ø) ⬆️
perspective/table/view.py 80.95% <0%> (ø) ⬆️
perspective/tests/table/test_table_pandas.py 100% <0%> (ø) ⬆️
perspective/tests/table/test_view.py 98.55% <0%> (+0.09%) ⬆️
perspective/table/_accessor.py 92.13% <0%> (+1.99%) ⬆️
perspective/table/table.py 94.94% <0%> (+3.94%) ⬆️
perspective/tests/table/test_update.py 100% <0%> (+11.23%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9481128...159af03. Read the comment docs.

@texodus
Copy link
Member

texodus commented Sep 25, 2019

Thanks for the PR!

@texodus texodus merged commit 82155a7 into master Sep 25, 2019
@texodus texodus deleted the dataframe-fix branch September 25, 2019 19:48
@texodus texodus added the bug Concrete, reproducible bugs label Jan 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Concrete, reproducible bugs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants