Distributed Series 🎉 by devin-petersohn · Pull Request #522 · modin-project/modin

devin-petersohn · 2019-03-29T22:28:35Z

What do these changes do?

Adds distributed Series compatibility to Modin. Currently a work in progress.

New addition can be tested with:

import modin.pandas as pd 
import numpy as np 
frame_data = np.random.randint(0, 100, size=(10**5, 1000)) 
df = pd.DataFrame(frame_data).add_prefix("col")
df["col1"] + df["col2"]

There is still several functionalities to add, in addition to the defaulting to pandas.

Related issue number

passes flake8 modin
passes black --check modin
tests added and passing

AmplabJenkins · 2019-03-29T22:29:05Z

Merged build finished. Test FAILed.

AmplabJenkins · 2019-03-29T22:29:07Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Modin-PRB/667/
Test FAILed.

codecov · 2019-03-29T22:52:23Z

Codecov Report

Merging #522 into master will decrease coverage by 0.01%.
The diff coverage is 92.44%.

@@            Coverage Diff             @@
##           master     #522      +/-   ##
==========================================
- Coverage   90.22%   90.21%   -0.02%     
==========================================
  Files          34       37       +3     
  Lines        4849     5406     +557     
==========================================
+ Hits         4375     4877     +502     
- Misses        474      529      +55

Impacted Files	Coverage Δ
modin/pandas/base.py	`95.03% <ø> (ø)`
modin/pandas/dataframe.py	`88.96% <ø> (-4.46%)`	⬇️
modin/data_management/utils.py	`83.72% <100%> (-0.73%)`	⬇️
modin/pandas/general.py	`97.72% <100%> (-2.28%)`	⬇️
modin/pandas/indexing.py	`81.28% <100%> (+0.51%)`	⬆️
modin/pandas/utils.py	`95.83% <100%> (ø)`	⬆️
modin/pandas/__init__.py	`85.1% <100%> (+0.32%)`	⬆️
modin/backends/base/query_compiler.py	`100% <100%> (ø)`	⬆️
modin/engines/ray/pandas_on_ray/frame/partition.py	`73.58% <66.66%> (-1.42%)`	⬇️
...din/engines/ray/generic/frame/partition_manager.py	`91.66% <80%> (-8.34%)`	⬇️
... and 13 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 62c2996...0f7736b. Read the comment docs.

AmplabJenkins · 2019-03-29T23:45:05Z

Merged build finished. Test FAILed.

AmplabJenkins · 2019-03-29T23:45:06Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Modin-PRB/668/
Test FAILed.

AmplabJenkins · 2019-03-30T05:29:18Z

Merged build finished. Test FAILed.

AmplabJenkins · 2019-03-30T05:29:18Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Modin-PRB/669/
Test FAILed.

williamma12

Looked over it quickly

modin/backends/pandas/query_compiler.py

modin/pandas/base.py

modin/backends/pandas/query_compiler.py

Update implementation Fix repr Update mapreduce functionality Update Add more methods work in progress updates from home Fix test files Add several methods to Series Fix comment Remove duplicate code Add some defaulting to pandas Push today's progress Fix some test failures Fix failures Fix indexing Fix python2 compat Address comments, cleanup Lint Cleanup Fix test error for Ray, propagate errors Lint

devin-petersohn

Other things to add:

Documentation section about Series
Finish tests

modin/backends/pandas/query_compiler.py

modin/data_management/utils.py

modin/engines/base/frame/partition_manager.py

modin/engines/ray/utils.py

modin/pandas/series.py

* complete tests upto and including test_timestamp * Move to_timestamp back to original spot

…ohn/modin into features/distributed_series

williamma12

Left some quick comments on some documentation, but otherwise it looks good!

modin/backends/pandas/query_compiler.py

williamma12

thanks for the much needed addition @devin-petersohn!

williamma12 reviewed Apr 1, 2019

View reviewed changes

modin/backends/pandas/query_compiler.py Outdated Show resolved Hide resolved

modin/backends/pandas/query_compiler.py Outdated Show resolved Hide resolved

modin/backends/pandas/query_compiler.py Outdated Show resolved Hide resolved

modin/pandas/base.py Outdated Show resolved Hide resolved

devin-petersohn commented Apr 1, 2019

View reviewed changes

devin-petersohn changed the title ~~[WIP] Distributed Series 🎉~~ Distributed Series 🎉 Apr 2, 2019

devin-petersohn force-pushed the features/distributed_series branch from 8931c60 to d8f01a5 Compare April 3, 2019 05:24

devin-petersohn mentioned this pull request Apr 3, 2019

iloc fails with ValueError: cannot reindex from a duplicate axis #524

Closed

devin-petersohn added 5 commits April 3, 2019 20:34

Fix failures and improve empty DataFrame ops

ee4f9ef

More cleanup

cf1ed12

Merge branch 'master' into features/distributed_series

2fca479

Fix regression

341fd67

Add some Series tests

2122cbc

devin-petersohn commented Apr 5, 2019

View reviewed changes

devin-petersohn added 11 commits April 7, 2019 20:29

Update tests

feeab6b

Add more tests

0aad069

Add tests

8a0253b

Add tests

3b34b50

Add more tests

a934324

Merge branch 'master' into features/distributed_series

1f03204

Add tests

3a6f256

Add tests

c6f9536

Add tests

093d367

Address comments

a3ffeae

Add Series to coverage test

541dc95

devin-petersohn mentioned this pull request Apr 12, 2019

AttributeError: module 'modin.pandas' has no attribute 'datetime' #542

Closed

devin-petersohn and others added 16 commits April 12, 2019 09:37

Run series tests on travis

9dc17c6

Add tests

54f0255

Add tests

a32c3b9

Fix general tests

5c1aa62

Add tests (#28)

0826dfb

* complete tests upto and including test_timestamp * Move to_timestamp back to original spot

Finished everything below and including valid (#29)

fae2173

Add tests

b5c7ef2

Merge branch 'features/distributed_series' of github.com:devin-peters…

0004e01

…ohn/modin into features/distributed_series

Lint

b7431f9

Fix flipped axis

76e268e

Fix failures for unmaterialized DataFrames

0767a2f

Lint

9e58526

Fix failures

6748829

Fix Py2 error

746d24d

Fix defaulting to pandas for Series

ccf29a5

Cleanup

f3648a2

williamma12 reviewed Apr 14, 2019

View reviewed changes

modin/backends/pandas/query_compiler.py Show resolved Hide resolved

modin/backends/pandas/query_compiler.py Show resolved Hide resolved

Address comments

0f7736b

williamma12 approved these changes Apr 14, 2019

View reviewed changes

williamma12 merged commit a071b5e into modin-project:master Apr 14, 2019

devin-petersohn mentioned this pull request Apr 17, 2019

"TypeError: 'NoneType' object is not iterable" when using pd.concat #562

Closed

This was referenced Apr 25, 2019

numeric_only=True ignored for sum, prod, min, max #577

Closed

QueryCompilerView does not correctly extract dtypes #592

Closed

Conversation

devin-petersohn commented Mar 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What do these changes do?

Related issue number

Uh oh!

AmplabJenkins commented Mar 29, 2019

Uh oh!

AmplabJenkins commented Mar 29, 2019

Uh oh!

codecov bot commented Mar 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

AmplabJenkins commented Mar 29, 2019

Uh oh!

AmplabJenkins commented Mar 29, 2019

Uh oh!

AmplabJenkins commented Mar 30, 2019

Uh oh!

AmplabJenkins commented Mar 30, 2019

Uh oh!

williamma12 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

devin-petersohn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

williamma12 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

williamma12 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

devin-petersohn commented Mar 29, 2019 •

edited

Loading

codecov bot commented Mar 29, 2019 •

edited

Loading