Skip to content

[DataFrame] Apply() for Lists and Dicts #1973

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
devin-petersohn merged 17 commits into from
May 4, 2018
Merged

[DataFrame] Apply() for Lists and Dicts #1973

devin-petersohn merged 17 commits into from
May 4, 2018

Conversation

@osalpekar
Copy link
Contributor

@osalpekar osalpekar commented May 1, 2018

No description provided.

p-yang
p-yang reviewed May 1, 2018
View reviewed changes
python/ray/dataframe/dataframe.py Outdated
Copy link
Contributor

@p-yang p-yang May 1, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can auto-unbox this by replacing tup with (p, iwp), key or something like that.

python/ray/dataframe/dataframe.py Outdated
Copy link
Contributor

@p-yang p-yang May 1, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we define behavior with non-unique columns? That will cause self._col_metadata[key] to result in a DataFrame and not a Series.

python/ray/dataframe/dataframe.py Outdated
Copy link
Contributor

@p-yang p-yang May 1, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

p-yang
p-yang reviewed May 1, 2018
View reviewed changes
python/ray/dataframe/index_metadata.py Outdated
Copy link
Contributor

@p-yang p-yang May 1, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You shouldn't need to change this in _IndexMetadataBase, it should be implemented in _IndexMetadata. Either way, _IndexMetadataBase is about to be deprecated by #1965

p-yang
p-yang approved these changes May 1, 2018
View reviewed changes
Copy link
Contributor

@p-yang p-yang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but let's wait for #1965 to get merged in to rebase and change the constructor.

python/ray/dataframe/dataframe.py Outdated
Copy link
Contributor

@p-yang p-yang May 1, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add col_metadata=self._col_metadata after #1965

Copy link
Member

@devin-petersohn devin-petersohn May 3, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping @osalpekar. Rebase and make this change, then looks great!

Copy link
Contributor Author

@osalpekar osalpekar May 3, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rebased and made change in commit below!

@AmplabJenkins
Copy link

AmplabJenkins commented May 1, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5132/
Test FAILed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 3, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5164/
Test PASSed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 4, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5199/
Test FAILed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 4, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5197/
Test PASSed.

@devin-petersohn
Copy link
Member

devin-petersohn commented May 4, 2018

Jenkins, retest this please.

@AmplabJenkins
Copy link

AmplabJenkins commented May 4, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5206/
Test FAILed.

@devin-petersohn
Copy link
Member

devin-petersohn commented May 4, 2018

Jenkins, retest this please.

@AmplabJenkins
Copy link

AmplabJenkins commented May 4, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5216/
Test PASSed.

@devin-petersohn devin-petersohn merged commit a1d7bb3 into ray-project:master May 4, 2018
@devin-petersohn
Copy link
Member

devin-petersohn commented May 4, 2018

Merged, thanks @osalpekar

alok added a commit to alok/ray that referenced this pull request May 8, 2018
@alok
Merge branch 'master' into trpo
0fd8f4a 
* master: (21 commits)
  Expand local_dir in Trial init (ray-project#2013)
  Fixing ascii error for Python2 (ray-project#2009)
  [DataFrame] Implements df.update (ray-project#1997)
  [DataFrame] Implements df.as_matrix (ray-project#2001)
  [DataFrame] Implement quantile (ray-project#1992)
  [DataFrame] Impement sort_values and sort_index (ray-project#1977)
  [DataFrame] Implement rank (ray-project#1991)
  [DataFrame] Implemented prod, product, added test suite (ray-project#1994)
  [DataFrame] Implemented __setitem__, select_dtypes, and astype (ray-project#1941)
  [DataFrame] Implement diff (ray-project#1996)
  [DataFrame] Implemented nunique, skew (ray-project#1995)
  [DataFrame] Implements filter and dropna (ray-project#1959)
  [DataFrame] Implements df.pipe (ray-project#1999)
  [DataFrame] Apply() for Lists and Dicts (ray-project#1973)
  Clean up syntax for supported Python versions. (ray-project#1963)
  [DataFrame] Implements mode, to_datetime, and get_dummies (ray-project#1956)
  [DataFrame] Fix dtypes (ray-project#1930)
  keep_dims -> keepdims (ray-project#1980)
  add pthread linking (ray-project#1986)
  [DataFrame] Add layer of abstraction to allow OID instantiation (ray-project#1984)
  ...
alok added a commit to alok/ray that referenced this pull request May 9, 2018
@alok
Merge branch 'master' into comps
329834e 
* master: (25 commits)
  [DataFrame] Add direct pandas imports for MVP (ray-project#1960)
  Make ActorHandles pickleable, also make proper ActorHandle and ActorC… (ray-project#2007)
  Expand local_dir in Trial init (ray-project#2013)
  Fixing ascii error for Python2 (ray-project#2009)
  [DataFrame] Implements df.update (ray-project#1997)
  [DataFrame] Implements df.as_matrix (ray-project#2001)
  [DataFrame] Implement quantile (ray-project#1992)
  [DataFrame] Impement sort_values and sort_index (ray-project#1977)
  [DataFrame] Implement rank (ray-project#1991)
  [DataFrame] Implemented prod, product, added test suite (ray-project#1994)
  [DataFrame] Implemented __setitem__, select_dtypes, and astype (ray-project#1941)
  [DataFrame] Implement diff (ray-project#1996)
  [DataFrame] Implemented nunique, skew (ray-project#1995)
  [DataFrame] Implements filter and dropna (ray-project#1959)
  [DataFrame] Implements df.pipe (ray-project#1999)
  [DataFrame] Apply() for Lists and Dicts (ray-project#1973)
  Clean up syntax for supported Python versions. (ray-project#1963)
  [DataFrame] Implements mode, to_datetime, and get_dummies (ray-project#1956)
  [DataFrame] Fix dtypes (ray-project#1930)
  keep_dims -> keepdims (ray-project#1980)
  ...
alok added a commit to alok/ray that referenced this pull request May 9, 2018
@alok
Merge branch 'master' into magic-methods
601e67b 
* master:
  [DataFrame] Add direct pandas imports for MVP (ray-project#1960)
  Make ActorHandles pickleable, also make proper ActorHandle and ActorC… (ray-project#2007)
  Expand local_dir in Trial init (ray-project#2013)
  Fixing ascii error for Python2 (ray-project#2009)
  [DataFrame] Implements df.update (ray-project#1997)
  [DataFrame] Implements df.as_matrix (ray-project#2001)
  [DataFrame] Implement quantile (ray-project#1992)
  [DataFrame] Impement sort_values and sort_index (ray-project#1977)
  [DataFrame] Implement rank (ray-project#1991)
  [DataFrame] Implemented prod, product, added test suite (ray-project#1994)
  [DataFrame] Implemented __setitem__, select_dtypes, and astype (ray-project#1941)
  [DataFrame] Implement diff (ray-project#1996)
  [DataFrame] Implemented nunique, skew (ray-project#1995)
  [DataFrame] Implements filter and dropna (ray-project#1959)
  [DataFrame] Implements df.pipe (ray-project#1999)
  [DataFrame] Apply() for Lists and Dicts (ray-project#1973)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@devin-petersohn devin-petersohn devin-petersohn approved these changes

+1 more reviewer

@p-yang p-yang p-yang approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@osalpekar @AmplabJenkins @devin-petersohn @p-yang