Skip to content

[DataFrame] Fix bug with consistency between IndexMetadata and partitions #2088

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
pschafhalter wants to merge 1 commit into from
Closed

[DataFrame] Fix bug with consistency between IndexMetadata and partitions #2088

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion python/ray/dataframe/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -460,4 +460,11 @@ def _correct_column_dtypes(*column):
Args:
"""
concat_column = pd.concat(column, copy=False)
return create_blocks_helper(concat_column, len(column), 1)
partitions = []
Copy link
Member

@devin-petersohn devin-petersohn May 19, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this not be fixed in create_blocks_helper? It looks like that would be the source of the problem.

Copy link
Contributor Author

@pschafhalter pschafhalter May 19, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could if we want to add another argument to create_blocks_helper to specify the number of rows in each partition.

I wasn't sure if we wanted to change the behavior of create_blocks_helper.

i = 0
for old_part in column:
new_part = concat_column.iloc[i:i + len(old_part)]
new_part.index = pd.RangeIndex(0, len(new_part))
partitions.append(new_part)
i += len(new_part)
return partitions