[DataFrame] Fix bug with consistency between IndexMetadata and partitions

[pschafhalter]

Calling _correct_column_dtypes rebuilds partitions using create_blocks_helper. However, create_blocks_helper creates partitions of equal size which results in an inconsistency between IndexMetadata and column partitions. This PR aims to fix the issue by ensuring that _correct_column_dtypes constructs partitions of the same size as before.

Code which demonstrates the bug:

import ray
import ray.dataframe as rdf
df = rdf.DataFrame({"col": list(range(1000000))})
df.to_csv("test_df.csv")
read_df = rdf.read_csv("test_df.csv")
print(read_df._row_metadata._lengths) 
print(ray.get(rdf.utils._map_partitions(lambda df: len(df), read_df._row_partitions)))

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5465/
Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5484/
Test PASSed.

[devin-petersohn]

Can this not be fixed in create_blocks_helper? It looks like that would be the source of the problem.

[pschafhalter]

It could if we want to add another argument to create_blocks_helper to specify the number of rows in each partition.

I wasn't sure if we wanted to change the behavior of create_blocks_helper.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5493/
Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5494/
Test PASSed.

[pschafhalter]

All tests passed on private travis for current commit.

[pschafhalter]

Deprecated due to #2118.