Skip to content

[WIP] Arrow conversion at partitions #23

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
BryanCutler wants to merge 11 commits into from
Closed

[WIP] Arrow conversion at partitions #23

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
5eaa36d
added param description pydoc for toPandas
BryanCutler Feb 3, 2017
0e0f183
changed conversion to use Iterator[InternalRow] instead of Array
BryanCutler Feb 7, 2017
f7ecb4b
arrow conversion done at partition, tests not working yet
BryanCutler Feb 8, 2017
a68b496
modified python to concat tables, made Scala tests repartition to pass
BryanCutler Feb 8, 2017
fcd242d
changed empty frame test so that ArrowPayload on empty frame will be …
BryanCutler Feb 8, 2017
4b39e5a
removed option for initial vector capacity, using mapPartitionsIntern…
BryanCutler Feb 9, 2017
1fcd333
simplified column writer return types
BryanCutler Feb 15, 2017
274aaae
made naming consistent, moved Scala collectToArrow to test, cleanup
BryanCutler Feb 15, 2017
ef2ecb8
added some tests for complex types that are currently unsupported
BryanCutler Feb 15, 2017
3b8a8c6
changed protected member qualifiers, not really needed since all clas…
BryanCutler Feb 15, 2017
87a9b96
fixed partitioned conversion test, added test for empty partition col…
BryanCutler Feb 17, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions python/pyspark/sql/dataframe.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -398,7 +398,7 @@ def collectAsArrow(self):
"""
with SCCallSiteSync(self._sc) as css:
port = self._jdf.collectAsArrowToPython()
return list(_load_from_socket(port, ArrowSerializer()))[0]
return list(_load_from_socket(port, ArrowSerializer()))

@ignore_unicode_prefix
@since(2.0)
Expand Down Expand Up @@ -1573,6 +1573,9 @@ def toPandas(self, useArrow=False):

This is only available if Pandas is installed and available.

:param useArrow: Make use of Apache Arrow for conversion, pyarrow must be installed
on the calling Python process.

.. note:: This method should only be used if the resulting Pandas's DataFrame is expected
to be small, as all the data is loaded into the driver's memory.

Expand All @@ -1581,11 +1584,13 @@ def toPandas(self, useArrow=False):
0 2 Alice
1 5 Bob
"""
import pandas as pd

if useArrow:
return self.collectAsArrow().to_pandas()
from pyarrow.table import concat_tables
tables = self.collectAsArrow()
table = concat_tables(tables)
return table.to_pandas()
else:
import pandas as pd
return pd.DataFrame.from_records(self.collect(), columns=self.columns)

##########################################################################################
Expand Down
Loading