[SPARK-23030][SQL][PYTHON] Use Arrow stream format for creating from and collecting Pandas DataFrames#21546

Closed

BryanCutler wants to merge 34 commits intoapache:masterfrom

BryanCutler:arrow-toPandas-stream-SPARK-23030

Commits on Jun 7, 2018

Commits on Jun 8, 2018

toPandas is working with RecordBatch payloads, using custom handler to stream ordered partitions
BryanCutler
committed
cleanup and removed ArrowPayload, createDataFrame now working
BryanCutler
committed
toPandas and createDataFrame working but tests fail with date cols
BryanCutler
committed
removed usage of seekableByteChannel
BryanCutler
committed
for toPandas, set old collection result to null and add comments
BryanCutler
committed
cleanup, not yet passing ArrowConvertersSuite
BryanCutler
committed
fix to read Arrow stream with multiple batches, cleanup, add docs, scala tests pass, style pass
BryanCutler
committed
use base OutputStream for serveToStream instead of DataOutputStream
BryanCutler
committed
accidentally removed date type checks, passing pyspark tests
BryanCutler
committed

Commits on Jun 14, 2018

Commits on Jun 22, 2018

Commits on Aug 22, 2018

Merge remote-tracking branch 'upstream/master' into arrow-toPandas-stream-SPARK-23030
BryanCutler
committed

Commits on Aug 23, 2018

used tryWithResource, improved comments
BryanCutler
committed

Commits on Aug 27, 2018

Merge remote-tracking branch 'upstream/master' into arrow-toPandas-stream-SPARK-23030
BryanCutler
committed

Commits on Aug 28, 2018

Merge remote-tracking branch 'upstream/master' into arrow-toPandas-stream-SPARK-23030
BryanCutler
committed