Skip to content

Conversation

@HyukjinKwon
Copy link
Member

@HyukjinKwon HyukjinKwon commented Jul 14, 2020

What changes were proposed in this pull request?

This PR proposes to port forward #29098 to collectAsArrowToR. collectAsArrowToR follows collectAsArrowToPython in branch-2.4 due to the limitation of ARROW-4512. SparkR vectorization currently cannot use streaming format.

Why are the changes needed?

For simplicity and consistency.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

The same code is being tested in collectAsArrowToPython of branch-2.4.

@SparkQA
Copy link

SparkQA commented Jul 14, 2020

Test build #125819 has finished for PR 29100 at commit 8c6be21.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@ueshin
Copy link
Member

ueshin commented Jul 14, 2020

Note that you cannot create no partition Spark DataFrame in SparkR

How about something like filter(df, FALSE)?

@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Jul 14, 2020

I see, that case seems not working even after this fix. Let me forward port this first with #29098, and will work separately on that.

@HyukjinKwon
Copy link
Member Author

I am going to merge this since this is just a forward-port that doesn't cause any behaviour change. I am 100% confident about this fix.

@HyukjinKwon
Copy link
Member Author

Merged to master and branch-3.0.

HyukjinKwon added a commit that referenced this pull request Jul 14, 2020
…oPython

### What changes were proposed in this pull request?

This PR proposes to port forward #29098 to `collectAsArrowToR`. `collectAsArrowToR` follows `collectAsArrowToPython` in branch-2.4 due to the limitation of ARROW-4512. SparkR vectorization currently cannot use streaming format.

### Why are the changes needed?

For simplicity and consistency.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

The same code is being tested in `collectAsArrowToPython` of branch-2.4.

Closes #29100 from HyukjinKwon/minor-parts.

Authored-by: HyukjinKwon <[email protected]>
Signed-off-by: HyukjinKwon <[email protected]>
(cherry picked from commit 03b5707)
Signed-off-by: HyukjinKwon <[email protected]>
@HyukjinKwon HyukjinKwon deleted the minor-parts branch July 27, 2020 07:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants