Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: Additional export methods #236

Merged
merged 7 commits into from
Feb 25, 2023

Conversation

simicd
Copy link
Contributor

@simicd simicd commented Feb 25, 2023

Which issue does this PR close?

Closes #234
Closes #235

Rationale for this change

Implement additional export functions and fix bug if datafusion dataframe is empty

What changes are included in this PR?

  • Implement to_arrow_table(), to_pylist() and to_pydict()
  • Return empty dataframe/list/table instead of error if datafusion dataframe is empty
  • Document these export functions

Are there any user-facing changes?

See above

Comment on lines 50 to 52
# export to Polars dataframe
polars_df = pl.DataFrame(df.to_arrow_table())
assert polars_df.shape == (3, 2)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think it's worthwhile introducing a to_polars() helper function?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think that would be great

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback - added to_polars() method & unit tests for it

@simicd simicd changed the title Feature/convert dataframe feature: Additional export methods Feb 25, 2023
Copy link
Member

@andygrove andygrove left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great. Thanks @simicd

@andygrove andygrove merged commit d1f6567 into apache:main Feb 25, 2023
@simicd simicd deleted the feature/convert-dataframe branch February 26, 2023 21:04
@andygrove andygrove added the enhancement New feature or request label Mar 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Additional dataframe export functions Cannot convert empty datafusion dataframe to pandas
2 participants