Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(parquet): Implement AsyncFileWriter for object_store::buffered::BufWriter #6013

Merged
merged 9 commits into from
Aug 6, 2024

Conversation

Xuanwo
Copy link
Member

@Xuanwo Xuanwo commented Jul 7, 2024

Which issue does this PR close?

Closes #5766

Rationale for this change

Implement AsyncFileWriter for object_store::buffered::BufWriter

What changes are included in this PR?

Add a new impl for AsyncFileWriter

Are there any user-facing changes?

New impl added.

@github-actions github-actions bot added the parquet Changes to the parquet crate label Jul 7, 2024
@Xuanwo Xuanwo marked this pull request as draft July 7, 2024 05:25
Signed-off-by: Xuanwo <[email protected]>
@Xuanwo
Copy link
Member Author

Xuanwo commented Jul 7, 2024

This PR is blocked by BufWriter::put API is not released yet.

@Xuanwo
Copy link
Member Author

Xuanwo commented Jul 8, 2024

Hi, @alamb @tustvold, is it a good idea to run a patch release for object_store? It must be good for this change to be included in next minor parquet release.

@alamb
Copy link
Contributor

alamb commented Jul 8, 2024

Thanks for the reminder @Xuanwo

I filed this ticket to track the next release #6022

I looked at the commits that went into object_store: https://github.com/apache/arrow-rs/commits/master/object_store

As of today there do not appear to be any breaking API changes to object store and thus I think we can release a minor 0.10.1 version

@Xuanwo Xuanwo marked this pull request as ready for review August 2, 2024 04:07
@Xuanwo
Copy link
Member Author

Xuanwo commented Aug 2, 2024

Hi, @alamb, Sorry for the long waiting, this PR is ready to review now.

@alamb alamb changed the title feat(parquet): Implement AsyncFileWriter for obejct_store::buffered::BufWriter feat(parquet): Implement AsyncFileWriter for object_store::buffered::BufWriter Aug 2, 2024
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Xuanwo this is looking great. I had a few documentation and API naming suggestions

Maybe we could add a mention of this ParquetObjectWriter to the docs here too: https://docs.rs/parquet/latest/parquet/arrow/async_writer/index.html

parquet/Cargo.toml Show resolved Hide resolved
parquet/src/arrow/async_writer/store.rs Show resolved Hide resolved
parquet/src/arrow/async_writer/store.rs Outdated Show resolved Hide resolved
parquet/src/arrow/async_writer/store.rs Outdated Show resolved Hide resolved
parquet/src/arrow/async_writer/store.rs Outdated Show resolved Hide resolved
Xuanwo and others added 4 commits August 3, 2024 11:33
Signed-off-by: Xuanwo <[email protected]>
Signed-off-by: Xuanwo <[email protected]>
@Xuanwo
Copy link
Member Author

Xuanwo commented Aug 3, 2024

Hi, @alamb, I have addressed all comments. PTAL, thanks!

Signed-off-by: Xuanwo <[email protected]>
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @Xuanwo -- this looks great to me 🙏

/// [`ParquetObjectWriter`] for writing to parquet to [`ObjectStore`]
///
/// ```
/// # use arrow_array::{ArrayRef, Int64Array, RecordBatch};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️ -- this is great -- thank you

@alamb alamb merged commit 7f2d9ac into apache:master Aug 6, 2024
17 checks passed
@alamb
Copy link
Contributor

alamb commented Aug 6, 2024

Thanks again @Xuanwo

This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
parquet Changes to the parquet crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Impl AsyncFileWriter by object_store
2 participants