-
Notifications
You must be signed in to change notification settings - Fork 480
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ORC-1200: Extracting encryption setup logic from WriterImpl
#1156
base: main
Are you sure you want to change the base?
Conversation
Thank you for making a PR, @liujiawinds . |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's unclear to me why this is requested by Apache Flink community. For example,
- Do you have some documentation or reference in Apache Flink community?
- Is this Apache ORC design issue?
- Is this request aligned with other encryption handling in Apache Flink community (like Apache Parquet Modular Encryption handling)?
In general, branch-1.8
is already in feature freeze mode. I don't think this is required at v1.8.0 (currently).
Also, cc @omalley because he is the main author of Encryption Feature.
Yes, first we must create a stream based
Yes, I think it is.
Parquet has a similar class named
It's ok to release in later versions. I will extract this part of code in flink. |
Thank you for the details. |
What should I do next for this pr? I don't quite understand the meaning of it being marked as |
Instead of commenting, you are supposed to revise your PR description with your previous comment's content because only the PR title and description becomes a commit message.
|
After rethinking about this PR, I removed the milestone. |
@dongjoon-hyun The PR description has been revised. |
What changes were proposed in this pull request?
Extracting the encryption setup logic as a tool class.
Why are the changes needed?
Because of flink's ORC writer is based of stream, we must create a stream based PhysicalFsWriter before WriterImpl initial. WriterImpl
Then, You can look into flink orc writer implementation from OrcBulkWriterFactory.
Given the above, now I want to pass encryption settings to PhysicalFsWriter before WriterImpl initail, but the encryption setup at WriterImpl is so tightly coupled that the encryption variant cannot be obtained externally.
How was this patch tested?
It doesn't introduce new features and passed all test cases.