Flink: Add FlinkWriterFactory #2924
Conversation
|
I need a writer factory for Flink so that I can introduce new task writers. |
| builder.createWriterFunc((iSchema, typDesc) -> FlinkOrcWriter.buildWriter(dataFlinkType(), iSchema)); | ||
| } | ||
|
|
||
| private RowType dataFlinkType() { |
There was a problem hiding this comment.
should we move this logic to constructor?
There was a problem hiding this comment.
or maybe move this part to the Builder#build() method
There was a problem hiding this comment.
These are optional and may not be set/needed unless we actually write data files.
Pure DELETEs will not need it, for example.
| return this; | ||
| } | ||
|
|
||
| Builder dataFlinkType(RowType newDataFlinkType) { |
There was a problem hiding this comment.
Is this an optional param? In the other comment, I saw we are converting Schema (above) to RowType if dataFlinkType is not set.
Maybe add Javadoc to mark all the *FlinkType methods/params as optional.
There was a problem hiding this comment.
Added docs to optional methods.
| } | ||
|
|
||
| private RowType positionDeleteFlinkType() { | ||
| if (positionDeleteFlinkType == null) { |
There was a problem hiding this comment.
This one is similar to data files. The value is optional and may not be needed.
|
Thanks for reviewing, @openinx @stevenzwu! |
3 participants
This PR adds
FlinkWriterFactorythat extendsBaseWriterFactoryintroduced in #2873.