Move parquet-specific options to parquet::WriterOptions#10470
Move parquet-specific options to parquet::WriterOptions#10470pedroerp wants to merge 1 commit intofacebookincubator:mainfrom
Conversation
facebook-github-bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D59710079 |
✅ Deploy Preview for meta-velox canceled.
|
velox/dwio/common/Options.h
Outdated
There was a problem hiding this comment.
This needs to be concrete and we need to translate the format agnostic configurations here
There was a problem hiding this comment.
Why does it need to be concrete?
There was a problem hiding this comment.
Because it needs to translate format-agnostic options, and the subclasses will handle the format-specific ones
majetideepak
left a comment
majetideepak
left a comment
There was a problem hiding this comment.
@pedroerp thanks for taking this up. I have been wanting something like this for a while now. I have one comment.
pedroerp
force-pushed
the
export-D59710079
branch
from
e2aa9c8 to
1d789cf
Compare
|
This pull request was exported from Phabricator. Differential Revision: D59710079 |
|
This pull request was exported from Phabricator. Differential Revision: D59710079 |
pedroerp
force-pushed
the
export-D59710079
branch
from
1d789cf to
2d175d5
Compare
velox/dwio/parquet/writer/Writer.cpp
Outdated
There was a problem hiding this comment.
Can we have "hive.parquet.writer.timestamp_unit" and "hive.parquet.writer.timestamp-unit" as static constexpr const char* in parquet WriterOptions?
|
This pull request was exported from Phabricator. Differential Revision: D59710079 |
pedroerp
force-pushed
the
export-D59710079
branch
from
2d175d5 to
28a6043
Compare
|
This pull request was exported from Phabricator. Differential Revision: D59710079 |
pedroerp
force-pushed
the
export-D59710079
branch
from
28a6043 to
702fcd7
Compare
…bator#10470) Summary: Pull Request resolved: facebookincubator#10470 Moving parquet-specific options to the new parquet WriterOptions polymorphic type to remove file format specific configuration code in the general Hive connector. Reviewed By: Yuhta Differential Revision: D59710079
|
This pull request was exported from Phabricator. Differential Revision: D59710079 |
pedroerp
force-pushed
the
export-D59710079
branch
from
702fcd7 to
6f351a6
Compare
|
This pull request has been merged in 8156d7d. |
|
Conbench analyzed the 1 benchmark run on commit There were no benchmark performance regressions. 🎉 The full Conbench report has more details. |
|
@pedroerp Multiple CI jobs are failing with this merge. |
Ughh, looking at it now. |
Summary: Fixing Parquet writer test build broken in facebookincubator#10470 Differential Revision: D60066916
| // through insertTableHandle) | ||
| // 2. Otherwise, acquire user defined session properties. | ||
| // 3. Lastly, acquire general hive connector configs. | ||
| options->processSessionConfigs(*connectorSessionProperties); |
There was a problem hiding this comment.
Can we let Parquet writer to convert the common options to Parquet specific settings as dwrf does? Thanks!
6 participants
Summary:
Moving parquet-specific options to the new parquet WriterOptions
polymorphic type to remove file format specific configuration code in the
general Hive connector.
Differential Revision: D59710079