Refactor: more clearly delineate between `TableParquetOptions` and `ParquetWriterOptions` #11444

wiedld · 2024-07-12T20:22:38Z

Which issue does this PR close?

Is a cleanup before doing #11367.

This PR delineates btwn the session-level config vs the per-writer config.
~~It also adds tests to demonstrate how these two configurations get out-of-sync.~~ Tests in the followup PR.

Rationale for this change

We have two session-level configurations for parquet writes:
- TableParquetOptions and ParquetOptions.
We have two writer-level options for writer (a.k.a. props to be handed to a writer, per a single write action):
- WriterProperties and ParquetWriterOptions (which wraps WriterProperties).

At first, it looks like we should be able to get the writer props from either session-level config options. (In fact, we made exactly this suggestion ourselves.) But this is misleading; as the ParquetOptions has an incomplete set of all the config needed for the arrow writer (it's missing the kv_metadata & col-specific configurations).

What changes are included in this PR?

Instead, I've done some cleanup and added documentation to hopefully make the relationship more clear. Additionally, there's a new ParquetOptions::writer_props_from_global_opts() that does fulfill our suggestion -- while still making the difference clear.

Are these changes tested?

Yes.

Are there any user-facing changes?

A new ParquetOptions::writer_props_from_global_opts().

…s vs ParquetOptions vs WriterProperties

alamb · 2024-07-12T20:26:33Z

Possibly related: #11367

alamb

Thank you @wiedld. I think this PR is a good step towards #11367 (it almost does it all).

I had a few structural / naming suggestions but otherwise I think the basic ideas here

as the ParquetOptions has an incomplete set of all the config needed for the arrow writer (it's missing the kv_metadata & col-specific configurations).

I wonder if we should make ParquetOptions be consistent with the WriterOptions? It seems like missing kv_metadata and column overrides might be an oversight rather than an intentional design)

datafusion/common/src/config.rs

alamb · 2024-07-12T20:32:09Z

datafusion/common/src/config.rs

+            }
+        }
+
+        #[test]


Maybe we could also add a test here that described in #11367 -- that takes a default ParquetOptions, creates the ArrowWriterOption and then ensures the fields are the same as ArrowWriterOptions::default()

Added this^^ requested test in this commit, but did the equivalency assertion a bit differently. Parquet's WriterProperties and it's builder do not implement Eq or PartialEq.

Rather than asserting each prop independently based upon the getter, I elected to (a) explicitly define the props that do not match, and then (b) use the same identify function (session_config_from_writer_props) for the assertion.

I had to make a slight change to avoid a test regression in the sqllogictests.

That^^ change (to avoid test regression) also brings us further from the matching-default-settings between the extern parquet vs datafusion. I added more tests to demonstrate this divergence.

In the follow up PR, I plan to propose (and open for discussion) which defaults should be changed in datafusion vs parquet.

Per request, moving these into a follow up PR. (Make more reviewable.)

datafusion/common/src/config.rs

wiedld · 2024-07-13T03:27:07Z

I wonder if we should make ParquetOptions be consistent with the WriterOptions? It seems like missing kv_metadata and column overrides might be an oversight rather than an intentional design)

The ParquetOptions are explicitly stated as the global part of TableParquetOptions.

datafusion/datafusion/common/src/config.rs

Lines 1395 to 1397 in 8f8df07

    
           pub struct TableParquetOptions { 
        
               /// Global Parquet options that propagates to all columns. 
        
               pub global: ParquetOptions,

Is there any reason to have the global ParquetOptions, without the TableParquetOptions?

the two additional fields are kv_metadata a col-specific hashmap, both of those can be default/empty.
proposal: should we make ParquetOptions be private to TableParquetOptions?

wiedld · 2024-07-13T03:52:17Z

The SessionState contains multiple copies of the ParquetOptions:

(⊃ denotes "contained within")
SessionState.config ⊃ SessionConfig ⊃ ConfigOptions ⊃ ExecutionOptions ⊃ ParquetOptions
SessionState.file_formats ⊃ ParquetFormat ⊃ TableParquetOptions ⊃ ParquetOptions
SessionState.table_options ⊃ TableOptions ⊃ TableParquetOptions ⊃ ParquetOptions

The SessionState.config is used as a default for SessionState.table_options.

per the above suggestion, we could replace this with TableParquetOptions.

.

But what is the intended use case for table_options vs file_formats?

From what I can see, it appears that the table_options duplicates the information within the file_formats (e.g. CsvFormat ⊃ CsvOptions, JsonFormat ⊃ JsonOptions, etc).
table_options does have two unique fields: the assumed-file-type and extensions. But the rest is all file_formats.
file_formats was added 2 weeks ago.
proposal: unify to a single file_formats (with csv, json, parquet factories added) to be contained within the TableOptions.

alamb · 2024-07-14T15:32:53Z

But what is the intended use case for table_options vs file_formats?

I am not entirely sure. Given it is not clear from the code, what I would normally do in these circumstances is make a new PR with documentation to those structures and try and explain the differences

It would likely take me a while, but the process of trying to document the differences would force me to do the code archeology / spelunking to figure it out, and then once I did I would have a good artifact (new docs) that would

Ensure I didn't forget the answer
Help others who probably have the same questions
A vehicle for others to help correct and improve

I of course would love to help review such a PR 🎣

wiedld · 2024-07-15T17:29:45Z

ok, I'm going to:

clean up and scope this PR to the current work (delineating writer options vs session config)
include in this current work, making sure that the writer options (in arrow) have the same defaults as the session configs (in datafusion).
- Inconsistent value for data_page_max_rows setting in DataFusion ParquetOptions and in ArrowWriterOptions #11367

Then in separate tickets, and separate followup PRs, will handle these two proposals: cleanup ParquetOptions vs TableParquetOptions, and the proposed cleanup of file_formats.

alamb · 2024-07-15T18:11:38Z

Thanks @wiedld

A suggestion that I think will help speed up reviews / merging is to try and keep the PRs smaller (like for example make one PR to add docs, and other PRs to add new apis (like conversion from ParquetOptions to WriterOptions, etc)

…ubmodule, leaving only the config options in the config module.

… column configuration when none exists

…to be left as None

…ired to avoid test regression), result in different default behavior than parquet crate

datafusion/common/src/config.rs

…llowup PR

alamb

Thank you @wiedld

I think this is a step forward and improves the situation

I had a few API suggestions that I think would make it even better but I don't think they are required

datafusion/common/src/file_options/parquet_writer.rs

alamb · 2024-07-17T19:08:51Z

Thanks @wiedld

…arquetWriterOptions` (apache#11444) * refactor: make more explicit the relationship btwn TableParquetOptions vs ParquetOptions vs WriterProperties * test: demonstrate the relationship btwn session configs and writer props * refactor: move parquet-format specific functionality to the parquet submodule, leaving only the config options in the config module. * test: update test fixtures to use the ParquetOptions::default * test: update test helper session_config_from_writer_props, to not add column configuration when none exists * test(11367): write test to demonstrate issue 11367 * fix: existing sqllogictests require specific ParquetOptions settings to be left as None * test(11367): demonstrate how the require bloom filter defaults, (required to avoid test regression), result in different default behavior than parquet crate * chore: make more reviewable, by pulling tests for issue 11367 into followup PR * refactor: move all parquet-associated features into parquet-writer mod * chore: better function naming convention

wiedld added 2 commits July 12, 2024 13:02

refactor: make more explicit the relationship btwn TableParquetOption…

15fea6d

…s vs ParquetOptions vs WriterProperties

test: demonstrate the relationship btwn session configs and writer props

15a9fec

alamb reviewed Jul 12, 2024

View reviewed changes

alamb mentioned this pull request Jul 12, 2024

Inconsistent value for data_page_max_rows setting in DataFusion ParquetOptions and in ArrowWriterOptions #11367

Closed

wiedld added 4 commits July 15, 2024 14:53

refactor: move parquet-format specific functionality to the parquet s…

f67c189

…ubmodule, leaving only the config options in the config module.

test: update test fixtures to use the ParquetOptions::default

e9f1427

test: update test helper session_config_from_writer_props, to not add…

e693d43

… column configuration when none exists

test(11367): write test to demonstrate issue 11367

9c43625

wiedld force-pushed the refactor/session-options-ve-writer-props branch from a484974 to 9c43625 Compare July 16, 2024 01:57

wiedld added 2 commits July 15, 2024 20:37

fix: existing sqllogictests require specific ParquetOptions settings …

ad9f695

…to be left as None

test(11367): demonstrate how the require bloom filter defaults, (requ…

4f43390

…ired to avoid test regression), result in different default behavior than parquet crate

wiedld force-pushed the refactor/session-options-ve-writer-props branch from ab950c5 to 4f43390 Compare July 16, 2024 03:47

alamb reviewed Jul 16, 2024

View reviewed changes

datafusion/common/src/config.rs Outdated Show resolved Hide resolved

wiedld added 2 commits July 16, 2024 14:03

chore: make more reviewable, by pulling tests for issue 11367 into fo…

5a458fe

…llowup PR

refactor: move all parquet-associated features into parquet-writer mod

d5978d9

wiedld marked this pull request as ready for review July 16, 2024 22:25

alamb changed the title ~~Refactor: more clearly delineate btwn writer options vs session configuration~~ Refactor: more clearly delineate between writer options vs session configuration Jul 17, 2024

alamb changed the title ~~Refactor: more clearly delineate between writer options vs session configuration~~ Refactor: more clearly delineate between TableParquetOptions and ParquetWriterOptions Jul 17, 2024

alamb approved these changes Jul 17, 2024

View reviewed changes

datafusion/common/src/file_options/parquet_writer.rs Outdated Show resolved Hide resolved

datafusion/common/src/file_options/parquet_writer.rs Show resolved Hide resolved

This was referenced Jul 17, 2024

Minor: Clarify which parquet options are used for reading/writing #11511

Merged

Rename ColumnOptions to ParquetColumnOptions #11512

Merged

fix(11367): parquet writer defaults influxdata/arrow-datafusion#34

Closed

chore: better function naming convention

d60e07e

alamb merged commit c95556d into apache:main Jul 17, 2024
23 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor: more clearly delineate between `TableParquetOptions` and `ParquetWriterOptions` #11444

Refactor: more clearly delineate between `TableParquetOptions` and `ParquetWriterOptions` #11444

wiedld commented Jul 12, 2024 •

edited

Loading

alamb commented Jul 12, 2024

alamb left a comment

alamb Jul 12, 2024

wiedld Jul 16, 2024 •

edited

Loading

wiedld Jul 16, 2024 •

edited

Loading

wiedld Jul 16, 2024

wiedld Jul 16, 2024 •

edited

Loading

wiedld commented Jul 13, 2024 •

edited

Loading

wiedld commented Jul 13, 2024 •

edited

Loading

alamb commented Jul 14, 2024

wiedld commented Jul 15, 2024 •

edited

Loading

alamb commented Jul 15, 2024 •

edited

Loading

alamb left a comment

alamb commented Jul 17, 2024

+                          }
+                      }
+                      #[test]

Refactor: more clearly delineate between TableParquetOptions and ParquetWriterOptions #11444

Refactor: more clearly delineate between TableParquetOptions and ParquetWriterOptions #11444

Conversation

wiedld commented Jul 12, 2024 • edited Loading

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

alamb commented Jul 12, 2024

alamb left a comment

Choose a reason for hiding this comment

alamb Jul 12, 2024

Choose a reason for hiding this comment

wiedld Jul 16, 2024 • edited Loading

Choose a reason for hiding this comment

wiedld Jul 16, 2024 • edited Loading

Choose a reason for hiding this comment

wiedld Jul 16, 2024

Choose a reason for hiding this comment

wiedld Jul 16, 2024 • edited Loading

Choose a reason for hiding this comment

wiedld commented Jul 13, 2024 • edited Loading

wiedld commented Jul 13, 2024 • edited Loading

alamb commented Jul 14, 2024

wiedld commented Jul 15, 2024 • edited Loading

alamb commented Jul 15, 2024 • edited Loading

alamb left a comment

Choose a reason for hiding this comment

alamb commented Jul 17, 2024

Refactor: more clearly delineate between `TableParquetOptions` and `ParquetWriterOptions` #11444

Refactor: more clearly delineate between `TableParquetOptions` and `ParquetWriterOptions` #11444

wiedld commented Jul 12, 2024 •

edited

Loading

wiedld Jul 16, 2024 •

edited

Loading

wiedld Jul 16, 2024 •

edited

Loading

wiedld Jul 16, 2024 •

edited

Loading

wiedld commented Jul 13, 2024 •

edited

Loading

wiedld commented Jul 13, 2024 •

edited

Loading

wiedld commented Jul 15, 2024 •

edited

Loading

alamb commented Jul 15, 2024 •

edited

Loading