Change default Parquet writer settings to match arrow-rs (except for compression & statistics) #11558

wiedld · 2024-07-19T23:09:54Z

Which issue does this PR close?

Rationale for this change

Depending on how we construct our parquet writer settings, we are using a different set of default configurations as outlined here. This PR is the first step in reducing this misalignment.

What changes are included in this PR?

In order of commits:

define a set of datafusion constants based upon parquet (arrow-rs) defaults.
use the constants which are the same in each (a.k.a. no test changes)
use the constants which do change the datafusion defaults, and update tests to match
show the remaining defaults which datafusion is not yet using (a.k.a. what still doesn't match)

Are these changes tested?

Yes.

Are there any user-facing changes?

No, except if anyone is relying upon this configuration misalignment in their code.

… without the parquet feature enabled

…nd arrow-rs

wiedld · 2024-07-19T23:11:33Z

datafusion/common/src/config.rs

+    #[allow(dead_code)]
+    /// Default value for [`parquet::WriterProperties::statistics_enabled`]
+    pub const DEFAULT_STATISTICS_ENABLED: Option<&str> = Some("page");
+    #[allow(dead_code)]
+    /// Default value for [`parquet::BloomFilterProperties::fpp`]
+    pub const DEFAULT_BLOOM_FILTER_FPP: Option<f64> =
+        Some(props::DEFAULT_BLOOM_FILTER_FPP);
+    #[allow(dead_code)]
+    /// Default value for [`parquet::BloomFilterProperties::ndv`]
+    pub const DEFAULT_BLOOM_FILTER_NDV: Option<u64> =
+        Some(props::DEFAULT_BLOOM_FILTER_NDV);


These will be addressed in the next PR.

wiedld · 2024-07-19T23:19:14Z

datafusion/common/src/config.rs

+    #[allow(dead_code)]
+    /// Default value for [`parquet::WriterProperties::compression`]
+    pub const DEFAULT_COMPRESSION: Compression = Compression::UNCOMPRESSED;


Arrow-rs parquet's default is uncompressed. Whereas the datafusion default is zstd(3).

I think it makes sense for datafusion's default to be compressed, which argues to change the other default instead. But I'm unclear as to why the parquet (arrow-rs) default is uncompressed. Is this leftover? Or is intentional for the ArrowWriterOptions?

We discussed this in #7692 and decided that DataFusion would have parquet compression by default. I am not sure if arrow-rs has different factors that make uncompressed by default preferable.

Ok. Thank you.

I think there is a tension between raw read performance (uncompressed) and storage efficiency (compressed). @tustvold I think preferred arrow-rs to be on the read performance side of the equation

…writer settings

…further discussion

alamb

Thank you @wiedld

I changed the title of this PR to more accurately reflect its contents / effects for others

In general I think DataFusion should be following the arrow writer's default values so I think this is a good step forward.

I left a suggestion that I think would dramatically simplify this PR and reduce its size, but I think we could also do it as a follow on PR

datafusion/common/src/config.rs

alamb · 2024-07-20T11:33:00Z

cc @devinjdangelo -- I don't recall any reason the defaults for the parquet writer in datafusion were meant to be different than the defaults in parquet-rs. Do you?

devinjdangelo · 2024-07-20T13:05:04Z

cc @devinjdangelo -- I don't recall any reason the defaults for the parquet writer in datafusion were meant to be different than the defaults in parquet-rs. Do you?

The only default we changed intentionally after discussion was compression. I believe all others were intended to inherit the arrow-rs default.

alamb · 2024-07-21T11:08:21Z

The only default we changed intentionally after discussion was compression. I believe all others were intended to inherit the arrow-rs default.

@wiedld could you make sure this rationale is documented somewhere (comments and the option's description?)

…rquet feature, instead rely upon regression testing

alamb

Thank you @wiedld -- I think this PR nicely improves the system's behavior and is clear what has changed. ❤️

cc @hveiga

docs/source/user-guide/configs.md

alamb · 2024-07-22T11:56:41Z

datafusion/common/src/file_options/parquet_writer.rs

        // Confirm all other settings are equal.
        // First resolve the known discrepancies, (set as the same).
        // TODO: once we fix the above mis-matches, we should be able to remove this.
        let mut from_extern_parquet =
            session_config_from_writer_props(&default_writer_props);
        from_extern_parquet.global.compression = Some("zstd(3)".into());
-        from_extern_parquet.global.data_page_row_count_limit = usize::MAX;
-        from_extern_parquet.global.column_index_truncate_length = None;
-        from_extern_parquet.global.dictionary_enabled = None;
        from_extern_parquet.global.statistics_enabled = None;


Is there any rationale for not setting the statistics value to the same as in the arrow-rs writer?

The statistics_enabled and the bloom filter setting will require a bit more updates to do. In order to keep the diff small, (and more easily reviewable), I was going to do in a follow up PR. Is that ok @alamb ?

SOunds good to me 👍 thank you

alamb · 2024-07-22T11:57:23Z

I restarted the failed pyarrow check as it seemed to be an infrastructure problem

alamb · 2024-07-22T17:41:14Z

I'll plan to merge this tomorrow unless there are any other comments

alamb · 2024-07-23T17:56:53Z

Thanks again @wiedld and @devinjdangelo

wiedld added 2 commits July 19, 2024 15:36

feat(11367): define the default session's writer props, both with and…

8d6fb7f

… without the parquet feature enabled

feat(11367): map the defaults that already align between datafusion a…

01bc13e

…nd arrow-rs

wiedld commented Jul 19, 2024

View reviewed changes

wiedld added 2 commits July 19, 2024 16:39

feat(11367): move datafusion to use extern parquet defaults for most …

6883c21

…writer settings

chore(11367): denote what are the remaining misaligned defaults, for …

4899a00

…further discussion

wiedld force-pushed the 11367/update-defaults branch from 4828be1 to d863f01 Compare July 19, 2024 23:39

github-actions bot added documentation Improvements or additions to documentation sqllogictest SQL Logic Tests (.slt) labels Jul 19, 2024

wiedld added 2 commits July 19, 2024 16:47

chore: fix doc links

ac9969b

chore: update the readme demonstrating the default config setting

38124ba

wiedld force-pushed the 11367/update-defaults branch from d863f01 to 38124ba Compare July 19, 2024 23:47

wiedld marked this pull request as ready for review July 20, 2024 00:56

alamb changed the title ~~Minor: have datafusion use the same defaults as arrow-rs's parquet (except for compression & statistics)~~ Change default Parquet writer settings to match arrow-rs (except for compression & statistics) Jul 20, 2024

alamb approved these changes Jul 20, 2024

View reviewed changes

datafusion/common/src/config.rs Outdated Show resolved Hide resolved

wiedld force-pushed the 11367/update-defaults branch from 3bd55de to 3e85dc4 Compare July 22, 2024 03:08

refactor(11367): remove layer of indirection using the conditional pa…

d972416

…rquet feature, instead rely upon regression testing

wiedld force-pushed the 11367/update-defaults branch from 3e85dc4 to d972416 Compare July 22, 2024 03:21

alamb approved these changes Jul 22, 2024

View reviewed changes

docs/source/user-guide/configs.md Show resolved Hide resolved

alamb reviewed Jul 22, 2024

View reviewed changes

alamb merged commit 9f74dcc into apache:main Jul 23, 2024
25 checks passed

alamb deleted the 11367/update-defaults branch July 23, 2024 17:56

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change default Parquet writer settings to match arrow-rs (except for compression & statistics) #11558

Change default Parquet writer settings to match arrow-rs (except for compression & statistics) #11558

wiedld commented Jul 19, 2024 •

edited

Loading

wiedld Jul 19, 2024

wiedld Jul 19, 2024 •

edited

Loading

devinjdangelo Jul 20, 2024 •

edited

Loading

wiedld Jul 22, 2024

alamb Jul 23, 2024

alamb left a comment

alamb commented Jul 20, 2024

devinjdangelo commented Jul 20, 2024

alamb commented Jul 21, 2024

alamb left a comment

alamb Jul 22, 2024

wiedld Jul 22, 2024 •

edited

Loading

alamb Jul 22, 2024

alamb commented Jul 22, 2024

alamb commented Jul 22, 2024

alamb commented Jul 23, 2024

Change default Parquet writer settings to match arrow-rs (except for compression & statistics) #11558

Change default Parquet writer settings to match arrow-rs (except for compression & statistics) #11558

Conversation

wiedld commented Jul 19, 2024 • edited Loading

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

wiedld Jul 19, 2024

Choose a reason for hiding this comment

wiedld Jul 19, 2024 • edited Loading

Choose a reason for hiding this comment

devinjdangelo Jul 20, 2024 • edited Loading

Choose a reason for hiding this comment

wiedld Jul 22, 2024

Choose a reason for hiding this comment

alamb Jul 23, 2024

Choose a reason for hiding this comment

alamb left a comment

Choose a reason for hiding this comment

alamb commented Jul 20, 2024

devinjdangelo commented Jul 20, 2024

alamb commented Jul 21, 2024

alamb left a comment

Choose a reason for hiding this comment

alamb Jul 22, 2024

Choose a reason for hiding this comment

wiedld Jul 22, 2024 • edited Loading

Choose a reason for hiding this comment

alamb Jul 22, 2024

Choose a reason for hiding this comment

alamb commented Jul 22, 2024

alamb commented Jul 22, 2024

alamb commented Jul 23, 2024

wiedld commented Jul 19, 2024 •

edited

Loading

wiedld Jul 19, 2024 •

edited

Loading

devinjdangelo Jul 20, 2024 •

edited

Loading

wiedld Jul 22, 2024 •

edited

Loading