Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Minor: Clarify docs on EnabledStatistics #5812

Merged
merged 3 commits into from
May 29, 2024
Merged

Minor: Clarify docs on EnabledStatistics #5812

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
9c64d51
Minor: Clarify docs on `EnabledStatistics`
alamb May 28, 2024
ad1891c
refine
alamb May 28, 2024
4dbd3c3
Update parquet/src/file/properties.rs
alamb May 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 20 additions & 4 deletions parquet/src/file/properties.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -669,14 +669,30 @@ impl WriterPropertiesBuilder {
}
}

/// Controls the level of statistics to be computed by the writer
/// Controls the level of statistics to be computed by the writer and stored in
/// the parquet file.
///
/// Enabling statistics makes the resulting Parquet file larger and requires
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is useful to help people understand the implications of these settings as well so I tried to clarify the tradeoff here

/// more time to read the parquet footer.
///
/// Statistics can be used to improve query performance by efficient row groups
alamb marked this conversation as resolved.
Show resolved Hide resolved
/// and pages during query execution if the query engine supports evaluating the
/// predicate using the statistics.
#[derive(Debug, Clone, Copy, Eq, PartialEq)]
pub enum EnabledStatistics {
/// Compute no statistics
/// Compute no statistics.
None,
/// Compute chunk-level statistics but not page-level
/// Compute column chunk-level statistics but not page-level.
///
/// Setting this option will store one set of statistics for each relevant
/// column for each row group. The more row groups written, the more
/// statistics will be stored.
Chunk,
/// Compute page-level and chunk-level statistics
/// Compute page-level and column chunk-level statistics.
///
/// Setting this option will store one set of statistics for each relevant
/// column for each page and row group. The more row groups and the more
/// pages written, the more statistics will be stored.
Page,
}

Expand Down
Loading