Skip to content

Allow writers to control size of files generated #432

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rdblue merged 9 commits into from
Sep 24, 2019
Merged

Allow writers to control size of files generated #432

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
b5546d4
Allow writers to control size of files genenrated.
xabriel Aug 29, 2019
dcc777f
Add table property 'write.target-file-size'
xabriel Sep 3, 2019
560230a
Update docs
xabriel Sep 3, 2019
a5aff93
Address review comments on file name and checking for target size eve…
xabriel Sep 5, 2019
62dc280
Refactor Partitioned and UnpartitionedWriter to share common code in …
xabriel Sep 5, 2019
f45a317
Rename flag to target-file-size-bytes. Use PropertyUtil.
xabriel Sep 10, 2019
87c7907
Fix indentation of BaseWriter to match WriterFactory
xabriel Sep 10, 2019
d275322
Fix ParquetWriter#length() to account for currently buffered row group.
xabriel Sep 17, 2019
cce0d6a
Address latest review comments on Writer.java
xabriel Sep 23, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions core/src/main/java/org/apache/iceberg/TableProperties.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,4 +100,7 @@ private TableProperties() {}

public static final String WRITE_AUDIT_PUBLISH_ENABLED = "write.wap.enabled";
public static final String WRITE_AUDIT_PUBLISH_ENABLED_DEFAULT = "false";

public static final String WRITE_TARGET_FILE_SIZE = "write.target-file-size";
public static final long WRITE_TARGET_FILE_SIZE_DEFAULT = Long.MAX_VALUE;
}
2 changes: 2 additions & 0 deletions site/docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ Iceberg tables support table properties to configure table behavior, like the de
| write.metadata.compression-codec | none | Metadata compression codec; none or gzip |
| write.metadata.metrics.default | truncate(16) | Default metrics mode for all columns in the table; none, counts, truncate(length), or full |
| write.metadata.metrics.column.col1 | (not set) | Metrics mode for column 'col1' to allow per-column tuning; none, counts, truncate(length), or full |
| write.target-file-size | Long.MAX_VALUE | Controls the size of files generated to target about this many bytes. |
Comment thread
xabriel marked this conversation as resolved.
Outdated

### Table behavior properties

Expand Down Expand Up @@ -75,4 +76,5 @@ df.write
| Spark option | Default | Description |
| ------------ | -------------------------- | ------------------------------------------------------------ |
| write-format | Table write.format.default | File format to use for this write operation; parquet or avro |
| target-file-size | As per table property | Overrides this table's write.target-file-size |

Loading