Add copy-on-write support for Iceberg DELETE, UPDATE, and MERGE operations by nookcreed · Pull Request #27747 · trinodb/trino

nookcreed · 2025-12-23T23:47:26Z

Fixes: #26161
Addresses design considerations from: #17272

Description

This change implements configurable copy-on-write (CoW) mode for row-level operations in Iceberg tables, giving users control over the read vs write performance trade-off.

Motivation:
Iceberg supports two approaches for row-level modifications:

Merge-on-Read (MoR): Fast writes, slower reads (current Trino behavior)
Copy-on-Write (CoW): Slower writes, fast reads (previously unsupported)

Users with read-heavy workloads on frequently updated tables have been requesting CoW support to eliminate read-time overhead of merging delete files with data files. This is particularly valuable for:

Analytics workloads with frequent small updates
Tables with high read/write ratios
Use cases requiring predictable read performance

Implementation:
This change adds three new table properties to control write mode per operation type:

write_delete_mode: 'merge-on-read' (default) or 'copy-on-write'
write_update_mode: 'merge-on-read' (default) or 'copy-on-write'
write_merge_mode: 'merge-on-read' (default) or 'copy-on-write'

Key changes:

UpdateKind tracking: Added UpdateKind enum (DELETE/UPDATE/MERGE) to IcebergTableHandle to track operation type throughout query lifecycle
- applyDelete() sets UpdateKind.DELETE
- getUpdateLayout() sets UpdateKind.UPDATE
- beginMerge() sets UpdateKind.MERGE
UpdateMode resolution: Added UpdateMode enum (MERGE_ON_READ/COPY_ON_WRITE) to resolve write mode from table properties based on UpdateKind
CoW DELETE implementation: Modified finishWrite() to detect CoW DELETE operations and delegate to new rewriteDataFilesForCowDelete() method
- Reads manifests to locate DataFile objects for files to delete
- For each affected data file:
  - Reads position deletes from delete files into Roaring64Bitmap
  - Opens original data file via IcebergPageSourceProvider * Filters deleted positions page-by-page using Block.copyPositions() * Writes filtered data to new file with proper metrics * Manages rollback lifecycle for cleanup on errors
- Uses Iceberg's RewriteFiles API for atomic commit
Resource management: Proper handling of rollback lifecycle
- Rollback handle kept until file successfully added to transaction
- Cleanup guaranteed via try-catch-finally blocks
- Suppressed exceptions preserve original error context
Optimizations:
- Direct manifest reading with early termination when all files found
- Efficient position delete filtering using Roaring64Bitmap
- Reuses existing PositionDeleteFilter implementation for consistency
Safety features:
- Snapshot isolation via Iceberg's validation mechanisms
- Null checks for empty tables
- Missing file validation before commit
- Format version check (requires v2 for row-level operations)

Testing:

Added TestIcebergCopyOnWriteOperations with 17 comprehensive tests
- Basic operations (DELETE, UPDATE, MERGE)
- Partitioned tables
- Large batch operations (1000-5000 rows)
- Performance benchmarks with metrics collection
- Error cases (empty tables, format v1, etc.)
Added TestIcebergCopyOnWriteDeleteOperations with 15 unit tests
- File rewriting with position deletes
- Error handling (IO exceptions, missing files, etc.)
- Resource cleanup verification
- Edge cases (empty deletes, equality deletes, etc.)
Added TestIcebergCopyOnWriteIntegration with 5 integration tests
- Resource cleanup on failure
- Concurrent operations
- Snapshot isolation
- Conflict resolution
Updated 8 test files to reflect new IcebergMetadata constructor signature

Documentation:

Added comprehensive COPY_ON_WRITE_README.md with:
- Feature overview and motivation
- Configuration examples
- Usage patterns and recommendations
- Performance characteristics (CoW vs MoR comparison)
- Implementation deep dive with code examples
- Troubleshooting guide
- Known limitations and future enhancements

Limitations:

CoW DELETE currently only supports position deletes
Equality deletes documented as future enhancement
Requires Iceberg format version 2 or higher

Additional context and related issues

Documented in README

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
(x ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

## Section
* Fix some things. ({issue}`issuenumber`)

cla-bot · 2025-12-23T23:47:29Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2025-12-23T23:49:26Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2025-12-23T23:54:33Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2025-12-24T02:00:00Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2025-12-30T05:12:00Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2025-12-30T05:23:12Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2025-12-30T05:41:47Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2025-12-30T06:06:08Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2025-12-30T06:25:09Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2025-12-30T06:28:18Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2025-12-30T07:04:13Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2026-01-04T02:57:24Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2026-01-04T08:33:54Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2026-01-04T18:13:32Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2026-01-04T18:38:36Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2026-01-04T19:34:44Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2026-01-04T20:01:54Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2026-01-04T21:39:33Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2026-01-04T23:21:33Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2026-01-05T00:16:23Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2026-01-05T02:12:11Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2026-01-05T02:12:44Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2026-01-05T02:13:01Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2026-01-05T02:52:06Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

…tions Fix NullPointerException in BaseTrinoCatalogTest This commit fixes NullPointerException errors occurring in BaseTrinoCatalogTest by providing non-null OrcReaderOptions and ParquetReaderOptions objects in the test methods testNonLowercaseNamespace and testSchemaWithInvalidProperties. Instead of passing null values, we now use default instances. Fix tests Fix tests Empty commit

cla-bot · 2026-01-05T02:56:12Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

ebyhr · 2026-01-05T03:04:32Z

Duplicate of #27844

github-actions bot added the iceberg Iceberg connector label Dec 23, 2025

nookcreed force-pushed the master branch from fbec56b to b7e1ab8 Compare December 23, 2025 23:54

raunaqmorarka requested review from chenjian2664 and ebyhr December 24, 2025 05:15

nookcreed marked this pull request as draft December 24, 2025 05:19

nookcreed force-pushed the master branch from 2766fda to 06721a8 Compare December 30, 2025 05:23

nookcreed marked this pull request as ready for review December 30, 2025 05:25

nookcreed marked this pull request as draft December 30, 2025 05:25

nookcreed force-pushed the master branch from 06721a8 to 2db492b Compare December 30, 2025 05:41

nookcreed force-pushed the master branch from 0a1ef78 to fbc1a00 Compare January 4, 2026 08:33

nookcreed force-pushed the master branch from b8ddc75 to ed9a24b Compare January 4, 2026 18:38

nookcreed force-pushed the master branch from ae408fc to 21b5ecd Compare January 4, 2026 20:01

nookcreed force-pushed the master branch from 21b5ecd to 6492c09 Compare January 4, 2026 21:39

nookcreed force-pushed the master branch from 6492c09 to bb1f7fc Compare January 4, 2026 23:21

nookcreed force-pushed the master branch from 4f9cd3c to 6baed41 Compare January 5, 2026 02:12

nookcreed force-pushed the master branch from 6baed41 to 69ea2b9 Compare January 5, 2026 02:12

nookcreed force-pushed the master branch from 1ba9082 to bee34be Compare January 5, 2026 02:56

ebyhr marked this as a duplicate of #27844 Jan 5, 2026

ebyhr closed this Jan 5, 2026

Conversation

nookcreed commented Dec 23, 2025

Description

Additional context and related issues

Release notes

Uh oh!

cla-bot bot commented Dec 23, 2025

Uh oh!

cla-bot bot commented Dec 23, 2025

Uh oh!

cla-bot bot commented Dec 23, 2025

Uh oh!

cla-bot bot commented Dec 24, 2025

Uh oh!

cla-bot bot commented Dec 30, 2025

Uh oh!

cla-bot bot commented Dec 30, 2025

Uh oh!

cla-bot bot commented Dec 30, 2025

Uh oh!

cla-bot bot commented Dec 30, 2025

Uh oh!

cla-bot bot commented Dec 30, 2025

Uh oh!

cla-bot bot commented Dec 30, 2025

Uh oh!

cla-bot bot commented Dec 30, 2025

Uh oh!

cla-bot bot commented Jan 4, 2026

Uh oh!

cla-bot bot commented Jan 4, 2026

Uh oh!

cla-bot bot commented Jan 4, 2026

Uh oh!

cla-bot bot commented Jan 4, 2026

Uh oh!

cla-bot bot commented Jan 4, 2026

Uh oh!

cla-bot bot commented Jan 4, 2026

Uh oh!

cla-bot bot commented Jan 4, 2026

Uh oh!

cla-bot bot commented Jan 4, 2026

Uh oh!

cla-bot bot commented Jan 5, 2026

Uh oh!

cla-bot bot commented Jan 5, 2026

Uh oh!

cla-bot bot commented Jan 5, 2026

Uh oh!

cla-bot bot commented Jan 5, 2026

Uh oh!

cla-bot bot commented Jan 5, 2026

Uh oh!

cla-bot bot commented Jan 5, 2026

Uh oh!

ebyhr commented Jan 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants