Add support for concurrent write on Iceberg transformed column #24160

pajaks · 2024-11-18T11:08:54Z

Description

Add concurrent write to scenarios like:

-- create test table in iceberg
create table all_defaults_partitioned
with (
  partitioning = array['month(shipdate)']
)
as select * from tpch.sf1000.lineitem

-- first session
update all_defaults_partitioned
set orderkey = 654
where shipdate = date '1995-01-01'

-- second session 
update all_defaults_partitioned
set orderkey = 765
where shipdate = date '1996-12-01'

Release notes

## Iceberg
* Fix failure when writing concurrently with [transformed partition](https://iceberg.apache.org/spec/#partition-transforms) columns

findinpath · 2024-11-20T12:36:57Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

@@ -2740,7 +2740,8 @@ private void finishWrite(ConnectorSession session, IcebergTableHandle table, Col

        RowDelta rowDelta = transaction.newRowDelta();
        table.getSnapshotId().map(icebergTable::snapshot).ifPresent(s -> rowDelta.validateFromSnapshot(s.snapshotId()));
-        TupleDomain<IcebergColumnHandle> dataColumnPredicate = table.getEnforcedPredicate().filter((column, domain) -> !isMetadataColumnId(column.getId()));
+        TupleDomain<IcebergColumnHandle> dataColumnPredicate = table.getEnforcedPredicate().intersect(table.getUnenforcedPredicate())


Why do we intersect enforced with the unenforced predicate?
They are rather unrelated.

My understanding is that both carriers information about predicate.
The difference is that taken from @raunaqmorarka explanation:

enforced is the part of the predicate which is guaranteed to be satisfied by the connector, so the engine will not apply it on it's side. unenforced is the part of the predicate which connector cannot guarantee even if it is able to use it to reduce output, so the engine will apply it on the connector output

@raunaqmorarka Correct me if I'm wrong here

{2:part:date=[ SortedRangeSet[type=date, ranges=1, {[2024-01-01]}] ]}

I'm checking io.trino.plugin.iceberg.TestIcebergLocalConcurrentWritesTest#testConcurrentUpdateWithOverlappingPartitionTransformation (BTW really cool that we have this battery of new concurrency tests)

It's unclear to me why the partition predicate is not an "enforced predicate" while debugging your code.

Adding such predicates for transformed columns as enforced, would mean pushdowns of those values and connector would need to filter files and rows also during reading from such table as well. It's not supported right now and would require more work.

They are rather unrelated.

We already join those in IcebergSplitSource

trino/plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergSplitSource.java

Line 259 in 61a79dd

TupleDomain<IcebergColumnHandle> effectivePredicate = TupleDomain.intersect(

We are able to prune files in IcebergSplitSource based on unenforced filter, but not rows (hence "unenforced"). Filters on identity partitioned columns are enforced at the row level due to the data being partitioned by value and PartitionConstraintMatcher in IcebergSplitSource.
I think it's valid to intersect unenforced filter here because org.apache.iceberg.MergingSnapshotProducer is internally checking whether any files matching given filter have been added to the table.

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

raunaqmorarka · 2024-11-25T06:55:45Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

@@ -2740,7 +2740,8 @@ private void finishWrite(ConnectorSession session, IcebergTableHandle table, Col

        RowDelta rowDelta = transaction.newRowDelta();
        table.getSnapshotId().map(icebergTable::snapshot).ifPresent(s -> rowDelta.validateFromSnapshot(s.snapshotId()));
-        TupleDomain<IcebergColumnHandle> dataColumnPredicate = table.getEnforcedPredicate().filter((column, domain) -> !isMetadataColumnId(column.getId()));
+        TupleDomain<IcebergColumnHandle> dataColumnPredicate = table.getEnforcedPredicate().intersect(table.getUnenforcedPredicate())


We are able to prune files in IcebergSplitSource based on unenforced filter, but not rows (hence "unenforced"). Filters on identity partitioned columns are enforced at the row level due to the data being partitioned by value and PartitionConstraintMatcher in IcebergSplitSource.
I think it's valid to intersect unenforced filter here because org.apache.iceberg.MergingSnapshotProducer is internally checking whether any files matching given filter have been added to the table.

...rino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergLocalConcurrentWritesTest.java

raunaqmorarka

lgtm
But I would like a second opinion on it
@alexjo2144 @findepi @losipiuk @hashhar @electrum are any of you able to weigh in here ?

...in/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergLocalConcurrentWrites.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

findinpath · 2024-11-27T15:51:56Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

+        TupleDomain<IcebergColumnHandle> convertibleUnenforcedPredicate = table.getUnenforcedPredicate().filter((_, domain) -> isConvertableToIcebergExpression(domain));
+        TupleDomain<IcebergColumnHandle> effectivePredicate = dataColumnPredicate.intersect(convertibleUnenforcedPredicate);
+        if (!effectivePredicate.isAll()) {
+            rowDelta.conflictDetectionFilter(toIcebergExpression(effectivePredicate));


Pretty cool @pajaks 🎉

Outstanding test coverage.

pajaks · 2024-11-28T14:00:53Z

@ebyhr @raunaqmorarka Can this be merged?

cla-bot bot added the cla-signed label Nov 18, 2024

github-actions bot added the iceberg Iceberg connector label Nov 18, 2024

pajaks force-pushed the pajaks/iceberg_partition_concurrent_writes branch from 0f00cdd to ff872d7 Compare November 18, 2024 11:10

findinpath reviewed Nov 20, 2024

View reviewed changes

pajaks force-pushed the pajaks/iceberg_partition_concurrent_writes branch 2 times, most recently from 8da17cc to c9670a6 Compare November 21, 2024 10:40

pajaks marked this pull request as ready for review November 21, 2024 10:46

pajaks requested review from raunaqmorarka and ebyhr November 21, 2024 10:46

raunaqmorarka reviewed Nov 25, 2024

View reviewed changes

raunaqmorarka requested a review from alexjo2144 November 25, 2024 06:56

pajaks force-pushed the pajaks/iceberg_partition_concurrent_writes branch from c9670a6 to 7623816 Compare November 25, 2024 11:44

pajaks requested a review from raunaqmorarka November 25, 2024 11:45

ebyhr reviewed Nov 26, 2024

View reviewed changes

...rino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergLocalConcurrentWritesTest.java Outdated Show resolved Hide resolved

pajaks force-pushed the pajaks/iceberg_partition_concurrent_writes branch from 7623816 to 5df5282 Compare November 26, 2024 15:07

pajaks requested a review from ebyhr November 26, 2024 15:08

Add Iceberg concurrent write tests

4be393f

pajaks force-pushed the pajaks/iceberg_partition_concurrent_writes branch from 5df5282 to bb147d6 Compare November 27, 2024 08:28

raunaqmorarka reviewed Nov 27, 2024

View reviewed changes

...in/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergLocalConcurrentWrites.java Outdated Show resolved Hide resolved

...in/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergLocalConcurrentWrites.java Outdated Show resolved Hide resolved

pajaks force-pushed the pajaks/iceberg_partition_concurrent_writes branch from bb147d6 to 8daa87b Compare November 27, 2024 12:43

ebyhr approved these changes Nov 27, 2024

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java Outdated Show resolved Hide resolved

Add support for concurrent write on Iceberg transformed column

36c8476

pajaks force-pushed the pajaks/iceberg_partition_concurrent_writes branch from 8daa87b to 36c8476 Compare November 27, 2024 14:10

findinpath reviewed Nov 27, 2024

View reviewed changes

findinpath approved these changes Nov 28, 2024

View reviewed changes

ebyhr merged commit 5b82e10 into trinodb:master Nov 28, 2024
42 checks passed

github-actions bot added this to the 467 milestone Nov 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for concurrent write on Iceberg transformed column #24160

Add support for concurrent write on Iceberg transformed column #24160

pajaks commented Nov 18, 2024 •

edited by ebyhr

Loading

findinpath Nov 20, 2024

pajaks Nov 20, 2024 •

edited

Loading

findinpath Nov 21, 2024

pajaks Nov 22, 2024

pajaks Nov 22, 2024

raunaqmorarka Nov 25, 2024

raunaqmorarka Nov 25, 2024

raunaqmorarka left a comment

findinpath Nov 27, 2024

pajaks commented Nov 28, 2024

Add support for concurrent write on Iceberg transformed column #24160

Add support for concurrent write on Iceberg transformed column #24160

Conversation

pajaks commented Nov 18, 2024 • edited by ebyhr Loading

Description

Release notes

findinpath Nov 20, 2024

Choose a reason for hiding this comment

pajaks Nov 20, 2024 • edited Loading

Choose a reason for hiding this comment

findinpath Nov 21, 2024

Choose a reason for hiding this comment

pajaks Nov 22, 2024

Choose a reason for hiding this comment

pajaks Nov 22, 2024

Choose a reason for hiding this comment

raunaqmorarka Nov 25, 2024

Choose a reason for hiding this comment

raunaqmorarka Nov 25, 2024

Choose a reason for hiding this comment

raunaqmorarka left a comment

Choose a reason for hiding this comment

findinpath Nov 27, 2024

Choose a reason for hiding this comment

pajaks commented Nov 28, 2024

pajaks commented Nov 18, 2024 •

edited by ebyhr

Loading

pajaks Nov 20, 2024 •

edited

Loading