Handle Iceberg data file missing null count stats#11832
Handle Iceberg data file missing null count stats#11832findepi merged 1 commit intotrinodb:masterfrom
Conversation
There was a problem hiding this comment.
Please add a test to showcase this issue
There was a problem hiding this comment.
Here is a naive test case showcasing the bug you fixed
@Test
public void testNoMetrics()
{
assertUpdate("create table table_no_metrics_partitioned (name varchar, country varchar) WITH (partitioning = ARRAY['country'])");
Table table = IcebergUtil.loadIcebergTable(trinoCatalog, tableOperationsProvider, TestingConnectorSession.SESSION,
new SchemaTableName("tpch", "table_no_metrics_partitioned"));
// skip metrics
table.updateProperties()
.set("write.metadata.metrics.default", "none")
.commit();
assertUpdate("INSERT INTO table_no_metrics_partitioned VALUES ('Christoph', 'AT'), (NULL, 'RO')", 2);
assertEquals(computeActual("SELECT count(*) FROM \"table_no_metrics_partitioned$partitions\" WHERE data IS NOT NULL").getOnlyValue(), 0L);
}
There was a problem hiding this comment.
nice, this probably would exercise missing min/max as well, right? IIRC we didn't add a test then
There was a problem hiding this comment.
I tried this out, but I'm getting an empty Map for min/max/null-counts instead of null. I'd have to look at the Iceberg code, but maybe it changed at some point?
There was a problem hiding this comment.
We may also want to file an Issue to respect write.metadata.metrics.default and the other stats tuning properties at some point
There was a problem hiding this comment.
I did run the test on your branch @alexjo2144 and I inspected specifically to see whether the nullCounts map is NULL.
BTW I did the test on TestIcebergMergeAppend - it wouldn't fit for this PR, but I had there the trinoCatalog and tableOperationsProvider at hand.
I'm thinking that this is a good reason to handle #11828 in the near future to be covered against regressions that we can't catch when creating content with the latest version of Iceberg.
eb9e676 to
14cf94a
Compare
|
@alexjo2144 there are test failures |
f8a2961 to
342343f
Compare
342343f to
6f2c243
Compare
|
@alexjo2144 do you want RN for this? |
|
RN entry is in description .. I will use that |
@mosabua thanks, missed that
@alexjo2144 this should spell out NPE and should specify what operation could fail (SELECT? INSERT? SELECT from a system table?) |
Description
Null counts is an optional field and may be null. Handle that case appropriately by invalidating stats rather than throwing a NPE.
Fix
Iceberg connector
Related issues, pull requests, and links
Documentation
(x) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.
Release notes
( ) No release notes entries required.
(x) Release notes entries required with the following suggested text: