Add all_files system table to the Iceberg connector #11206

osscm · 2022-02-26T00:40:03Z

Description

This is to add $all_files system table support.
Spark already supports the iceberg's $data_files and $all_data_files metadata tables. Trino is already supporting $files.
$all_files table will include data_files from all the available snapshots to a table where the $files metadata table includes only the current snapshot one.

Use cases

This can be used when the user is debugging data issue. As in case when Trino/Spark is used for metadata/data optimization, then it can modify the metadata and data.
Another case is when snapshot is rolled back/future from Trino/Spark and now trying to understand what all data-files are present, and is there any implication because of the optimization/rollback operations.
This and $all_manifests can also be used to add the optimization features in Trino like purging orphan files or identifying partitions modified since last time, to implement moving window data-compaction feature. detail
Where we need to identify the

Design
Adding a new class AllFilesTable and used it as a parent for FilesTable, as both will be implementing similar responsibility,.

Testing
Added a test case in the existing TestIcebergSystemTable.
Thought of using rollback to show case that $all_files can give all the data_files, but then history was getting updated and testHistoryTable depends on the order and operations happen in the testAllFilesTable test. So have not used it. If we think, I can add it.

Is this change a fix, improvement, new feature, refactoring, or other?
Improvement

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)
NA

How would you describe this change to a non-technical end user or system administrator?
allows user to access all the data-files referred by the table ( rolled back snapshots, old snapshots). This is helpful for debugging issues like why my query is showing certain data and not current one (in case snapshots are rolled back)

syntax:
select * from "table$all_files"

Related issues, pull requests, and links

Fixes 11172

Documentation

( ) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

Release notes

( ) No release notes entries required.
( ) Release notes entries required with the following suggested text:

# Section
* Adds support for `all_files` metadata table, supported by Spark as well. ({issue}`11172`)

cla-bot · 2022-02-26T00:40:06Z

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please submit the signed CLA to [email protected]. For more information, see https://github.com/trinodb/cla.

alexjo2144

Can you add a short entry to https://github.com/trinodb/trino/blob/master/docs/src/main/sphinx/connector/iceberg.rst#metadata-tables as well?

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/AllFilesTable.java

cla-bot · 2022-03-01T06:40:25Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Manish Malhotra.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email [email protected]
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

cla-bot · 2022-03-01T06:46:57Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Manish Malhotra.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email [email protected]
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

osscm · 2022-03-01T06:51:18Z

@alexjo2144 thanks for the review!
please check, the refactored code.

cla-bot · 2022-03-01T09:28:22Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Manish Malhotra.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email [email protected]
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

cla-bot · 2022-03-01T09:29:04Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Manish Malhotra.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email [email protected]
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

cla-bot · 2022-03-01T20:35:08Z

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please submit the signed CLA to [email protected]. For more information, see https://github.com/trinodb/cla.

osscm · 2022-03-01T20:39:04Z

closing to fix the git email issue

cla-bot · 2022-03-02T08:20:05Z

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please submit the signed CLA to [email protected]. For more information, see https://github.com/trinodb/cla.

cla-bot · 2022-03-02T08:24:08Z

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please submit the signed CLA to [email protected]. For more information, see https://github.com/trinodb/cla.

martint · 2022-03-02T21:21:21Z

@cla-bot check

cla-bot · 2022-03-02T21:21:24Z

The cla-bot has been summoned, and re-checked this pull request!

osscm · 2022-03-03T17:04:08Z

thanks a lot @martint!

osscm · 2022-03-03T17:06:32Z

@alexjo2144 would you mind checking it, and seeing if changes are looking ok now?
Also, fixed a doc formatting and TestIcebergMetastoreAccessOperations.java as well. thanks!

alexjo2144

Besides a few nitpicks, looks good to me.

docs/src/main/sphinx/connector/iceberg.rst

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/AllFilesTable.java

docs/src/main/sphinx/connector/iceberg.rst

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergSystemTables.java

mosabua

Only reviewed docs and left some suggestions on top of the good ideas from @findinpath

docs/src/main/sphinx/connector/iceberg.rst

osscm · 2022-03-10T21:49:58Z

@findinpath @mosabua thanks for the review! will try to add the above changes today/tomorrow.

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/AllFilesTable.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/FilesTable.java

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergSystemTables.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/AbstractFilesTable.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/AllFilesTable.java

findinpath · 2022-09-23T08:07:51Z

Build is red. Are the trino-iceberg failures related?

osscm · 2022-10-11T08:44:41Z

Ok @ebyhr let me check your comments, thanks

…

On Wed, Jul 20, 2022 at 10:31 PM Yuya Ebihara ***@***.***> wrote: ***@***.**** commented on this pull request. I don't think we can merge this PR as-is. Please take a look at my comment about $files table change. ------------------------------ In docs/src/main/sphinx/connector/iceberg.rst <#11206 (comment)>: > + +``$all_files`` table +^^^^^^^^^^^^^^^^^^^^ + +The ``$all_files`` table exposes the valid data files of the Iceberg table. +A valid data file is one that is readable from any snapshot currently tracked by the table. + +To retrieve the information about the data files from all the snapshots of the Iceberg table ``test_table`` use the following query:: + + SELECT * FROM "test_table$all_files" + +The output of the query has the columns, which is similar to the ``$files`` metadata table. + +.. code-block:: text + + content | file_path | record_count | file_format | file_size_in_bytes | column_sizes | value_counts | null_value_counts | nan_value_counts | lower_bounds | upper_bounds | key_metadata | split_offsets | equality_ids Please replace the order > record_count & file_format. ------------------------------ In docs/src/main/sphinx/connector/iceberg.rst <#11206 (comment)>: > + 0 | hdfs://hadoop-master:9000/user/hive/warehouse/test_table/data/c1=3/c2=2021-01-14/af9872b2-40f3-428f-9c87-186d2750d84e.parquet | 1 | PARQUET | 442 | {1=40, 2=40, 3=44} | {1=1, 2=1, 3=1} | {1=0, 2=0, 3=0} | <null> | {1=3, 2=2021-01-14, 3=1.3} | {1=3, 2=2021-01-14, 3=1.3} | <null> | <null> | <null> + 0 | hdfs://hadoop-master:9000/user/hive/warehouse/test_table/data/c1=3/c2=2021-01-14/af9872b2-40f3-428f-9c87-186d2750d84e.parquet | 1 | PARQUET | 442 | {1=40, 2=40, 3=44} | {1=1, 2=1, 3=1} | {1=0, 2=0, 3=0} | <null> | {1=3, 2=2021-01-14, 3=1.3} | {1=3, 2=2021-01-14, 3=1.2} | <null> | <null> | <null> The 1st & 2nd file path are identical, but their upper bound differ. When do we face such situation? Providing those 2 files as the example might be little confusing and not helpful. Removing the 2nd file row is fine in my opinion. ------------------------------ In plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/FilesTable.java <#11206 (comment)>: > public class FilesTable - implements SystemTable + extends AbstractFilesTable Please separate a commit to introduce AbstractFilesTable. ------------------------------ In plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/FilesTable.java <#11206 (comment)>: > - TableScan tableScan = icebergTable.newScan() - .useSnapshot(snapshotId) + TableScan tableScan = filesTable This change prevents accessing $files system table if delete-files exists. Such failure didn't happen before this commit. You can reproduce by below steps: 1. Start IcebergQueryRunner 2. Execute DELETE FROM region WHERE regionkey = 1 3. Execute SELECT * FROM "region$files" class org.apache.iceberg.GenericDeleteFile cannot be cast to class org.apache.iceberg.DataFile (org.apache.iceberg.GenericDeleteFile and org.apache.iceberg.DataFile are in unnamed module of loader 'app') Please add this test. ------------------------------ In plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergSystemTables.java <#11206 (comment)>: > + MaterializedResult expectedStatistics = + MaterializedResult.resultBuilder(getSession(), VARCHAR, BIGINT, new MapType(INTEGER, VARCHAR, new TypeOperators()), + new MapType(IntegerType.INTEGER, VARCHAR, new TypeOperators())) + .row("ORC", 1L, Map.of(1, "0", 2, "2019-09-08"), Map.of(1, "0", 2, "2019-09-08")) + .row("ORC", 2L, Map.of(1, "1", 2, "2019-09-09"), Map.of(1, "2", 2, "2019-09-09")) + .row("ORC", 1L, Map.of(1, "3", 2, "2019-09-09"), Map.of(1, "3", 2, "2019-09-09")) + .row("ORC", 2L, Map.of(1, "4", 2, "2019-09-10"), Map.of(1, "5", 2, "2019-09-10")) + .build(); + + assertEquals(result, expectedStatistics); + } + + @test + public void testAllFilesTable() + { + try (TestTable table = new TestTable(getQueryRunner()::execute, "all_files_table", "(a int, b VARCHAR)", ImmutableList.of("(1, 'a')"))) { int ... VARCHAR nit: Let's uppercase int. ------------------------------ In plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergSystemTables.java <#11206 (comment)>: > + "VALUES ('content', 'integer', '', '')," + + "('file_path', 'varchar', '', '')," + + "('file_format', 'varchar', '', '')," + + "('record_count', 'bigint', '', '')," + + "('file_size_in_bytes', 'bigint', '', '')," + + "('column_sizes', 'map(integer, bigint)', '', '')," + + "('value_counts', 'map(integer, bigint)', '', '')," + + "('null_value_counts', 'map(integer, bigint)', '', '')," + + "('nan_value_counts', 'map(integer, bigint)', '', '')," + + "('lower_bounds', 'map(integer, varchar)', '', '')," + + "('upper_bounds', 'map(integer, varchar)', '', '')," + + "('key_metadata', 'varbinary', '', '')," + + "('split_offsets', 'array(bigint)', '', '')," + + "('equality_ids', 'array(integer)', '', '')"); + + assertQuerySucceeds("SELECT * FROM \"" + table.getName() + "$all_files\""); Is it possible to verify all fields' values? Currently, this assertion confirms only query success and some fields (e.g. key_metadata, split_offsets & equality_ids) are null. ------------------------------ In plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergSystemTables.java <#11206 (comment)>: > + assertQuerySucceeds("SELECT * FROM \"" + table.getName() + "$all_files\""); + + MaterializedResult filesAfterSingleRow = computeActual("SELECT file_format, record_count, lower_bounds, upper_bounds FROM \"" + table.getName() + "$files\""); + MaterializedResult allFilesAfterSingleRow = computeActual("SELECT file_format, record_count, lower_bounds, upper_bounds FROM \"" + table.getName() + "$all_files\""); + assertEquals(filesAfterSingleRow, allFilesAfterSingleRow); + + MaterializedResult expectedFilesStatsAfterSingleRow = + MaterializedResult.resultBuilder(getSession(), VARCHAR, BIGINT, new MapType(INTEGER, VARCHAR, new TypeOperators()), new MapType(IntegerType.INTEGER, VARCHAR, new TypeOperators())) + .row("ORC", 1L, Map.of(1, "1", 2, "a"), Map.of(1, "1", 2, "a")) + .build(); + + assertEquals(filesAfterSingleRow, expectedFilesStatsAfterSingleRow); + + Long snapshotAfterSingleRow = (Long) computeScalar("SELECT snapshot_id FROM \"" + table.getName() + "$snapshots\" ORDER BY committed_at DESC LIMIT 1"); + + assertUpdate("INSERT INTO " + table.getName() + " VALUES (2, 'b')", 1); Please add DLETE statement to this test. It would be better to add other statements and procedures too. ------------------------------ In plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergSystemTables.java <#11206 (comment)>: > + MaterializedResult filesAfterTwoRows = computeActual("SELECT file_format, record_count, lower_bounds, upper_bounds FROM \"" + table.getName() + "$files\" order by file_path"); + MaterializedResult allFilesAfterTwoRows = computeActual("SELECT file_format, record_count, lower_bounds, upper_bounds FROM \"" + table.getName() + "$all_files\" order by file_path"); nit: Uppercase ORDER BY ------------------------------ In plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergSystemTables.java <#11206 (comment)>: > + + assertUpdate("INSERT INTO " + table.getName() + " VALUES (2, 'b')", 1); + + MaterializedResult filesAfterTwoRows = computeActual("SELECT file_format, record_count, lower_bounds, upper_bounds FROM \"" + table.getName() + "$files\" order by file_path"); + MaterializedResult allFilesAfterTwoRows = computeActual("SELECT file_format, record_count, lower_bounds, upper_bounds FROM \"" + table.getName() + "$all_files\" order by file_path"); + + MaterializedResult expectedAllFilesStatsAfterTwoRows = + MaterializedResult.resultBuilder(getSession(), VARCHAR, BIGINT, new MapType(INTEGER, VARCHAR, new TypeOperators()), new MapType(IntegerType.INTEGER, VARCHAR, new TypeOperators())) + .row("ORC", 1L, Map.of(1, "1", 2, "a"), Map.of(1, "1", 2, "a")) + .row("ORC", 1L, Map.of(1, "2", 2, "b"), Map.of(1, "2", 2, "b")) + .build(); + assertEquals(filesAfterTwoRows, allFilesAfterTwoRows); + + assertEquals(filesAfterTwoRows, expectedAllFilesStatsAfterTwoRows); + + assertUpdate("CALL iceberg.system.rollback_to_snapshot('tpch', '" + table.getName() + "' , " + snapshotAfterSingleRow + ")"); nit: Remove a redundant space. ⬇️ Suggested change - assertUpdate("CALL iceberg.system.rollback_to_snapshot('tpch', '" + table.getName() + "' , " + snapshotAfterSingleRow + ")"); + assertUpdate("CALL iceberg.system.rollback_to_snapshot('tpch', '" + table.getName() + "', " + snapshotAfterSingleRow + ")"); ------------------------------ In plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergSystemTables.java <#11206 (comment)>: > + MaterializedResult filesAfterRollback = computeActual("SELECT file_format, record_count, lower_bounds, upper_bounds FROM \"" + table.getName() + "$files\" order by file_path"); + MaterializedResult allFilesAfterRollback = computeActual("SELECT file_format, record_count, lower_bounds, upper_bounds FROM \"" + table.getName() + "$all_files\" order by file_path"); nit: Uppercase ORDER BY — Reply to this email directly, view it on GitHub <#11206 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXQ2PYRPIJYTZYIMGIXQP63VVDOBXANCNFSM5PLUJCCA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

bitsondatadev · 2022-11-19T06:14:21Z

👋 @osscm - this PR has become inactive. If you're still interested in working on it, please let us know, and we can try to get reviewers to help with that.

We're working on closing out old and inactive PRs, so if you're too busy or this has too many merge conflicts to be worth picking back up, we'll be making another pass to close it out in a few weeks.

osscm · 2022-11-20T02:26:59Z

Thanks @bitsondatadev Yes I think all_files will be useful for the community as well.

…

On Fri, Nov 18, 2022 at 10:14 PM bitsondatadev ***@***.***> wrote: 👋 @osscm <https://github.com/osscm> - this PR has become inactive. If you're still interested in working on it, please let us know, and we can try to get reviewers to help with that. We're working on closing out old and inactive PRs, so if you're too busy or this has too many merge conflicts to be worth picking back up, we'll be making another pass to close it out in a few weeks. — Reply to this email directly, view it on GitHub <#11206 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXQ2PYXIMC2QIXZIBJHJ7Y3WJBV4RANCNFSM5PLUJCCA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

bitsondatadev · 2022-11-20T15:27:32Z

Great! We won't close it out then! I figured as such but we just need to gauge the interest in continuing all older PRs. I'll work on getting a maintainer to take a look!

osscm · 2022-11-20T22:58:03Z

Thanks!

…

On Sun, Nov 20, 2022 at 7:27 AM bitsondatadev ***@***.***> wrote: Great! We won't close it out then! I figured as such but we just need to gauge the interest in continuing all older PRs. I'll work on getting a maintainer to take a look! — Reply to this email directly, view it on GitHub <#11206 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXQ2PYVULBAHFLYPCCB7SJ3WJI7PBANCNFSM5PLUJCCA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

ebyhr · 2022-11-21T03:28:06Z

docs/src/main/sphinx/connector/iceberg.rst

+    - List of recommended split locations.
+  * - ``equality_ids``
+    - ``array(integer)``
+    - The set of field IDs used for equality comparison in equality delete files


The above changes are unrelated to a new $all_files system table. Please extract a commit.

ebyhr · 2022-11-21T03:28:09Z

docs/src/main/sphinx/connector/iceberg.rst

+
+    SELECT * FROM "test_table$all_files"
+
+The output of the query has the columns, which is similar to the ``$files`` metadata table.


is similar to

I think they're the "same". Is my understanding wrong?

yes @ebyhr its same.
are you suggesting to reword similar to to same as?

I think so.

ebyhr · 2022-11-21T03:28:14Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/AbstractFilesTable.java

+        implements SystemTable
+{
+    private final ConnectorTableMetadata tableMetadata;
+    private final Table icebergTable;


Change to protected and use it from FilesTable and AllFilesTable

ebyhr · 2022-11-21T03:28:20Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/AllFilesTable.java

+import static org.apache.iceberg.MetadataTableType.ALL_DATA_FILES;
+import static org.apache.iceberg.MetadataTableUtils.createMetadataTableInstance;
+
+public class AllFilesTable


The class name is AllFilesTable, but MetadataTableType is ALL_DATA_FILES. It would be nice to leave a comment.

ebyhr · 2022-11-21T03:28:23Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/AllFilesTable.java

+        Table allFilesTable = createMetadataTableInstance(getIcebergTable(), ALL_DATA_FILES);
+
+        TableScan tableScan = allFilesTable
+                .newScan()
+                .includeColumnStats();
+        return tableScan;


Suggested change

Table allFilesTable = createMetadataTableInstance(getIcebergTable(), ALL_DATA_FILES);

TableScan tableScan = allFilesTable

.newScan()

.includeColumnStats();

return tableScan;

return createMetadataTableInstance(getIcebergTable(), ALL_DATA_FILES).newScan()

.includeColumnStats();

ebyhr · 2022-11-21T03:29:30Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergSystemTables.java

+
+            assertEquals(result, expectedStatistics);
+        }
+    }


The above changes are unrelated to a new $all_files system table. Please extract a commit.

ebyhr · 2022-11-21T03:29:32Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergSystemTables.java

+    }
+
+    @Test
+    public void testAllFilesTable() throws IOException


Move throws to the next line.

ebyhr · 2022-11-21T03:29:55Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergSystemTables.java

+            MaterializedResult allFilesAfterSingleDelete = computeActual("SELECT content, file_format, file_size_in_bytes, record_count, column_sizes, value_counts," +
+                    "null_value_counts, nan_value_counts, key_metadata, split_offsets, equality_ids, lower_bounds, upper_bounds FROM \"" + table.getName() + "$all_files\" ORDER BY file_path");


I would recommend using SELECT *.

ebyhr · 2022-11-21T03:30:05Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergSystemTables.java

+                            .row(0, "ORC", fileSizes[0], 1L, null, Map.of(1, Long.valueOf(1), 2, Long.valueOf(1)), Map.of(1, Long.valueOf(0), 2, Long.valueOf(0)), null, null, null, null, Map.of(1, "1", 2, "a"), Map.of(1, "1", 2, "a"))
+                            .row(0, "ORC", fileSizes[1], 1L, null, Map.of(1, Long.valueOf(1), 2, Long.valueOf(1)), Map.of(1, Long.valueOf(0), 2, Long.valueOf(0)), null, null, null, null, Map.of(1, "2", 2, "b"), Map.of(1, "2", 2, "b"))


Remove boxing.

ebyhr · 2022-11-21T03:30:07Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergSystemTables.java

+        }
+    }
+
+    private long[] getFileSizes(List<MaterializedRow> materializedRows) throws IOException


Move throws to the next line.

bitsondatadev · 2022-11-21T16:50:56Z

Thanks @ebyhr! @osscm, reach out to me if you make changes and a few days pass without review please!

colebow · 2023-03-30T18:04:37Z

Hey @osscm, are you still working on this?

colebow · 2023-05-23T20:46:21Z

I'm going to close this out due to inactivity. Feel free to re-open if you want to pick this back up at any point in the future.

alexjo2144 reviewed Feb 28, 2022

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/AllFilesTable.java Outdated Show resolved Hide resolved

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/AllFilesTable.java Outdated Show resolved Hide resolved

osscm force-pushed the add-all—files branch from 8147368 to 48219a6 Compare March 1, 2022 06:46

github-actions bot added the docs label Mar 1, 2022

osscm force-pushed the add-all—files branch from 326dabc to b778c55 Compare March 1, 2022 09:29

osscm force-pushed the add-all—files branch from b778c55 to bff3223 Compare March 1, 2022 20:35

osscm closed this Mar 1, 2022

osscm reopened this Mar 1, 2022

osscm force-pushed the add-all—files branch from 4b29e95 to 55bfd72 Compare March 2, 2022 08:24

cla-bot bot added the cla-signed label Mar 2, 2022

alexjo2144 approved these changes Mar 4, 2022

View reviewed changes

alexjo2144 requested review from electrum and findepi March 4, 2022 19:21

findinpath reviewed Mar 9, 2022

View reviewed changes

mosabua requested changes Mar 9, 2022

View reviewed changes

findinpath reviewed Mar 11, 2022

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/AllFilesTable.java Outdated Show resolved Hide resolved

findinpath reviewed Sep 23, 2022

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/FilesTable.java Outdated Show resolved Hide resolved

findinpath reviewed Sep 23, 2022

View reviewed changes

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergSystemTables.java Outdated Show resolved Hide resolved

findinpath reviewed Sep 23, 2022

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/AbstractFilesTable.java Show resolved Hide resolved

findinpath reviewed Sep 23, 2022

View reviewed changes

osscm force-pushed the add-all—files branch 5 times, most recently from 90d533d to db3a6e5 Compare September 24, 2022 17:25

Extract common code to AbstractFilesTable

0908b1c

osscm force-pushed the add-all—files branch 3 times, most recently from bed7c17 to caf938e Compare September 25, 2022 01:33

osscm added 3 commits September 25, 2022 08:16

Introduce $all_files Iceberg metadata table

3f10c87

Wrap scan.planFiles and task.rows in try-with-resource

bfbce9f

Do not cast StructLike to DataFile in the addRow()

597faa6

osscm force-pushed the add-all—files branch from caf938e to 597faa6 Compare September 25, 2022 15:16

ebyhr reviewed Nov 21, 2022

View reviewed changes

findinpath mentioned this pull request Mar 7, 2023

Fix iceberg $files metadata table not show delete files #16232

Closed

kokosing force-pushed the master branch from 3f05134 to 58d6356 Compare March 14, 2023 11:34

colebow closed this May 23, 2023


		SELECT * FROM "test_table$all_files"

		The output of the query has the columns, which is similar to the ``$files`` metadata table.

		MaterializedResult allFilesAfterSingleDelete = computeActual("SELECT content, file_format, file_size_in_bytes, record_count, column_sizes, value_counts," +
		"null_value_counts, nan_value_counts, key_metadata, split_offsets, equality_ids, lower_bounds, upper_bounds FROM \"" + table.getName() + "$all_files\" ORDER BY file_path");

		.row(0, "ORC", fileSizes[0], 1L, null, Map.of(1, Long.valueOf(1), 2, Long.valueOf(1)), Map.of(1, Long.valueOf(0), 2, Long.valueOf(0)), null, null, null, null, Map.of(1, "1", 2, "a"), Map.of(1, "1", 2, "a"))
		.row(0, "ORC", fileSizes[1], 1L, null, Map.of(1, Long.valueOf(1), 2, Long.valueOf(1)), Map.of(1, Long.valueOf(0), 2, Long.valueOf(0)), null, null, null, null, Map.of(1, "2", 2, "b"), Map.of(1, "2", 2, "b"))

Add all_files system table to the Iceberg connector #11206

Add all_files system table to the Iceberg connector #11206

Conversation

osscm commented Feb 26, 2022

Description

Related issues, pull requests, and links

Documentation

Release notes

cla-bot bot commented Feb 26, 2022

alexjo2144 left a comment

Choose a reason for hiding this comment

cla-bot bot commented Mar 1, 2022

cla-bot bot commented Mar 1, 2022

osscm commented Mar 1, 2022

cla-bot bot commented Mar 1, 2022

cla-bot bot commented Mar 1, 2022

cla-bot bot commented Mar 1, 2022

osscm commented Mar 1, 2022

cla-bot bot commented Mar 2, 2022

cla-bot bot commented Mar 2, 2022

martint commented Mar 2, 2022

cla-bot bot commented Mar 2, 2022

osscm commented Mar 3, 2022

osscm commented Mar 3, 2022

alexjo2144 left a comment

Choose a reason for hiding this comment

mosabua left a comment

Choose a reason for hiding this comment

osscm commented Mar 10, 2022

findinpath commented Sep 23, 2022

osscm commented Oct 11, 2022 via email

bitsondatadev commented Nov 19, 2022

osscm commented Nov 20, 2022 via email

bitsondatadev commented Nov 20, 2022

osscm commented Nov 20, 2022 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bitsondatadev commented Nov 21, 2022 • edited Loading

colebow commented Mar 30, 2023

colebow commented May 23, 2023

bitsondatadev commented Nov 21, 2022 •

edited

Loading