support create table like in flink catalog and watermark in windows #12116

swapna267 · 2025-01-27T21:41:29Z

This PR addresses

Creation of dynamic Iceberg table in Flink Catalog using the underlying physical Iceberg table using LIKE clause.
Iceberg Source to support Source Watermark, so it can be used in Flink WINDOW functions. https://github.com/apache/flink/blob/release-1.18/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/abilities/SupportsSourceWatermark.java enables Flink to rely on the watermark strategy provided by the ScanTableSource itself.

CREATE TABLE table_wm (
      eventTS AS CAST(t1 AS TIMESTAMP(3)),
      WATERMARK FOR eventTS AS SOURCE_WATERMARK()
) WITH (
  'watermark-column'='t1'
) LIKE iceberg_catalog.db.table;

Reference:
#10219
#9346

pvary · 2025-01-28T15:00:53Z

@swapna267: Started the test runs so we can see the status of the PR.
Could you please remove the Flink 1.18/1.9 version part of the changes?
It makes the review easier if we concentrate only a single Flink version, and do the backport later when the changes to the main version are merged.

swapna267 · 2025-01-28T18:32:37Z

@pvary reverted 1.18/1.19 changes.

swapna267 · 2025-01-29T03:04:16Z

Background:

Creation of dynamic Iceberg table in Flink Catalog using the underlying physical Iceberg table using LIKE clause.

Currently (without the changes in PR), create table in flink catalog works by configuring flink connector as described in,
flink-connector

But that needs user to provide the schema for the table. A way to tackle that is to do create table LIKE using below DDL.

CREATE TABLE table_wm (
      eventTS AS CAST(t1 AS TIMESTAMP(3)),
      WATERMARK FOR eventTS AS SOURCE_WATERMARK()
) WITH (
  'connector'='iceberg',
  'catalog-name'='iceberg_catalog',
  'catalog-database'='testdb',
  'catalog-table'='t'
) LIKE iceberg_catalog.testdb.t;

Options like connector, catalog-name, catalog-database, catalog-table need to be duplicated as Iceberg FlinkCatalog doesn't return any catalog related properties during getTable. This PR addresses the issue by including these properties when getTable is called , which will be used by Flink when creating table in Flink Catalog.

Iceberg Source to support Source Watermark
As raised in The "Emitting watermarks" feature can't be used in flink sql? #10219, Source watermark implemented as part of https://iceberg.apache.org/docs/nightly/flink-queries/#emitting-watermarks cannot be used in Flink window functions.

Flink let's user to push down watermark to source using interface SupportSourceWatermark.java .

Here we are falling back to read options implemented in #9346 , to configure the watermark column on Iceberg Source.

pvary · 2025-01-29T09:12:39Z

flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java

-    if (Objects.equals(
-        table.getOptions().get("connector"), FlinkDynamicTableFactory.FACTORY_IDENTIFIER)) {
-      throw new IllegalArgumentException(
-          "Cannot create the table with 'connector'='iceberg' table property in "
-              + "an iceberg catalog, Please create table with 'connector'='iceberg' property in a non-iceberg catalog or "
-              + "create table without 'connector'='iceberg' related properties in an iceberg table.");
-    }


Why do we remove this check?

Tables can be created using LIKE in

Flink Catalog - Not supported currently.

Another table in Iceberg catalog itself as detailed in doc

This check basically fails, if we try to create table using LIKE in Iceberg catalog, basically case#2 if we have connector=iceberg in options . For example, DDL like below,

CREATE TABLE `hive_catalog`.`default`.`sample_like` LIKE `hive_catalog`.`default`.`sample` WITH ('connector'='iceberg')

In order to support Case#1 without user setting any extra Options using WITH clause, we need to add connector in getTable,

iceberg/flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java

Line 344 in 52bfbdc

catalogAndTableProps.put("connector", FlinkDynamicTableFactory.FACTORY_IDENTIFIER);

This check was added in very old PR,
#2666
#2666 (comment) where Flink SQL didn't support CREATE TABLE A LIKE B , where A and B are in different Catalogs.

So, in this case by removing this check, we are ignoring connector option being passed, so following DDL can create table table_like in Flink catalog backed by iceberg_catalog.db.table. As we know source table is an Iceberg table, adding connector=iceberg would be redundant.

CREATE TABLE table_like ( eventTS AS CAST(t1 AS TIMESTAMP(3)), ) LIKE iceberg_catalog.db.table;

What happens when the source table is not an Iceberg table?
I'm trying to understand here, where we get the missing information in this case, and wether we have a way to check that we actually get the missing information. If we can create such a check, then we can still throw an exception when we don't get this information from any source

createTable in this catalog is called only when Source table is an Iceberg Table. Currently the Catalog information comes when Catalog is created.

Following are the scenarios when getTable / createTable methods in this Catalog are being used.

Create Iceberg table in Iceberg catalog -> Only createTable is called, where Catalog instance has all catalog related info.

Create table in iceberg catalog catalog1 like table in iceberg catalog catalog1 -> getTable() sets schema/partitioning info , which is used to create the table with same schema/partitioning as source in Catalog1.

Create table in iceberg catalog catalog2 like table in iceberg catalog catalog1 -> getTable() sets schema/partitioning info , which is used to create the table with same schema/partitioning as source in Catalog2.

Create table in Flink catalog like table in iceberg catalog -> getTable() is only called to get the source table info and createTable called on Flink catalog, where connector/iceberg catalog properties are being used to instantiate FlinkDynamicTableFactory.

When createTable is invoked, currently there is no easy way to differentiate between Case 2) / Case 3) / Case 4) Or user is doing

CREATE TABLE `hive_catalog`.`default`.`sample_like` WITH ('connector'='iceberg', 'catalog-name'='')

With current changes in PR, there is one side-effect. In Case 2 & 3 mentioned above, the extra properties added in getTable,

iceberg/flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java

Lines 339 to 344 in 52bfbdc

catalogAndTableProps.put(FlinkCreateTableOptions.CATALOG_NAME.key(), getName());

catalogAndTableProps.put(

FlinkCreateTableOptions.CATALOG_DATABASE.key(), tablePath.getDatabaseName());

catalogAndTableProps.put(

FlinkCreateTableOptions.CATALOG_TABLE.key(), tablePath.getObjectName());

catalogAndTableProps.put("connector", FlinkDynamicTableFactory.FACTORY_IDENTIFIER);

will be added as Table properties in destination table. Looking into nicer ways to avoid that. An option can be to add an extraProperty like say , createtabletype -> like in getTable , which can be used to differentiate the way table is being created. And extra props can be dropped in case 2&3 .

I don't get this:

Currently the Catalog information comes when Catalog is created.

Let's talk offline

pvary · 2025-01-29T09:14:34Z

flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/source/IcebergTableSource.java

        SupportsFilterPushDown,
-        SupportsLimitPushDown {
+        SupportsLimitPushDown,
+        SupportsSourceWatermark {


I think we have 2 feature in a single PR:

CREATE TABLE LIKE

Watermark support

Could we separate out these features to different PRs?
Could we write tests for both features?

These features were driven by mainly a use case, where an iceberg table is needed to be used in Flink window functions. This needs an incoming table to have MILLISECOND precision timestamp column and also watermark to be defined on source table.

As iceberg only supports MICROSECOND timestamp columns, we need to have a table with computed columns and we can create these only in Flink Catalog. Iceberg catalog doesn't support creating tables with computed columns.

i am happy to split them into 2 separate PR's .
I have tests for CREATE TABLE LIKE.

As Watermark support is just making Source to implement interface and falling back to #9346 for core logic, i didn't have a test case. I can add a validation on if watermark-column is configured or not , so it can fail fast. And a test case around that.

Please separate out the features to 2 PR

pvary · 2025-01-29T09:15:20Z

flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/TestIcebergConnector.java

-  @TestTemplate
-  public void testConnectorTableInIcebergCatalog() {
-    // Create the catalog properties
-    Map<String, String> catalogProps = Maps.newHashMap();


Why is this test removed?

This is testing the check mentioned in #12116 (comment)

Fail creating a table in Iceberg Catalog if connector=iceberg is specified in the Option. As the check is been deleted, i removed this test case.

I think that this is still a valid check in most cases. Only not valid when the table is created with "CREATE TABLE.. LIKE" and only if the source table is an iceberg table.
Do I miss something?

github-actions · 2025-03-13T00:16:38Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

github-actions · 2025-03-21T00:16:13Z

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

support create table like in flink catalog and watermark in windows

a9e3864

github-actions bot added the flink label Jan 27, 2025

swapna267 added 2 commits January 28, 2025 09:31

undo 1.18 changes

38df8d8

undo 1.19 changes

52bfbdc

pvary reviewed Jan 29, 2025

View reviewed changes

swapna267 requested a review from pvary January 30, 2025 17:23

This was referenced Feb 7, 2025

Flink: Support source watermark for flink sql windows #12191

Merged

support create table like in flink catalog #12199

Merged

github-actions bot added the stale label Mar 13, 2025

github-actions bot closed this Mar 21, 2025

	catalogAndTableProps.put(FlinkCreateTableOptions.CATALOG_NAME.key(), getName());
	catalogAndTableProps.put(
	FlinkCreateTableOptions.CATALOG_DATABASE.key(), tablePath.getDatabaseName());
	catalogAndTableProps.put(
	FlinkCreateTableOptions.CATALOG_TABLE.key(), tablePath.getObjectName());
	catalogAndTableProps.put("connector", FlinkDynamicTableFactory.FACTORY_IDENTIFIER);

support create table like in flink catalog and watermark in windows #12116

support create table like in flink catalog and watermark in windows #12116

Uh oh!

Conversation

swapna267 commented Jan 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pvary commented Jan 28, 2025

Uh oh!

swapna267 commented Jan 28, 2025

Uh oh!

swapna267 commented Jan 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

swapna267 Jan 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

swapna267 Feb 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

swapna267 Jan 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 13, 2025

Uh oh!

github-actions bot commented Mar 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

swapna267 commented Jan 27, 2025 •

edited

Loading

swapna267 commented Jan 29, 2025 •

edited

Loading

swapna267 Jan 29, 2025 •

edited

Loading

swapna267 Feb 8, 2025 •

edited

Loading

swapna267 Jan 29, 2025 •

edited

Loading