Skip to content

Conversation

@rmoff
Copy link
Contributor

@rmoff rmoff commented Jul 21, 2025

Per https://issues.apache.org/jira/browse/FLINK-21634, these features are now available.

e.g.

Flink SQL> SET 'execution.checkpointing.interval' = '30s';
[INFO] Execute statement succeeded.

Flink SQL> CREATE TABLE test01 (col1 INT)
>     WITH (
>         'connector' = 'iceberg',
>         'catalog-name' = 'foo',
>         'catalog-database' = 'my_glue_db',
>         'warehouse' = 's3://rmoff-lakehouse/00/',
>         'catalog-impl' = 'org.apache.iceberg.aws.glue.GlueCatalog',
>         'ioImpl' = 'org.apache.iceberg.aws.s3.S3FileIO');
[INFO] Execute statement succeeded.

-- Add a column
Flink SQL> ALTER TABLE test01 ADD test_col INT;
[INFO] Execute statement succeeded.

Flink SQL> DESCRIBE test01;
+----------+------+------+-----+--------+-----------+
|     name | type | null | key | extras | watermark |
+----------+------+------+-----+--------+-----------+
|     col1 |  INT | TRUE |     |        |           |
| test_col |  INT | TRUE |     |        |           |
+----------+------+------+-----+--------+-----------+
2 rows in set


-- Drop a column
Flink SQL> ALTER TABLE test01 DROP test_col;
[INFO] Execute statement succeeded.

Flink SQL> DESCRIBE test01;
+------+------+------+-----+--------+-----------+
| name | type | null | key | extras | watermark |
+------+------+------+-----+--------+-----------+
| col1 |  INT | TRUE |     |        |           |
+------+------+------+-----+--------+-----------+
1 row in set

-- Rename a column
Flink SQL> ALTER TABLE test01 RENAME test_col TO foo;
[INFO] Execute statement succeeded.

Flink SQL> DESCRIBE test01;
+------+------+------+-----+--------+-----------+
| name | type | null | key | extras | watermark |
+------+------+------+-----+--------+-----------+
| col1 |  INT | TRUE |     |        |           |
|  foo |  INT | TRUE |     |        |           |
+------+------+------+-----+--------+-----------+
2 rows in set


-- Modify a column
Flink SQL> ALTER TABLE test01 MODIFY col1 STRING;
[INFO] Execute statement succeeded.

Per https://issues.apache.org/jira/browse/FLINK-21634, these features are now available.

e.g.

```sql
Flink SQL> SET 'execution.checkpointing.interval' = '30s';
[INFO] Execute statement succeeded.

Flink SQL> CREATE TABLE test01 (col1 INT)
>     WITH (
>         'connector' = 'iceberg',
>         'catalog-name' = 'foo',
>         'catalog-database' = 'my_glue_db',
>         'warehouse' = 's3://rmoff-lakehouse/00/',
>         'catalog-impl' = 'org.apache.iceberg.aws.glue.GlueCatalog',
>         'ioImpl' = 'org.apache.iceberg.aws.s3.S3FileIO');
[INFO] Execute statement succeeded.

-- Add a column
Flink SQL> ALTER TABLE test01 ADD test_col INT;
[INFO] Execute statement succeeded.

Flink SQL> DESCRIBE test01;
+----------+------+------+-----+--------+-----------+
|     name | type | null | key | extras | watermark |
+----------+------+------+-----+--------+-----------+
|     col1 |  INT | TRUE |     |        |           |
| test_col |  INT | TRUE |     |        |           |
+----------+------+------+-----+--------+-----------+
2 rows in set


-- Drop a column
Flink SQL> ALTER TABLE test01 DROP test_col;
[INFO] Execute statement succeeded.

Flink SQL> DESCRIBE test01;
+------+------+------+-----+--------+-----------+
| name | type | null | key | extras | watermark |
+------+------+------+-----+--------+-----------+
| col1 |  INT | TRUE |     |        |           |
+------+------+------+-----+--------+-----------+
1 row in set

-- Rename a column
Flink SQL> ALTER TABLE test01 RENAME test_col TO foo;
[INFO] Execute statement succeeded.

Flink SQL> DESCRIBE test01;
+------+------+------+-----+--------+-----------+
| name | type | null | key | extras | watermark |
+------+------+------+-----+--------+-----------+
| col1 |  INT | TRUE |     |        |           |
|  foo |  INT | TRUE |     |        |           |
+------+------+------+-----+--------+-----------+
2 rows in set


-- Modify a column
Flink SQL> ALTER TABLE test01 MODIFY col1 STRING;
[INFO] Execute statement succeeded.
```
@github-actions github-actions bot added the docs label Jul 21, 2025
@rmoff rmoff changed the title Flink can now add/drop/modify columns [docs] Flink can now add/drop/modify columns Jul 21, 2025
Copy link
Contributor

@mxm mxm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @rmoff for spotting this important change!


* Don't support creating iceberg table with hidden partitioning. [Discussion](http://mail-archives.apache.org/mod_mbox/flink-dev/202008.mbox/%3cCABi+2jQCo3MsOa4+ywaxV5J-Z8TGKNZDX-pQLYB-dG+dVUMiMw@mail.gmail.com%3e) in flink mail list.
* Don't support creating iceberg table with computed column.
* Don't support creating iceberg table with watermark.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Side note: This is also not true anymore: #12191

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we remove that in this PR too? or open a new one?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While we're on it, we can simply remove this one as well. What do you think?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, makes sense. So the two remaining features to list as not implemented are:

  • Don't support creating iceberg table with hidden partitioning. Discussion in flink mail list.
  • Don't support creating iceberg table with computed column.

is that right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes @rmoff, those are the 2 last missing features mentioned here.
Could you please update the doc accordingly?

Thanks,
Peter

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is that right?

Correct.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed d60a9a8

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pvary could you review/merge please? thanks.

@pvary pvary merged commit dd8f5e3 into apache:main Aug 6, 2025
2 checks passed
@pvary
Copy link
Contributor

pvary commented Aug 6, 2025

Merged to main.
Thanks for the PR @rmoff and @mxm for the review!

@Fokko Fokko changed the title [docs] Flink can now add/drop/modify columns Docs: Flink can add/drop/modify columns Aug 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants