Core: Registering tables to nonexistent target namespace leads to metadata deletion in HiveCatalog #13434

hsiang-c · 2025-06-30T20:26:48Z

Related to: #1533 and #14083

Context

We're registering existing Iceberg tables to HiveCatalog and realize that the metadata.json files used are deleted when the target namespace doesn't exist.
When the target namespace doesn't exist, an ValidationException is thrown from the doCommit() method of HiveTableOperations, as Error when creating a table InvalidObjectException #1533 indicated.
Therefore, the finally block of the doCommit method deletes the metadata.json from existing Iceberg tables and leads to data corruption.

    finally {
      HiveOperationsBase.cleanupMetadataAndUnlock(io(), commitStatus, newMetadataLocation, lock);
    }

Proposed changes

In BaseMetastoreCatalog and RESTSessionCatalog, registerTable method now throws NoSuchNamespaceException if namespace doesn't exist.
One exception is JdbcCatalog. It can be configured with jdbc.strict-mode=false and skip namespace existence checks for createTable and createView. Therefore, I follow this behavior in registerTable.

Tests

Catalog Impl.	Covered By
`BigQueryMetastoreCatalog`	`CatalogTests`
`InMemoryCatalog`	`CatalogTests`
`RESTCatalog`	`CatalogTests`
`HiveCatalog`	`CatalogTests`
`NessieCatalog`	`CatalogTests`
`JdbcCatalog`	`TestJdbcCatalog`
`DynamoDbCatalog`	`TestDynamoDbCatalog`
`EcsCatalog`	`TestEcsCatalog`
`GlueCatalog`	`TestGlueCatalog`
`HadoopCatalog`	`TestHadoopCatalog`
`SnowflakeCatalog`	`SnowflakeCatalogTest`

RESTCompatibilityKitCatalogTests is using RESTCatalog (client) and JdbcCatalog (server) combination. Therefore, only in iceberg-open-api module, I turned on strict mode on JdbcCatalog so that client and server have consistent behavior.
Also test from the perspective of Spark's RegisterTableProcedure

nastra · 2025-07-01T06:40:48Z

spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/RegisterTableProcedure.java

        "Cannot handle an empty argument metadata_file");

    Catalog icebergCatalog = ((HasIcebergCatalog) tableCatalog()).icebergCatalog();
+    if (tableName.hasNamespace() && icebergCatalog instanceof SupportsNamespaces) {


I don't think this check should be part of the procedure. Instead, such a check should be added in registerTable() in BaseMetastoreCatalog / RestSessionCatalog. Tests can then be added to CatalogTests for RestSessionCatalog. Tests for registerTable in BaseMetastoreCatalog are spread across different catalog implementations, so tests should be added there as well

@nastra Thank you for your feedback, I think it makes sense to me.

@nastra I've addressed your feedback. Please take another look when you have time, thanks.

hsiang-c · 2025-07-07T18:32:24Z

open-api/src/test/java/org/apache/iceberg/rest/RESTCompatibilityKitCatalogTests.java

    return PropertyUtil.propertyAsBoolean(
-        restCatalog.properties(),
-        RESTCompatibilityKitSuite.RCK_REQUIRES_NAMESPACE_CREATE,
-        super.requiresNamespaceCreate());


super.requiresNamespaceCreate() is false in parent class.

hsiang-c · 2025-07-07T18:42:34Z

core/src/main/java/org/apache/iceberg/jdbc/JdbcTableOperations.java

        && !JdbcUtil.namespaceExists(catalogName, connections, namespace)) {
      throw new NoSuchNamespaceException(
-          "Cannot create table %s in catalog %s. Namespace %s does not exist",
+          "Cannot create table %s in catalog %s. Namespace does not exist: %s",


Change the error message b/c CatalogTests.tableCreationWithoutNamespace() expects this format.

assertThatThrownBy( () -> catalog().buildTable(TableIdentifier.of("non-existing", "table"), SCHEMA).create()) .isInstanceOf(NoSuchNamespaceException.class) .hasMessageContaining("Namespace does not exist: non-existing");

dramaticlly · 2025-07-16T23:02:20Z

We're registering existing Iceberg tables to HiveCatalog and realize that the metadata.json files used are deleted when the target namespace doesn't exist.

I think the other way to look at this problem might be disable the clean up for failed commit on table registration

Today the TableOperations.commit() method assume the caller is the one author the metadata.json and it's safe to clean on commit failure. However for register table, metadata.json is usually written ahead of time (by a separate process or even in used in other catalog), so clean up might not be appropriate. Some more of the problem in https://lists.apache.org/thread/b5k7vdng904zr3n3q8wv83y8l30rnd4c

github-actions · 2025-08-16T00:17:16Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

hsiang-c · 2025-08-18T16:38:44Z

Keep alive

nastra · 2025-08-20T14:04:14Z

core/src/main/java/org/apache/iceberg/BaseMetastoreCatalog.java

    return new BaseTable(ops, fullTableName(name(), identifier), metricsReporter());
  }

+  protected void targetNamespaceExists(TableIdentifier identifier) {


We're adding the namespace check at a central place that now affects all catalogs extending this class. I think the main issue is that not every catalog actually requires an explicit namespace creation before e.g. creating a table inside a namespace or registering a table. That's also why we introduced the requiresNamespaceCreate() flag in the tests.
That being said, I don't think we can actually perform this check here as it should be specific to the respective catalog implementation and depending on whether that catalog implementation actually requires a namespace to be created or not.
For example, the default behavior of the JDBC catalog is to not require a namespace to exist when you create a table. That means registering a table should also not require for that namespace to exist beforehand.
However, if you configure strict-mode, then namespace existence is required, meaning that the targetNamespaceExists check should only be performed when strict-mode is on

@nastra Thank you for the feedback and I agree w/ you.

That's also why we introduced the requiresNamespaceCreate() flag in the tests.

I like this idea when I am working on the test.

How about we make the requiresNamespaceCreate() a default method (return false by default) in SupportsNamespaces?

Doing so:

Makes the requirement explicitly per implementation, i.e. promoting this semantics from src/test to src/main.

Skips targetNamespaceExists check if the catalog implementation doesn't require it.

nastra · 2025-08-20T14:08:35Z

dell/src/test/java/org/apache/iceberg/dell/ecs/TestEcsCatalog.java


  @Test
  public void testRegisterTable() {
+    ecsCatalog.createNamespace(Namespace.of("a"));


can you elaborate why this change is needed?

It is because

EcsCatalog implements SupportsNamespaces and the targetNamespaceExists I introduced in this PR requires an existent namespace.

TestEcsCatalog doesn't implement CatalogTests<EcsCatalog> and it doesn't know about the requiresNamespaceCreate method. Therefore, unlike the testRegisterTable in CatalogTests, this test doesn't create a namespace beforehand so I created it here.

// From CatalogTests @Test public void testRegisterTable() { C catalog = catalog(); if (requiresNamespaceCreate()) { catalog.createNamespace(TABLE.namespace()); } // omitted }

stevenzwu · 2025-08-20T17:42:02Z

It seems reasonable to throw NoSuchNamespaceException in this case for metastore catalog.

I am also thinking about @dramaticlly ' suggestion on not cleaning up metadata files for registerTable. It can probably help avoid corruption for other types of failures?

hsiang-c · 2025-08-20T19:11:19Z

@stevenzwu Thank you for your review.

not cleaning up metadata files for registerTable. It can probably help avoid corruption for other types of failures?

I agree w/ you and @dramaticlly, doing so would be helpful during catalog migration.

hantangwangd · 2025-09-16T01:46:44Z

@hsiang-c thanks for the link to PR #14083! It allowed me to catch up on the discussion here. I encountered a similar issue in a concurrent environment where the HiveCatalog is configured with lock disabled. When one operation is executing the commission of registerTable, another concurrent operation happens to commit a create-table for the same target table name. At this point, the registerTable operation fails due to an AlreadyExistsException. In this scenario, the source table becomes corrupted. This situation (along with some other cases maybe) does not seem to be entirely avoidable through pre-checks.

In PR #14083, I implemented a solution aligned with the approach mentioned above. That is, the metadata file is only deleted upon failure if it was newly created by the register table operation. Could you please take a look when you have a moment @nastra @dramaticlly @stevenzwu @hsiang-c, thanks a lot!

github-actions · 2025-10-17T00:17:12Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

hsiang-c · 2025-10-17T17:04:44Z

keep alive

github-actions · 2025-11-17T00:18:20Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

github-actions · 2025-11-24T00:19:14Z

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

Fail table registration when target namespace doesn't exist

90aa75c

github-actions bot added the spark label Jun 30, 2025

hsiang-c changed the title ~~Table registration to nonexistent target namespace leads to metadata deletion in HiveCatalog~~ Spark: Table registration to nonexistent target namespace leads to metadata deletion in HiveCatalog Jun 30, 2025

hsiang-c changed the title ~~Spark: Table registration to nonexistent target namespace leads to metadata deletion in HiveCatalog~~ Spark: Registering tables to nonexistent target namespace leads to metadata deletion in HiveCatalog Jun 30, 2025

nastra reviewed Jul 1, 2025

View reviewed changes

hsiang-c added 3 commits July 4, 2025 16:03

Revert changes to Spark procedure

aa1f797

Check namespace existence before registering tables

25871ba

Test catalog implementations not coverted by CatalogTests

a4ba226

github-actions bot added core AWS DELL SNOWFLAKE labels Jul 5, 2025

hsiang-c added 3 commits July 5, 2025 13:55

Merge branch 'main' into namespace_check

5c1f393

Align catalog client and server behavior by JDBC strict mode

6438f16

Make error messages consistent in core module

8374e71

github-actions bot added the OPENAPI label Jul 6, 2025

Turn on strict mode only in the tests of iceberg-open-api module

58e9711

github-actions bot added the build label Jul 7, 2025

hsiang-c changed the title ~~Spark: Registering tables to nonexistent target namespace leads to metadata deletion in HiveCatalog~~ Core: Registering tables to nonexistent target namespace leads to metadata deletion in HiveCatalog Jul 7, 2025

hsiang-c commented Jul 7, 2025

View reviewed changes

github-actions bot added the stale label Aug 16, 2025

github-actions bot removed the stale label Aug 19, 2025

nastra reviewed Aug 20, 2025

View reviewed changes

This was referenced Sep 17, 2025

Core: Avoid removing reused metadata file when registerTable fails #14083

Closed

[Iceberg]registerTable failures in Nessie catalog cause data corruption prestodb/presto#26098

Open

github-actions bot added the stale label Oct 17, 2025

github-actions bot removed the stale label Oct 18, 2025

github-actions bot added the stale label Nov 17, 2025

github-actions bot closed this Nov 24, 2025

Core: Registering tables to nonexistent target namespace leads to metadata deletion in HiveCatalog #13434

Core: Registering tables to nonexistent target namespace leads to metadata deletion in HiveCatalog #13434

Uh oh!

Conversation

hsiang-c commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

Proposed changes

Tests

Uh oh!

nastra Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

hsiang-c Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

hsiang-c Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

hsiang-c Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

hsiang-c Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

dramaticlly commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 16, 2025

Uh oh!

hsiang-c commented Aug 18, 2025

Uh oh!

nastra Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hsiang-c Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nastra Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

hsiang-c Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stevenzwu commented Aug 20, 2025

Uh oh!

hsiang-c commented Aug 20, 2025

Uh oh!

hantangwangd commented Sep 16, 2025

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

hsiang-c commented Oct 17, 2025

Uh oh!

github-actions bot commented Nov 17, 2025

Uh oh!

github-actions bot commented Nov 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

hsiang-c commented Jun 30, 2025 •

edited

Loading

dramaticlly commented Jul 16, 2025 •

edited

Loading

nastra Aug 20, 2025 •

edited

Loading

hsiang-c Aug 20, 2025 •

edited

Loading

hsiang-c Aug 20, 2025 •

edited

Loading