BigQuery: Add table validity check for BigQueryMetastoreCatalog #14113

thomaschow · 2025-09-19T02:52:23Z

This PR fixes a table validity check for the BigQueryMetastoreCatalog introduced in #12808. We need this validity check in order to access metadata tables using the syntax documented in: https://iceberg.apache.org/docs/1.10.0/docs/hive/?h=metadata#querying-metadata-tables. For example inspecting metadata tables. This is similar to the HiveCatalog implementation. Without this check Iceberg will attempt to load a regular table and fail in the BigQueryMetastoreCatalog implementation because the namespace has more than one level.

Added tests as well.

nastra · 2025-09-19T14:02:48Z

bigquery/src/test/java/org/apache/iceberg/gcp/bigquery/TestBigQueryCatalog.java

+  }
+
+  @Test
+  public void testLoadMetadataTableIsCalled() {


not sure I follow why this is being tested here, can you elaborate please?

I want to test that loading a metadata table will actually happen with the fix in place, since that's technically a valid case of specifying an invalid table name.

I am bit confused too, if the goal is that that the metadata table loading works correctly post this fix, shouldnt we enable this test instead ?

iceberg/bigquery/src/test/java/org/apache/iceberg/gcp/bigquery/TestBigQueryCatalog.java

Lines 147 to 149 in a73a5c0

@Disabled("BigQuery Metastore does not support multi layer namespaces")

@Test

public void testLoadMetadataTable() {}

I will move the test case there!

any reason why can't re-use the same test ? as if just enable this test from the base class it test if we can read the Files metadata table ?

yeah I agree, we should be able to remove this test method entirely and rely on the testing behavior defined in CatalogTests. The reason this method existed here is so that it could be disabled

good catch, heres the test with the same name in CatalogTests.java

iceberg/core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java

Lines 929 to 950 in a73a5c0

@Test

public void testLoadMetadataTable() {

C catalog = catalog();

TableIdentifier tableIdent = TableIdentifier.of("ns", "tbl");

TableIdentifier metaIdent = TableIdentifier.of("ns", "tbl", "files");

if (requiresNamespaceCreate()) {

catalog.createNamespace(tableIdent.namespace());

}

catalog.buildTable(tableIdent, SCHEMA).create();

Table table = catalog.loadTable(metaIdent);

assertThat(table).isNotNull();

assertThat(table).isInstanceOf(FilesTable.class);

// check that the table metadata can be refreshed

table.refresh();

assertThat(table.name()).isEqualTo(catalog.name() + "." + metaIdent);

}

Sounds good thank you.

nastra

the fix itself LGTM but I left a question for the test around metadata loading

nastra · 2025-09-19T14:04:08Z

/cc @talatuyarer

kevinjqliu

LGTM! Thanks @thomaschow :)

## Summary - Iceberg 1.10's BQMS has a bug that's fixed in: apache/iceberg#14113. Since this isn't included in the 1.10 release we'll build the jar locally and include it in `cloud_gcp` module for now. ## Checklist - [ ] Added Unit Tests - [ ] Covered by existing CI - [ ] Integration tested - [ ] Documentation update   ## Summary by CodeRabbit - New Features - Bundled an updated Iceberg BigQuery integration JAR on the classpath by default. - Bug Fixes - Improved identifier handling when namespaces are empty, reducing errors in BigQuery catalog operations. - Tests - Added end-to-end checks for partition parsing, namespace compatibility, table reachability, and partition spec validation in BigQuery catalog flows. - Chores - Removed the previous external dependency and stopped preloading additional Dataproc JARs by default. - Style - Minor string formatting cleanup with no behavioral changes.  --------- Co-authored-by: thomaschow <[email protected]>

bigquery/src/test/java/org/apache/iceberg/gcp/bigquery/TestBigQueryCatalog.java

…QueryCatalog.java

bigquery/src/test/java/org/apache/iceberg/gcp/bigquery/TestBigQueryCatalog.java

…QueryCatalog.java

nastra · 2025-09-19T18:24:37Z

bigquery/src/test/java/org/apache/iceberg/gcp/bigquery/TestBigQueryCatalog.java

+
+  @Test
+  public void testIsValidIdentifierWithInvalidMultiLevelNamespace() {
+    TableIdentifier invalidIdentifier =


nit: all of these can be inlined

done, formatting creates multiple lines for a few of the more verbose cases.

bigquery/src/test/java/org/apache/iceberg/gcp/bigquery/TestBigQueryCatalog.java

kevinjqliu · 2025-09-19T20:46:34Z

Thanks @thomaschow and thanks everyone for the review!

fix: Add table validity check for BigQueryMetastoreCatalog

bef7569

github-actions bot added the GCP label Sep 19, 2025

lint

b050563

thomaschow changed the title ~~fix: Add table validity check for BigQueryMetastoreCatalog~~ Catalog: Add table validity check for BigQueryMetastoreCatalog Sep 19, 2025

nastra reviewed Sep 19, 2025

View reviewed changes

nastra approved these changes Sep 19, 2025

View reviewed changes

nastra changed the title ~~Catalog: Add table validity check for BigQueryMetastoreCatalog~~ BigQuery: Add table validity check for BigQueryMetastoreCatalog Sep 19, 2025

tchow-zlai mentioned this pull request Sep 19, 2025

fix: Bring in snapshot of iceberg-bigquery jar for a fix zipline-ai/chronon#1161

Merged

4 tasks

move test

9a2e653

kevinjqliu approved these changes Sep 19, 2025

View reviewed changes

nikhil-zlai approved these changes Sep 19, 2025

View reviewed changes

nastra reviewed Sep 19, 2025

View reviewed changes

bigquery/src/test/java/org/apache/iceberg/gcp/bigquery/TestBigQueryCatalog.java Outdated Show resolved Hide resolved

Update bigquery/src/test/java/org/apache/iceberg/gcp/bigquery/TestBig…

acddd8f

…QueryCatalog.java

nastra reviewed Sep 19, 2025

View reviewed changes

bigquery/src/test/java/org/apache/iceberg/gcp/bigquery/TestBigQueryCatalog.java Outdated Show resolved Hide resolved

Apply suggestions from code review

8a33121

nastra reviewed Sep 19, 2025

View reviewed changes

bigquery/src/test/java/org/apache/iceberg/gcp/bigquery/TestBigQueryCatalog.java Show resolved Hide resolved

Update bigquery/src/test/java/org/apache/iceberg/gcp/bigquery/TestBig…

c9979c1

…QueryCatalog.java

nastra reviewed Sep 19, 2025

View reviewed changes

bigquery/src/test/java/org/apache/iceberg/gcp/bigquery/TestBigQueryCatalog.java Outdated Show resolved Hide resolved

nastra reviewed Sep 19, 2025

View reviewed changes

bigquery/src/test/java/org/apache/iceberg/gcp/bigquery/TestBigQueryCatalog.java Outdated Show resolved Hide resolved

nastra reviewed Sep 19, 2025

View reviewed changes

bigquery/src/test/java/org/apache/iceberg/gcp/bigquery/TestBigQueryCatalog.java Outdated Show resolved Hide resolved

nastra and others added 2 commits September 19, 2025 20:26

Apply suggestions from code review

16a835a

review feedback

056da20

kevinjqliu approved these changes Sep 19, 2025

View reviewed changes

kevinjqliu merged commit 998eb87 into apache:main Sep 19, 2025
42 checks passed

	@Disabled("BigQuery Metastore does not support multi layer namespaces")
	@Test
	public void testLoadMetadataTable() {}

	@Test
	public void testLoadMetadataTable() {
	C catalog = catalog();

	TableIdentifier tableIdent = TableIdentifier.of("ns", "tbl");
	TableIdentifier metaIdent = TableIdentifier.of("ns", "tbl", "files");

	if (requiresNamespaceCreate()) {
	catalog.createNamespace(tableIdent.namespace());
	}

	catalog.buildTable(tableIdent, SCHEMA).create();

	Table table = catalog.loadTable(metaIdent);
	assertThat(table).isNotNull();
	assertThat(table).isInstanceOf(FilesTable.class);

	// check that the table metadata can be refreshed
	table.refresh();

	assertThat(table.name()).isEqualTo(catalog.name() + "." + metaIdent);
	}

BigQuery: Add table validity check for BigQueryMetastoreCatalog #14113

BigQuery: Add table validity check for BigQueryMetastoreCatalog #14113

Uh oh!

Conversation

thomaschow commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nastra left a comment

Choose a reason for hiding this comment

Uh oh!

nastra commented Sep 19, 2025

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kevinjqliu commented Sep 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

thomaschow commented Sep 19, 2025 •

edited

Loading