Catalog: Add BigQuery Metastore Catalog Support #12808
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
|
@nastra Would you like to review a brand new Catalog which uses CatalogTests 😃 |
nastra
reviewed
Apr 16, 2025
kravikumar
reviewed
Apr 16, 2025
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetaStoreClient.java
Outdated
Show resolved
Hide resolved
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetaStoreClientImpl.java
Outdated
Show resolved
Hide resolved
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java
Outdated
Show resolved
Hide resolved
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetaStoreClientImpl.java
Outdated
Show resolved
Hide resolved
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetaStoreClientImpl.java
Outdated
Show resolved
Hide resolved
gkalra18
reviewed
Apr 16, 2025
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetaStoreClientImpl.java
Outdated
Show resolved
Hide resolved
gkalra18
reviewed
Apr 16, 2025
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java
Outdated
Show resolved
Hide resolved
ebyhr
reviewed
Apr 17, 2025
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetaStoreClient.java
Outdated
Show resolved
Hide resolved
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetaStoreClientImpl.java
Outdated
Show resolved
Hide resolved
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetaStoreClientImpl.java
Outdated
Show resolved
Hide resolved
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetaStoreClientImpl.java
Outdated
Show resolved
Hide resolved
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetaStoreClient.java
Outdated
Show resolved
Hide resolved
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryTableOperations.java
Outdated
Show resolved
Hide resolved
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/FakeBigQueryMetaStoreClient.java
Outdated
Show resolved
Hide resolved
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/FakeBigQueryMetaStoreClient.java
Outdated
Show resolved
Hide resolved
bigquery/src/test/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreTestUtils.java
Outdated
Show resolved
Hide resolved
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetaStoreClientImpl.java
Outdated
Show resolved
Hide resolved
Contributor
Author
|
Thank you all @nastra @ebyhr @kravikumar @gkalra18 for your review. I addressed all your comments feel free to add more :) |
nastra
reviewed
Apr 22, 2025
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java
Outdated
Show resolved
Hide resolved
nastra
reviewed
Apr 22, 2025
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java
Outdated
Show resolved
Hide resolved
nastra
reviewed
Apr 22, 2025
nastra
reviewed
Apr 22, 2025
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java
Outdated
Show resolved
Hide resolved
nastra
reviewed
Apr 22, 2025
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java
Outdated
Show resolved
Hide resolved
nastra
reviewed
Apr 22, 2025
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreClientImpl.java
Outdated
Show resolved
Hide resolved
nastra
reviewed
Apr 22, 2025
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreUtils.java
Outdated
Show resolved
Hide resolved
nastra
reviewed
Apr 22, 2025
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java
Outdated
Show resolved
Hide resolved
nastra
reviewed
Apr 22, 2025
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java
Outdated
Show resolved
Hide resolved
…viour some signature access modifiers changes on Classes
ebyhr
reviewed
May 6, 2025
ebyhr
reviewed
May 6, 2025
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreClientImpl.java
Show resolved
Hide resolved
ebyhr
reviewed
May 6, 2025
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java
Show resolved
Hide resolved
nastra
reviewed
May 9, 2025
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java
Outdated
Show resolved
Hide resolved
nastra
reviewed
May 12, 2025
Contributor
nastra
left a comment
nastra
left a comment
There was a problem hiding this comment.
this LGTM once the remaining comments from @danielcweeks have been addressed as I think we need to fix those
Contributor
|
Is there any thought to make this catalog implementation available in the kafka connect runtime? I guess by adding it here (not sure if it's the only thing to do) |
…id redundant API calls when setting/removing dataset properties.
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreClientImpl.java
Outdated
Show resolved
Hide resolved
…doc to accurately reflect the new behavior.
3 tasks
danielcweeks
approved these changes
May 13, 2025
Contributor
danielcweeks
left a comment
danielcweeks
left a comment
There was a problem hiding this comment.
Thanks @talatuyarer, I think this is good to go.
danielcweeks
pushed a commit
to danielcweeks/iceberg
that referenced
this pull request
May 14, 2025
* Initial Commit for BigQuery MetaStore
3 tasks
|
Hello @stevenzwu I mention you because I saw you handling the 1.10.0 milestone and I wanted to ask if this PR will be included in the 1.10.0 version |
Contributor
|
@locchipinti yes this will be shipped with the next release |
Fokko
pushed a commit
to apache/iceberg-python
that referenced
this pull request
Aug 26, 2025
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace
${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->
# Rationale for this change
This PR brings BigQuery Metastore support to Python after it was merged
into the [Java
implementation](apache/iceberg#12808).
This allows Iceberg catalog functionality to be backed by BigQuery. It
supports creating/deleting/listing namespaces (datasets in BigQuery
terminology), creating/deleting/listing tables, and registering tables.
This is my first PR of size to iceberg-python, so any advice would be
appreciated!
# Are these changes tested?
Integration and unit tests included.
# Are there any user-facing changes?
Introduces a new Catalog type.
<!-- In the case of user-facing changes, please add the changelog label.
-->
4 tasks
github-merge-queue bot
pushed a commit
to zipline-ai/chronon
that referenced
this pull request
Sep 22, 2025
## Summary - Upgrade to iceberg 1.10.0 to grab column [stats](apache/iceberg#10659), and some CVE's: [CVE](apache/iceberg#13561) (and parquet, avro transitively), and [BigQueryMetastoreCatalog](apache/iceberg#12808), [Google Auth](apache/iceberg#13212). - Column stats is the key feature here - we rely on extracting the puffin files and grabbing stats metadata. <img width="1342" height="461" alt="Screenshot 2025-09-20 at 4 30 35 PM" src="https://github.com/user-attachments/assets/bc8eeb80-6ff7-4abe-8ffb-a0eebf48bc4e" /> ## Checklist - [ ] Added Unit Tests - [ ] Covered by existing CI - [ ] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Per-partition, per-column statistics extraction with optional persistence to a new data-quality metrics KV store; platform APIs can produce a metrics-specific KV store. * **Breaking Changes** * Extraction API signatures and summary/key formats changed; thrift summary shapes updated; config token renamed "groupbys" → "group_bys"; /api/summary-series now returns null. * **Refactor** * Large-scale test package reorganizations and import consolidations across the codebase. <!-- end of auto-generated comment: release notes by coderabbit.ai --> <!-- av pr metadata This information is embedded by the av CLI when creating PRs to track the status of stacks when using Aviator. Please do not delete or edit this section of the PR. ``` {"parent":"main","parentHead":"","trunk":"main"} ``` --> --------- Co-authored-by: thomaschow <[email protected]>
devendra-nr
pushed a commit
to devendra-nr/iceberg
that referenced
this pull request
Dec 8, 2025
* Initial Commit for BigQuery MetaStore * Spotless Fix on Core module * Addressed comments on PR from @nastra * Addressed comments on PR from @ebyhr and @gkalra18 * Addressed comments on PR from @nastra * Removed PROJECT_ID from GCPProperties class. * Removed BIGQUERY_LOCATION from GCPProperties class. * Addressed @nastra and @amogh-jahagirdar comments * Changed Catalog initialization, removed TESTING_ENABLED property and removed optional field tests from BigQueryCatalogTests * Used toTableReference in newTableOps method. Dropped Dataset name from Client Interface's methods, Disabled testLoadMetadataTable test due to multi level namespacing. * Fixed failed testListNonExistingNamespace * Fixed assertThat * Removed BigQueryMetastoreTestUtils class and addressed @nastra's comment on pr * Addressed comments from nastra and danielcweeks * Missing changes from previous update * last @SuppressWarnings("FormatStringAnnotation") * last @SuppressWarnings("FormatStringAnnotation") * Addressed Latest Comments from @danielcweeks about exception types * Addressed Latest Comments from @danielcweeks about serialVersionUID and interface * Addressed Latest Comments from @danielcweeks about Listnamespace behaviour some signature access modifiers changes on Classes * Removed Hive and Hadoop dependencies from BigQuery Catalog * Removed double condition in listNamespaces. Thank you @nastra * Optimize Dataset property updates. Moves logic into the client to avoid redundant API calls when setting/removing dataset properties. * Renamed `filterUnsupportedTables` to `listAllTables` and updated Javadoc to accurately reflect the new behavior.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses comments from PR's that introduces initial support for using Google BigQuery as a Metastore Catalog for Apache Iceberg. Unfortunately creator of the PR is on leave. I had to recreate a new PR for that.
Key changes include:
BigQueryMetaStoreClientand related classes to interact with the BigQuery API for Metastore operations.BigQueryMetastoreCatalogto provide Iceberg catalog functionality backed by BigQuery.