Skip to content

fix: No longer support BigQueryNative table loads #1035

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

fix: No longer support BigQueryNative table loads #1035

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 3 additions & 5 deletions cloud_gcp/src/main/scala/ai/chronon/integrations/cloud_gcp/GcpFormatProvider.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@ package ai.chronon.integrations.cloud_gcp

import ai.chronon.spark.catalog.{DefaultFormatProvider, Format, Iceberg}
import com.google.cloud.bigquery._
import com.google.cloud.spark.bigquery.v2.Spark31BigQueryTable
import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog
import org.apache.iceberg.spark.SparkCatalog
import org.apache.iceberg.spark.source.SparkTable
Expand Down Expand Up @@ -31,10 +30,9 @@ class GcpFormatProvider(override val sparkSession: SparkSession) extends Default
Try {
val tbl = delegating.loadTable(identifier)
tbl match {
case iceberg: SparkTable => Iceberg
case bigquery: Spark31BigQueryTable => BigQueryNative
case parquet: ParquetTable => BigQueryExternal
case unsupported => throw new IllegalStateException(s"Unsupported provider type: ${unsupported}")
case iceberg: SparkTable => Iceberg
case parquet: ParquetTable => BigQueryExternal
case unsupported => throw new IllegalStateException(s"Unsupported provider type: ${unsupported}")
Comment on lines +33 to +35
Copy link
Contributor

@coderabbitai coderabbitai bot Aug 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

🧩 Analysis chain

Verify removal of BigQueryNative support doesn't break existing functionality.

Pattern matching logic correctly handles remaining formats. Confirm all BigQueryNative table references are migrated to Import() staging queries.

🏁 Script executed:

#!/bin/bash
# Search for any remaining references to BigQueryNative or Spark31BigQueryTable
rg -i "bigquerynative|spark31bigquerytable" --type scala -A 3 -B 3

Length of output: 2481

Remove remaining BigQueryNative artifacts

The GcpFormatProvider no longer maps to BigQueryNative, but the codebase still defines and tests it, leading to broken builds:

• cloud_gcp/src/main/scala/ai/chronon/integrations/cloud_gcp/BigQueryNative.scala
– File and its case classes/objects are now unused.

• cloud_gcp/src/test/scala/ai/chronon/integrations/cloud_gcp/BigQueryCatalogTest.scala
– Lines asserting Some(BigQueryNative) for native tables must be updated or removed.

Please delete the BigQueryNative.scala file and adjust or remove these tests to reflect the new behavior (either expect an exception or map native tables to BigQueryExternal).

🤖 Prompt for AI Agents 
In
cloud_gcp/src/main/scala/ai/chronon/integrations/cloud_gcp/GcpFormatProvider.scala
around lines 33 to 35, remove all references to BigQueryNative since it is no
longer supported. Delete the entire BigQueryNative.scala file and update or
remove tests in
cloud_gcp/src/test/scala/ai/chronon/integrations/cloud_gcp/BigQueryCatalogTest.scala
that expect BigQueryNative, changing them to expect exceptions or mapping native
tables to BigQueryExternal as appropriate.

}
} match {
case s @ Success(_) => s.toOption
Expand Down
Loading