Skip to content

feat: BigQuery views #698

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Apr 29, 2025
Merged

feat: BigQuery views #698

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
194b5d1
view
tchow-zlai Apr 29, 2025
d034ead
rename the column variable
tchow-zlai Apr 29, 2025
4f181cb
fmt
tchow-zlai Apr 29, 2025
a23d8fe
Merge branch 'main' into tchow/views
tchow-zlai Apr 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions cloud_gcp/src/main/scala/ai/chronon/integrations/cloud_gcp/BigQueryNative.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ case object BigQueryNative extends Format {
private val bqFormat = classOf[Spark35BigQueryTableProvider].getName
private lazy val bqOptions = BigQueryOptions.getDefaultInstance

private val internalBQCol = "__chronon_internal_bq_col__"
private val internalBQPartitionCol = "__chronon_internal_bq_partition_col__"

// TODO(tchow): use the cache flag
override def table(tableName: String, partitionFilters: String, cacheDf: Boolean = false)(implicit
Expand Down Expand Up @@ -50,13 +50,14 @@ case object BigQueryNative extends Format {
// Next, we query the BQ table using the requested partitionFilter to grab all the distinct partition values that match the filter.
val partitionWheres = if (partitionFilters.nonEmpty) s"WHERE ${partitionFilters}" else partitionFilters
val partitionFormat = TableUtils(sparkSession).partitionFormat
val select = s"SELECT distinct(${partColName}) AS ${internalBQCol} FROM ${bqFriendlyName} ${partitionWheres}"
val select =
s"SELECT distinct(${partColName}) AS ${internalBQPartitionCol} FROM ${bqFriendlyName} ${partitionWheres}"
val selectedParts = sparkSession.read
.format(bqFormat)
.option("viewsEnabled", true)
.option("materializationDataset", bqTableId.getDataset)
.load(select)
.select(date_format(col(internalBQCol), partitionFormat))
.select(date_format(col(internalBQPartitionCol), partitionFormat))
.as[String]
.collect
.toList
Expand Down
4 changes: 4 additions & 0 deletions ...src/main/scala/ai/chronon/integrations/cloud_gcp/DelegatingBigQueryMetastoreCatalog.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import com.google.cloud.bigquery.{
BigQueryOptions,
ExternalTableDefinition,
StandardTableDefinition,
ViewDefinition,
TableDefinition,
TableId
}
Expand Down Expand Up @@ -95,6 +96,9 @@ class DelegatingBigQueryMetastoreCatalog extends TableCatalog with SupportsNames
.Option(bigQueryClient.getTable(tId))
.getOrElse(throw new NoSuchTableException(s"BigQuery table $identNoCatalog not found."))
table.getDefinition.asInstanceOf[TableDefinition] match {
case view: ViewDefinition => {
connectorCatalog.loadTable(Identifier.of(Array(tId.getDataset), tId.getTable))
}
case externalTable: ExternalTableDefinition => {
val uris = externalTable.getSourceUris.asScala
val uri = scala
Expand Down