feat: GCPFormatProvider should do the default checks first #617
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: Thomas Chow <[email protected]>
Contributor
WalkthroughThe changes update the Changes
Possibly related PRs
Suggested reviewers
Poem
Warning Review ran into problems🔥 ProblemsGitHub Actions and Pipeline Checks: Resource not accessible by integration - https://docs.github.com/rest/actions/workflow-runs#list-workflow-runs-for-a-repository. Please grant the required permissions to the CodeRabbit GitHub App under the organization or repository settings. 📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (1)
⏰ Context from checks skipped due to timeout of 90000ms (18)
🔇 Additional comments (2)
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
cloud_gcp/src/main/scala/ai/chronon/integrations/cloud_gcp/GcpFormatProvider.scala (2)
12-12: Prefer a non-case class.
case classis usually for simple data holders. Consider a standard class to avoid unintended copying.Proposed diff:
- case class GcpFormatProvider(override val sparkSession: SparkSession) extends DefaultFormatProvider(sparkSession) { + class GcpFormatProvider(override val sparkSession: SparkSession) extends DefaultFormatProvider(sparkSession) {
25-33: Add logging and null safety.
- Suggest a debug log indicating fallback usage.
- Consider handling null from
bigQueryClient.getTable(btTableIdentifier)to avoid potential NPE.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro (Legacy)
📒 Files selected for processing (1)
cloud_gcp/src/main/scala/ai/chronon/integrations/cloud_gcp/GcpFormatProvider.scala(3 hunks)
🧰 Additional context used
🧠 Learnings (1)
cloud_gcp/src/main/scala/ai/chronon/integrations/cloud_gcp/GcpFormatProvider.scala (2)
Learnt from: tchow-zlai
PR: zipline-ai/chronon#263
File: cloud_gcp/src/main/scala/ai/chronon/integrations/cloud_gcp/BigQueryFormat.scala:29-60
Timestamp: 2025-04-09T21:40:05.504Z
Learning: In BigQuery integration, table existence check is performed outside the BigQueryFormat.createTable method, at a higher level in TableUtils.createTable.
Learnt from: tchow-zlai
PR: zipline-ai/chronon#263
File: cloud_gcp/src/main/scala/ai/chronon/integrations/cloud_gcp/BigQueryFormat.scala:56-57
Timestamp: 2025-04-09T21:40:05.505Z
Learning: For BigQuery table creation operations in BigQueryFormat.scala, allow exceptions to propagate directly without wrapping them in try-catch blocks, as the original BigQuery exceptions provide sufficient context.
🧬 Code Graph Analysis (1)
cloud_gcp/src/main/scala/ai/chronon/integrations/cloud_gcp/GcpFormatProvider.scala (2)
spark/src/main/scala/ai/chronon/spark/format/DefaultFormatProvider.scala (2)
DefaultFormatProvider(12-53)readFormat(17-25)cloud_gcp/src/main/scala/ai/chronon/integrations/cloud_gcp/SparkBQUtils.scala (2)
SparkBQUtils(6-17)toTableId(8-15)
⏰ Context from checks skipped due to timeout of 90000ms (4)
- GitHub Check: non_spark_tests
- GitHub Check: non_spark_tests
- GitHub Check: scala_compile_fmt_fix
- GitHub Check: enforce_triggered_workflows
🔇 Additional comments (1)
cloud_gcp/src/main/scala/ai/chronon/integrations/cloud_gcp/GcpFormatProvider.scala (1)
2-2: OK.Imports look correct and minimal.
Co-authored-by: Thomas Chow <[email protected]>
Co-authored-by: Thomas Chow <[email protected]>
Co-authored-by: Thomas Chow <[email protected]>
Co-authored-by: Thomas Chow <[email protected]>
Co-authored-by: Thomas Chow <[email protected]>
Co-authored-by: Thomas Chow <[email protected]>
Co-authored-by: Thomas Chow <[email protected]>
Co-authored-by: Thomas Chow <[email protected]>
tchow-zlai
changed the title
piyush-zlai
approved these changes
Apr 10, 2025
Collaborator
Author
|
I will follow up on this to make this more robust - likely depend on the spark catalog name. This is a temp workaround |
This was referenced Apr 10, 2025
kumar-zlai
pushed a commit
that referenced
this pull request
Apr 25, 2025
## Summary
- Now that we use Polaris for etsy, there's a case where we don't go to
BigQuery to find a table. We'll need to bring in the normal iceberg
check to do that.
## Checklist
- [ ] Added Unit Tests
- [ ] Covered by existing CI
- [ ] Integration tested
- [ ] Documentation update
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Refactor**
- Enhanced the cloud integration’s data formatting process by
streamlining the resolution strategy, checking for pre-existing formats
before applying a fallback.
- Improved accessibility of the `sparkSession` parameter in the
`DefaultFormatProvider` class for direct access from instances.
- Improved overall consistency and efficiency in data handling for
cloud-based operations.
- Updated the instantiation method for the `GcpFormatProvider` class in
tests for clarity.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- av pr metadata
This information is embedded by the av CLI when creating PRs to track
the status of stacks when using Aviator. Please do not delete or edit
this section of the PR.
```
{"parent":"main","parentHead":"","trunk":"main"}
```
-->
---------
Co-authored-by: Thomas Chow <[email protected]>
kumar-zlai
pushed a commit
that referenced
this pull request
Apr 29, 2025
## Summary
- Now that we use Polaris for etsy, there's a case where we don't go to
BigQuery to find a table. We'll need to bring in the normal iceberg
check to do that.
## Checklist
- [ ] Added Unit Tests
- [ ] Covered by existing CI
- [ ] Integration tested
- [ ] Documentation update
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Refactor**
- Enhanced the cloud integration’s data formatting process by
streamlining the resolution strategy, checking for pre-existing formats
before applying a fallback.
- Improved accessibility of the `sparkSession` parameter in the
`DefaultFormatProvider` class for direct access from instances.
- Improved overall consistency and efficiency in data handling for
cloud-based operations.
- Updated the instantiation method for the `GcpFormatProvider` class in
tests for clarity.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- av pr metadata
This information is embedded by the av CLI when creating PRs to track
the status of stacks when using Aviator. Please do not delete or edit
this section of the PR.
```
{"parent":"main","parentHead":"","trunk":"main"}
```
-->
---------
Co-authored-by: Thomas Chow <[email protected]>
chewy-zlai
pushed a commit
that referenced
this pull request
May 15, 2025
## Summary
- Now that we use Polaris for our clients, there's a case where we don't go to
BigQuery to find a table. We'll need to bring in the normal iceberg
check to do that.
## Checklist
- [ ] Added Unit Tests
- [ ] Covered by existing CI
- [ ] Integration tested
- [ ] Documentation update
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Refactor**
- Enhanced the cloud integration’s data formatting process by
streamlining the resolution strategy, checking for pre-existing formats
before applying a fallback.
- Improved accessibility of the `sparkSession` parameter in the
`DefaultFormatProvider` class for direct access from instances.
- Improved overall consistency and efficiency in data handling for
cloud-based operations.
- Updated the instantiation method for the `GcpFormatProvider` class in
tests for clarity.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- av pr metadata
This information is embedded by the av CLI when creating PRs to track
the status of stacks when using Aviator. Please do not delete or edit
this section of the PR.
```
{"parent":"main","parentHead":"","trunk":"main"}
```
-->
---------
Co-authored-by: Thomas Chow <[email protected]>
chewy-zlai
pushed a commit
that referenced
this pull request
May 15, 2025
## Summary
- Now that we use Polaris for our clients, there's a case where we don't go to
BigQuery to find a table. We'll need to bring in the normal iceberg
check to do that.
## Checklist
- [ ] Added Unit Tests
- [ ] Covered by existing CI
- [ ] Integration tested
- [ ] Documentation update
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Refactor**
- Enhanced the cloud integration’s data formatting process by
streamlining the resolution strategy, checking for pre-existing formats
before applying a fallback.
- Improved accessibility of the `sparkSession` parameter in the
`DefaultFormatProvider` class for direct access from instances.
- Improved overall consistency and efficiency in data handling for
cloud-based operations.
- Updated the instantiation method for the `GcpFormatProvider` class in
tests for clarity.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- av pr metadata
This information is embedded by the av CLI when creating PRs to track
the status of stacks when using Aviator. Please do not delete or edit
this section of the PR.
```
{"parent":"main","parentHead":"","trunk":"main"}
```
-->
---------
Co-authored-by: Thomas Chow <[email protected]>
chewy-zlai
pushed a commit
that referenced
this pull request
May 16, 2025
…617) ## Summary - Now that we use Polaris for our clients, there's a case where we don't go to BigQuery to find a table. We'll need to bring in the normal iceberg cheour clients to do that. ## Cheour clientslist - [ ] Added Unit Tests - [ ] Covered by existing CI - [ ] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Refactor** - Enhanced the cloud integration’s data formatting process by streamlining the resolution strategy, cheour clientsing for pre-existing formats before applying a fallbaour clients. - Improved accessibility of the `sparkSession` parameter in the `DefaultFormatProvider` class for direct access from instances. - Improved overall consistency and efficiency in data handling for cloud-based operations. - Updated the instantiation method for the `GcpFormatProvider` class in tests for clarity. <!-- end of auto-generated comment: release notes by coderabbit.ai --> <!-- av pr metadata This information is embedded by the av CLI when creating PRs to traour clients the status of staour clientss when using Aviator. Please do not delete or edit this section of the PR. ``` {"parent":"main","parentHead":"","trunk":"main"} ``` --> --------- Co-authored-by: Thomas Chow <[email protected]>
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Checklist
Summary by CodeRabbit
sparkSessionparameter in theDefaultFormatProviderclass for direct access from instances.GcpFormatProviderclass in tests for clarity.