feat: Refactor BatchNodeRunner for cleaner working logging setup ZIP-791 #1011
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
WalkthroughThe Changes
Sequence Diagram(s)sequenceDiagram
participant CLI
participant BatchNodeRunner
participant TableUtils
CLI->>BatchNodeRunner: Parse CLI args, load Node config JSON
CLI->>BatchNodeRunner: Instantiate BatchNodeRunner(node)
BatchNodeRunner->>TableUtils: Initialize once per instance
CLI->>BatchNodeRunner: runFromArgs(api, startDs, endDs, tablePartitionsDataset)
BatchNodeRunner->>BatchNodeRunner: Execute run logic using instance TableUtils
BatchNodeRunner->>CLI: Return exit code
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
Poem
📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (12)
✨ Finishing Touches🧪 Generate unit tests
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
tchow-zlai
force-pushed
the
tchow/logging-fix
branch
from
0b79a82 to
3b60edd
Compare
tchow-zlai
force-pushed
the
tchow/logging-fix
branch
from
3b60edd to
6fa1e6a
Compare
tchow-zlai
changed the title
tchow-zlai
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
spark/src/main/scala/ai/chronon/spark/batch/BatchNodeRunner.scala(5 hunks)spark/src/main/scala/ai/chronon/spark/kv_store/KVUploadNodeRunner.scala(1 hunks)spark/src/test/scala/ai/chronon/spark/test/batch/ShortNamesTest.scala(1 hunks)
✅ Files skipped from review due to trivial changes (2)
- spark/src/main/scala/ai/chronon/spark/kv_store/KVUploadNodeRunner.scala
- spark/src/test/scala/ai/chronon/spark/test/batch/ShortNamesTest.scala
🧰 Additional context used
🧠 Learnings (8)
📚 Learning: in `mockkvstore` located at `spark/src/test/scala/ai/chronon/spark/test/mockkvstore.scala`, the `mul...
Learnt from: chewy-zlai
PR: zipline-ai/chronon#50
File: spark/src/test/scala/ai/chronon/spark/test/MockKVStore.scala:19-28
Timestamp: 2024-10-31T18:29:45.027Z
Learning: In `MockKVStore` located at `spark/src/test/scala/ai/chronon/spark/test/MockKVStore.scala`, the `multiPut` method is intended to be a simple implementation without dataset existence validation, duplicate validation logic elimination, or actual storage of key-value pairs for verification.
Applied to files:
spark/src/main/scala/ai/chronon/spark/batch/BatchNodeRunner.scala
📚 Learning: in the `kvstore` trait located at `online/src/main/scala/ai/chronon/online/kvstore.scala`, there are...
Learnt from: chewy-zlai
PR: zipline-ai/chronon#47
File: online/src/main/scala/ai/chronon/online/MetadataStore.scala:232-0
Timestamp: 2024-10-17T00:12:09.763Z
Learning: In the `KVStore` trait located at `online/src/main/scala/ai/chronon/online/KVStore.scala`, there are two `create` methods: `def create(dataset: String): Unit` and `def create(dataset: String, props: Map[String, Any]): Unit`. The version with `props` ignores the `props` parameter, and the simpler version without `props` is appropriate when `props` are not needed.
Applied to files:
spark/src/main/scala/ai/chronon/spark/batch/BatchNodeRunner.scala
📚 Learning: in the `kvstore` trait located at `online/src/main/scala/ai/chronon/online/api.scala`, the default i...
Learnt from: piyush-zlai
PR: zipline-ai/chronon#33
File: online/src/main/scala/ai/chronon/online/Api.scala:69-69
Timestamp: 2024-10-08T16:18:45.669Z
Learning: In the `KVStore` trait located at `online/src/main/scala/ai/chronon/online/Api.scala`, the default implementation of the `create` method (`def create(dataset: String, props: Map[String, Any]): Unit = create(dataset)`) doesn't leverage the `props` parameter, but subclasses like `DynamoDBKVStoreImpl` use the `props` parameter in their overridden implementations.
Applied to files:
spark/src/main/scala/ai/chronon/spark/batch/BatchNodeRunner.scala
📚 Learning: in the codebase, the `kvstore` implementation provides an implicit `executioncontext` in scope, so i...
Learnt from: piyush-zlai
PR: zipline-ai/chronon#33
File: cloud_aws/src/main/scala/ai/chronon/integrations/aws/DynamoDBKVStoreImpl.scala:29-30
Timestamp: 2024-10-08T16:18:45.669Z
Learning: In the codebase, the `KVStore` implementation provides an implicit `ExecutionContext` in scope, so it's unnecessary to import another.
Applied to files:
spark/src/main/scala/ai/chronon/spark/batch/BatchNodeRunner.scala
📚 Learning: hardcoding future timestamps in tests within `dynamodbkvstoretest.scala` is acceptable when data is ...
Learnt from: piyush-zlai
PR: zipline-ai/chronon#33
File: cloud_aws/src/test/scala/ai/chronon/integrations/aws/DynamoDBKVStoreTest.scala:175-175
Timestamp: 2024-10-07T15:09:51.567Z
Learning: Hardcoding future timestamps in tests within `DynamoDBKVStoreTest.scala` is acceptable when data is generated and queried within the same time range, ensuring the tests remain valid over time.
Applied to files:
spark/src/main/scala/ai/chronon/spark/batch/BatchNodeRunner.scala
📚 Learning: the bigtablekvstoreimpl in the chronon codebase only interacts with 4 bigtable tables total, so unbo...
Learnt from: piyush-zlai
PR: zipline-ai/chronon#657
File: cloud_gcp/src/main/scala/ai/chronon/integrations/cloud_gcp/BigTableKVStoreImpl.scala:93-97
Timestamp: 2025-04-21T15:10:40.819Z
Learning: The BigTableKVStoreImpl in the chronon codebase only interacts with 4 BigTable tables total, so unbounded caching in tableToContext is not a concern.
Applied to files:
spark/src/main/scala/ai/chronon/spark/batch/BatchNodeRunner.scala
📚 Learning: in bigquery integration, table existence check is performed outside the bigqueryformat.createtable m...
Learnt from: tchow-zlai
PR: zipline-ai/chronon#263
File: cloud_gcp/src/main/scala/ai/chronon/integrations/cloud_gcp/BigQueryFormat.scala:29-60
Timestamp: 2025-01-24T23:55:30.256Z
Learning: In BigQuery integration, table existence check is performed outside the BigQueryFormat.createTable method, at a higher level in TableUtils.createTable.
Applied to files:
spark/src/main/scala/ai/chronon/spark/batch/BatchNodeRunner.scala
📚 Learning: for bigquery table creation operations in bigqueryformat.scala, allow exceptions to propagate direct...
Learnt from: tchow-zlai
PR: zipline-ai/chronon#263
File: cloud_gcp/src/main/scala/ai/chronon/integrations/cloud_gcp/BigQueryFormat.scala:56-57
Timestamp: 2025-01-24T23:55:40.650Z
Learning: For BigQuery table creation operations in BigQueryFormat.scala, allow exceptions to propagate directly without wrapping them in try-catch blocks, as the original BigQuery exceptions provide sufficient context.
Applied to files:
spark/src/main/scala/ai/chronon/spark/batch/BatchNodeRunner.scala
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (17)
- GitHub Check: streaming_tests
- GitHub Check: analyzer_tests
- GitHub Check: fetcher_tests
- GitHub Check: join_tests
- GitHub Check: cloud_aws_tests
- GitHub Check: batch_tests
- GitHub Check: aggregator_tests
- GitHub Check: spark_tests
- GitHub Check: online_tests
- GitHub Check: flink_tests
- GitHub Check: service_tests
- GitHub Check: service_commons_tests
- GitHub Check: cloud_gcp_tests
- GitHub Check: api_tests
- GitHub Check: scala_compile_fmt_fix
- GitHub Check: python_tests
- GitHub Check: enforce_triggered_workflows
🔇 Additional comments (8)
spark/src/main/scala/ai/chronon/spark/batch/BatchNodeRunner.scala (8)
49-53: Good refactoring to class-based architecture.Converting from singleton object to class with
Nodeparameter improves testability and dependency injection. CentralizingTableUtilscreation is clean.
55-87: Method signature simplified appropriately.Removing
tableUtilsparameter makes sense since it's now an instance variable. Logic remains unchanged.
89-102: Clean parameter removal.Method now uses instance
tableUtilsinstead of parameter. Implementation unchanged.
104-111: Consistent with refactoring pattern.Parameter removal and use of instance variable aligns with overall design changes.
113-138: Parameter cleanup maintained.Removal of
tableUtilsparameter consistent with class-based approach.
140-171: Method signature updated correctly.Removed redundant parameters, logic flow preserved. Good encapsulation.
173-255: Simplified method interface.Removing
confPathparameter and using constructor-providednodeis cleaner. Core logic unchanged.
258-276: Well-structured companion object.Main method properly handles configuration loading and runner instantiation. Good separation of concerns.
spark/src/main/scala/ai/chronon/spark/batch/BatchNodeRunner.scala
Outdated
Show resolved
Hide resolved
tchow-zlai
requested review from
david-zlai,
nikhil-zlai,
piyush-zlai and
varant-zlai
tchow-zlai
force-pushed
the
tchow/logging-fix
branch
from
2663a91 to
abdccd1
Compare
david-zlai
approved these changes
Jul 31, 2025
This was referenced Jul 31, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Results after this PR showing the logging:
Checklist
Summary by CodeRabbit
Summary by CodeRabbit