Driver Summarizer #62
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This pr contains a notebook with code to 1. Generate a schema for a training set 2. Generate dummy data against that schema 3. Inject null rate spike and data drift anomalies 4. implement cardinality estimation 5. implement summarization logic based on high vs. low cardinalities 6. compute drift on summaries This PR also contains some book keeping changes 1. change discord to slack as the primary communication channel for the OSS chronon channel 2. Edit build scripts to remove non 2.12 builds - 2.12 is the only version that is supported across modern spark and flink. 3. pull in unit test fixes - [x] Added Unit Tests - [ ] Covered by existing CI - [ ] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> - **New Features** - Added functionality for enhanced data drift analysis and statistical distance calculations. - Implemented methods for generating and handling partition ranges and summarization in Spark. - Expanded API with new enumerations and structures for drift metrics. - Introduced new methods for generating synthetic data for fraud detection models. - Enhanced capabilities for user-defined aggregate functions (UDAFs) in Spark. - Introduced a new utility class for managing and computing missing partitions in data workflows. - **Bug Fixes** - Corrected dependency versions and improved error handling in various components. - **Documentation** - Expanded documentation on monitoring systems for ML and AI data, detailing drift computation. - **Tests** - Added comprehensive test suites for UDAFs, data summarization, and drift analysis functionalities. - Enhanced test coverage for various scenarios, including error handling and logging improvements. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Piyush Narang <[email protected]> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: ken-zlai <[email protected]> Co-authored-by: Chewy Shaw <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: chewy-zlai <[email protected]> Co-authored-by: ezvz <[email protected]>
This was referenced Nov 26, 2024
This was referenced Dec 6, 2024
This was referenced Dec 17, 2024
Merged
This was referenced Jan 2, 2025
This was referenced Jan 16, 2025
4 tasks
kumar-zlai
pushed a commit
that referenced
this pull request
Apr 25, 2025
## Summary This completes the path of generating data, summarizing it, and uploading it to DynamoDB. ## Checklist - [ ] Added Unit Tests - [ ] Covered by existing CI - [x] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Summary by CodeRabbit - **New Features** - Enhanced data generation with updated schema for fraud data, including a new date string field. - Added functionality to load summary data into DynamoDB with improved error handling. - Introduced summarization and upload capabilities for data processing. - Updated Docker configuration to include new environment variables for Spark and AWS integration. - **Bug Fixes** - Improved error handling during summary data uploads to ensure better reliability. - **Refactor** - Updated variable handling in scripts for better clarity and maintainability. - Refactored test cases to utilize a mock API for uploading summary statistics. - **Documentation** - Comments and documentation have been updated to reflect changes in data structures and functionalities. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: nikhil-zlai <[email protected]> Co-authored-by: Chewy Shaw <[email protected]> Co-authored-by: Piyush Narang <[email protected]> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: ken-zlai <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: ezvz <[email protected]>
kumar-zlai
pushed a commit
that referenced
this pull request
Apr 29, 2025
## Summary This completes the path of generating data, summarizing it, and uploading it to DynamoDB. ## Checklist - [ ] Added Unit Tests - [ ] Covered by existing CI - [x] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Summary by CodeRabbit - **New Features** - Enhanced data generation with updated schema for fraud data, including a new date string field. - Added functionality to load summary data into DynamoDB with improved error handling. - Introduced summarization and upload capabilities for data processing. - Updated Docker configuration to include new environment variables for Spark and AWS integration. - **Bug Fixes** - Improved error handling during summary data uploads to ensure better reliability. - **Refactor** - Updated variable handling in scripts for better clarity and maintainability. - Refactored test cases to utilize a mock API for uploading summary statistics. - **Documentation** - Comments and documentation have been updated to reflect changes in data structures and functionalities. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: nikhil-zlai <[email protected]> Co-authored-by: Chewy Shaw <[email protected]> Co-authored-by: Piyush Narang <[email protected]> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: ken-zlai <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: ezvz <[email protected]>
chewy-zlai
added a commit
that referenced
this pull request
May 16, 2025
## Summary This completes the path of generating data, summarizing it, and uploading it to DynamoDB. ## Cheour clientslist - [ ] Added Unit Tests - [ ] Covered by existing CI - [x] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Summary by CodeRabbit - **New Features** - Enhanced data generation with updated schema for fraud data, including a new date string field. - Added functionality to load summary data into DynamoDB with improved error handling. - Introduced summarization and upload capabilities for data processing. - Updated Doour clientser configuration to include new environment variables for Spark and AWS integration. - **Bug Fixes** - Improved error handling during summary data uploads to ensure better reliability. - **Refactor** - Updated variable handling in scripts for better clarity and maintainability. - Refactored test cases to utilize a moour clients API for uploading summary statistics. - **Documentation** - Comments and documentation have been updated to reflect changes in data structures and functionalities. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: nikhil-zlai <[email protected]> Co-authored-by: Chewy Shaw <[email protected]> Co-authored-by: Piyush Narang <[email protected]> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: ken-zlai <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: ezvz <[email protected]>
This was referenced Jul 21, 2025
This was referenced Jul 30, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This completes the path of generating data, summarizing it, and uploading it to DynamoDB.
Checklist
Summary by CodeRabbit
Summary by CodeRabbit
New Features
Bug Fixes
Refactor
Documentation