PR - Add Delta Lake Backend for Cryptofeed #1054
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add DeltaLakeCallback class with support for various data types - Implement partitioning, Z-ordering, and time travel features - Add schema documentation for each data type - Include Delta Lake dependencies in setup.py - Create demo file for Delta Lake usage with S3 configuration - Update extras_require in setup.py to include deltalake option
… during Delta Lake write
…dle null values correctly
…y and maintainability
…type and batch_size in DeltaLakeCallback
…timezone info after conversion
… timestamp columns to UTC datetime
|
Hi @bmoscon, Thank you very much for the project. I built my backend for delta lake and I would like to ask for a review for PR. The code is following postgres and kafka backends with support for queued updates. I have tested with cryptostore project with this backend in test env. Thank you. Br, |
|
@tommy-ca thanks for the PR, let me review and give it a try and I'll merge it |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add Delta Lake Backend for Cryptofeed
Description
This PR introduces a comprehensive Delta Lake backend for Cryptofeed. It adds the
DeltaLakeCallbackclass and several data-specific subclasses to handle various types of cryptocurrency data. The new backend allows for efficient storage and retrieval of high-frequency trading data using Delta Lake technology.Key Features and Improvements
DeltaLakeCallbackbase class with configurable options for Delta Lake integration.Checklist
Additional Notes
This PR significantly enhances Cryptofeed's capabilities by adding a robust Delta Lake backend. It allows users to store and analyze cryptocurrency market data using Delta Lake's features such as ACID transactions, schema evolution, and time travel.
The implementation includes proper error handling, logging, and configuration options to make it adaptable to various use cases. Each data type (trades, funding, ticker, etc.) has its own specialized class to handle specific schema requirements.
Before merging, we should ensure comprehensive testing of all Delta Lake operations, update the changelog to reflect this major feature addition, and run Flake8 to catch any remaining style issues.
Affected Components
cryptofeed/backends/deltalake.py