feat: integrate iceberg initial load support via jvm grpc #1792
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- add quarkus with grpc for jvm - expose required routes for appends, changes and setup tables
iamKunalGupta
force-pushed
the
feat/iceberg-support
branch
from
e1195eb
to
27f7fb5
Compare
- Introduce significant enhancements to Iceberg support, including the handling of Iceberg tables and improved JVM compatibility. - Refactor codebase to improve readability and maintainability, including the extraction of common functionality into separate methods and the removal of redundant code. - Implement new methods for handling Iceberg tables, including table creation, record appending, and table dropping. - Introduce a new `RecordWriterFactory` to manage record writing tasks. - Update `build.gradle` and `application.yaml` files to reflect changes in the project setup. - Remove unused test files and update the logging system for better debugging. - Update Go files in the `flow/connectors/iceberg` directory to improve Iceberg handling. - Add new protobuf messages in `protos/peers.proto` for Google Cloud Storage (GCS) configuration. - Update `protos/flow-jvm.proto` to include an idempotency key in the `AppendRecordsRequest` message.
iamKunalGupta
force-pushed
the
feat/iceberg-support
branch
2 times, most recently
from
079d627
to
e008421
Compare
iamKunalGupta
force-pushed
the
feat/iceberg-support
branch
from
e008421
to
4c88206
Compare
iamKunalGupta
requested review from
serprex,
iskakaushik and
heavycrystal
and removed request for
serprex
serprex
reviewed
Jun 11, 2024
serprex
reviewed
Jun 11, 2024
|
PR description should outline why gradle files need to be brought in |
serprex
reviewed
Jun 11, 2024
protos/peers.proto
Outdated
| optional int32 client_pool_size = 4; | ||
| optional bool cache_enabled = 5; | ||
| // This helps in testing, where we can pass additional properties to the catalog | ||
| // map<string, string> additional_properties = 6; |
There was a problem hiding this comment.
should this be kept in final code?
serprex
reviewed
Jun 11, 2024
serprex
reviewed
Jun 11, 2024
- additionally add ci build actions
iamKunalGupta
force-pushed
the
feat/iceberg-support
branch
2 times, most recently
from
10f7d7f
to
9b93886
Compare
iamKunalGupta
force-pushed
the
feat/iceberg-support
branch
from
9b93886
to
f72adde
Compare
- Modify the condition in IcebergService to correctly check if a namespace exists before creating it.
…y footprint (#1835) This uses streams instead of sending all the records in 1 go.
iamKunalGupta
changed the title
iamKunalGupta
changed the title
| @@ -0,0 +1,97 @@ | |||
| #### | |||
There was a problem hiding this comment.
what's the point of the Dockerfiles in this directory?
There was a problem hiding this comment.
These are generated by quarkus, have used them further for reference to create our own under stacks.
flow-jvm/src/main/java/io/peerdb/flow/jvm/RequestLoggingInterceptor.java
Outdated
Show resolved
Hide resolved
flow-jvm/src/main/java/io/peerdb/flow/jvm/iceberg/lock/LockManager.java
Outdated
Show resolved
Hide resolved
flow-jvm/src/main/java/io/peerdb/flow/jvm/iceberg/resource/IcebergResource.java
Show resolved
Hide resolved
 TODO: - [x] Add required asterisk - [ ] Make validate and create work - [x] Test and fix the wiring of all fields to Flow API - [x] Add form validation and error toast - [x] Add tips and helpful links --------- Co-authored-by: Kunal Gupta <[email protected]>
iamKunalGupta
changed the title
- Add `AppendAlreadyDoneException` to handle specific scenario when append operation is already done - Enhance record appending process in `IcebergService` by introducing better error handling and more precise control flow
iamKunalGupta
force-pushed
the
feat/iceberg-support
branch
from
145ec44
to
237ac32
Compare
This was
unlinked from
issues
Jun 24, 2024
7 tasks
serprex
reviewed
Jun 25, 2024
| req *protos.GetTableSchemaBatchInput, | ||
| ) (*protos.GetTableSchemaBatchOutput, error) { | ||
| // TODO implement me | ||
| panic("implement me") |
There was a problem hiding this comment.
TODO, or remove implementing this interface
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently Gradle is used for build and dev and additionally does the following (via Quarkus Tasks):
TODOS:
Table namespace supportplanned for later