Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ADLS integration #120
ADLS integration #120
Changes from all commits
2d59981
61a679b
d895da8
16f99ce
cf80ec1
bb71ee3
0c27c50
bd9d60e
ae2092b
a04e81f
399aaef
a900689
240102b
692149c
c9ba0cc
b7e62b1
ed3fbed
347384b
913a37c
3027a1f
321c335
169135a
d42ce94
5861448
8a521f4
File filter
Filter by extension
Conversations
Jump to
There are no files selected for viewing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is needed only for the ADLS storage.
My recommendation will be to provide different spark-shell commands here for different recipes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 to the above recommendation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good, fixed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am actually not sure why
io-impl
is needed to be passed as part of spark conf. An instance of this class should be returned by OpenHouseTableOperations. CC: @HotSushiThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not really sure why, but the spark cluster only successfully completes the requests when it is initiated with it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Currently the code to map storageType -> client FileIO on tableOperations does not exist, but will be added later when supporting multi-storage per cluster. Till then, this is a cleaner way of supporting a fileio per cluster
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Again is this something we need user to pass or something we were letting the catalog retrieve. How were we thinking of this for s3/minio? CC: @HotSushi
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seemed like in the s3 PR variables were passed in a similar fashion.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
configuring a fileIO on client side needs such details to be provided at the client side. Its similar to hdfs, OH server doesn't communicate all confs necessary to instantiate hdfs client completely. Its assumed that client has the right hdfs communication already established.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we do this, then do we need to copy code? It might be ok to pull in library for iceberg-azure at version 1.5.2 for use in openhouse repo. Thoughts @HotSushi ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried this - it gives more dependency issues.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, we gave this a shot. Compatibility mismatch because FileIO interface has evolved from 1.2 -> 1.4
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So iceberg common is needed as
implementation
with ADLS integration?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As of right now, yes. If I do the submodules later I may reevaluate