-
Notifications
You must be signed in to change notification settings - Fork 2.9k
Aliyun: Add OSSFileIO #3553
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Aliyun: Add OSSFileIO #3553
Conversation
aliyun/src/main/java/org/apache/iceberg/aliyun/oss/OSSFileIO.java
Outdated
Show resolved
Hide resolved
aliyun/src/main/java/org/apache/iceberg/aliyun/DefaultAliyunClientFactory.java
Outdated
Show resolved
Hide resolved
aliyun/src/main/java/org/apache/iceberg/aliyun/AliyunClientFactory.java
Outdated
Show resolved
Hide resolved
aliyun/src/test/java/org/apache/iceberg/aliyun/oss/TestOSSFileIO.java
Outdated
Show resolved
Hide resolved
aliyun/src/test/java/org/apache/iceberg/aliyun/oss/AliyunOSSTestBase.java
Outdated
Show resolved
Hide resolved
aliyun/src/main/java/org/apache/iceberg/aliyun/oss/OSSFileIO.java
Outdated
Show resolved
Hide resolved
aliyun/src/main/java/org/apache/iceberg/aliyun/AliyunClientFactory.java
Outdated
Show resolved
Hide resolved
| * | ||
| * @return oss client. | ||
| */ | ||
| OSS oss(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think this method name is clear. How about newClient?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sounds good. I will make it.
| * different endpoints. For more information, see: | ||
| * https://www.alibabacloud.com/help/doc-detail/31837.htm | ||
| */ | ||
| public static final String OSS_ENDPOINT = "oss.endpoint"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We generally try to avoid nesting in places that have a small set of specific properties. Is it possible to make these shorter and more standard?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As we will introduce the aliyun catalog services named DLF in future, the DLF services also has its endpoint. we need to prefix oss. and dlf. to distingush it's an oss endpoint or dlf endpoint. That's why we use oss.endpoint here.
aliyun/src/main/java/org/apache/iceberg/aliyun/AliyunProperties.java
Outdated
Show resolved
Hide resolved
aliyun/src/main/java/org/apache/iceberg/aliyun/oss/OSSInputFile.java
Outdated
Show resolved
Hide resolved
|
Looks good to me. Since I don't see an approval from @openinx, I'll wait and let him merge this when he is ready. |
| * If set, all Aliyun clients will be initialized by the specified factory. | ||
| * If not set, {@link DefaultAliyunClientFactory} is used as default factory. | ||
| */ | ||
| public static final String CLIENT_FACTORY = "client.factory-impl"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
would it be better to make this the same as https://github.com/apache/iceberg/blob/master/aws/src/main/java/org/apache/iceberg/aws/AwsProperties.java#L145?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be nice if they were the same, but I like that this conforms to the -impl convention that we use for dynamically loaded classes elsewhere.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can make another PR to line up the behavior of aws part as "client.factory-impl"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool, that works for me as well
| import java.util.Map; | ||
| import org.apache.iceberg.relocated.com.google.common.base.Preconditions; | ||
|
|
||
| public class DefaultAliyunClientFactory implements AliyunClientFactory { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When we were writing AwsClientFactory we had AwsClientFactories to vend a default client factory implementation, and that was because it seems like Iceberg does not expose the default implementation as a public class for other classes like LocationProvider, etc. I wonder if we should follow the same pattern here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I agree. As we will also introduce many different factories for aliyun clients, it's good to have a proxy to delegate to create the factories.
| * | ||
| * @return oss client. | ||
| */ | ||
| OSS newClient(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's better to name it as ossClient because I think we will introduce another aliyun client, such as dlf client etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How aboutnewOSSClient then?
@xingbowu , Could you help to fix the checkstyles ? |
jackye1995
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me, thanks for addressing the comments!
| * | ||
| * @return oss client. | ||
| */ | ||
| OSS ossClient(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you make this newOSSClient? We typically prefix factory method names with new.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rework and follow up this guide
openinx
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM now ! Let's get this merged, Thanks @xingbowu for contribution and thanks @jackye1995 and @rdblue for the careful reviewing !
No description provided.