-
Notifications
You must be signed in to change notification settings - Fork 8.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HADOOP-18516. ABFS: Support fixed SAS token config in addition to SASTokenProvider class #5148
base: trunk
Are you sure you want to change the base?
Conversation
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
:::: AGGREGATED TEST RESULT :::: HNS-OAuth[INFO] Results: HNS-SharedKey[INFO] Results: NonHNS-SharedKey[INFO] Results: AppendBlob-HNS-OAuth[INFO] Results: Time taken: 36 mins 21 secs. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
Outdated
Show resolved
Hide resolved
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
Outdated
Show resolved
Hide resolved
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/utils/SASGenerator.java
Outdated
Show resolved
Hide resolved
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
Outdated
Show resolved
Hide resolved
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
Outdated
Show resolved
Hide resolved
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
Outdated
Show resolved
Hide resolved
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
Outdated
Show resolved
Hide resolved
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
Outdated
Show resolved
Hide resolved
...oop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemChooseSAS.java
Show resolved
Hide resolved
// clearing any previously configured SAS Token Provider class | ||
testAbfsConfig.unset(FS_AZURE_SAS_TOKEN_PROVIDER_TYPE); | ||
|
||
// setting an account SAS token in the fixed token field | ||
testAbfsConfig.set(FS_AZURE_SAS_FIXED_TOKEN, accountSAS); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As per https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/pom.xml#L566-L613, this test would be run in parallel in same JVM process of other test, unset and set config can have affect on other tests. Kindly check if synchronization can be done. If not, we can add this class in https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/pom.xml#L643-L652
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/utils/SASGenerator.java
Show resolved
Hide resolved
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
This PR introduces a new configuration for Fixed SAS Tokens.
Using this new configuration
fs.azure.sas.fixed.token
, users can configure a fixed SAS Token in the account settings files itself. Ideally, this should be used with SAS Tokens that are scoped at a container or account level (Service or Account SAS), which can be considered to be a constant for one account or container, over multiple operations.The other method of using a SAS Token remains valid as well, where a user provides a custom implementation of the SASTokenProvider interface, using which a SAS Token are obtained.
A change for appending SAS Tokens to the queries of 4 filesystem level APIs is also introduced, helped by the introduction of the fixed token. When an Account SAS Token is configured as the fixed SAS Token, and it is used, it is ensured that these operations are within the scope of the SAS Token. The configuration introduced also bypasses the need for having an implementation of the SASTokenProvider interface in cases where a token simply be directly set.
The code checks for whether the fixed token and the token provider class implementation are configured. In the case of both being set, preference is given to the SASTokenProvider implementation. It must be noted that if such an implementation provides a SAS Token which has a lower scope than Account SAS, some filesystem and service level operations might be out of scope and may not succeed.