ADLS Gen2 API Implementation #8280
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| The MIT License (MIT) | ||
|
|
||
| Copyright (c) 2019 Microsoft | ||
|
|
||
| Permission is hereby granted, free of charge, to any person obtaining a copy | ||
| of this software and associated documentation files (the "Software"), to deal | ||
| in the Software without restriction, including without limitation the rights | ||
| to use, copy, modify, merge, publish, distribute, sublicense, and/or sell | ||
| copies of the Software, and to permit persons to whom the Software is | ||
| furnished to do so, subject to the following conditions: | ||
|
|
||
| The above copyright notice and this permission notice shall be included in all | ||
| copies or substantial portions of the Software. | ||
|
|
||
| THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
| IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
| FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE | ||
| AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER | ||
| LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | ||
| OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE | ||
| SOFTWARE. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| include *.md | ||
| include azure/__init__.py | ||
| include azure/storage/__init__.py | ||
| include azure/storage/file/__init__.py | ||
| include LICENSE.txt | ||
| recursive-include tests *.py *.yaml | ||
|
xiafu-msft marked this conversation as resolved.
Outdated
|
||
| recursive-include samples *.py *.md | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,199 @@ | ||
| # Azure DataLake service client library for Python | ||
| Overview | ||
|
|
||
| This preview package for Python includes ADLS Gen2 specific API support made available in Storage SDK. This includes: | ||
| 1. New directory level operations (Create, Rename/Move, Delete) for both hierarchical namespace enabled (HNS) storage accounts and HNS disabled storage accounts. For HNS enabled accounts, the rename/move operations are atomic. | ||
| (the current SDK only supports HNS operations) | ||
| 2. Permission related operations (Get/Set ACLs) for hierarchical namespace enabled (HNS) accounts. | ||
|
|
||
|
|
||
| [Source code](https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/storage/azure-storage-file-datalake/azure/storage/file/datalake) | [Package (PyPi)](https://pypi.org/project/azure-storage-file-datalake/) | [API reference documentation](https://aka.ms/azsdk-python-storage-file-datalake-ref) | [Product documentation](https://docs.microsoft.com/azure/storage/) | [Samples](https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/storage/azure-storage-file-datalake/samples) | ||
|
|
||
|
|
||
| ## Getting started | ||
|
|
||
| ### Prerequisites | ||
| * Python 2.7, or 3.5 or later is required to use this package. | ||
| * You must have an [Azure subscription](https://azure.microsoft.com/free/) and an | ||
| [Azure storage account](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-quickstart-create-account) to use this package. | ||
|
|
||
| ### Install the package | ||
| Install the Azure DataLake Storage client library for Python with [pip](https://pypi.org/project/pip/): | ||
|
|
||
| ```bash | ||
| pip install azure-storage-file-datalake --pre | ||
| ``` | ||
|
|
||
| ### Create a storage account | ||
| If you wish to create a new storage account, you can use the | ||
| [Azure Portal](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-quickstart-create-account#create-an-account-using-the-azure-portal), | ||
| [Azure PowerShell](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-quickstart-create-account#create-an-account-using-powershell), | ||
| or [Azure CLI](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-quickstart-create-account#create-an-account-using-azure-cli): | ||
|
|
||
| ```bash | ||
| # Create a new resource group to hold the storage account - | ||
| # if using an existing resource group, skip this step | ||
| az group create --name my-resource-group --location westus2 | ||
|
|
||
| # Create the storage account | ||
| az storage account create -n my-storage-account-name -g my-resource-group --hierarchical-namespace true | ||
| ``` | ||
|
|
||
| ### Authenticate the client | ||
|
|
||
| Interaction with DataLake Storage starts with an instance of the DataLakeServiceClient class. You need an existing storage account, its URL, and a credential to instantiate the client object. | ||
|
|
||
| #### Get credentials | ||
|
|
||
| To authenticate the client you have a few options: | ||
| 1. Use a SAS token string | ||
| 2. Use an account shared access key | ||
| 3. Use a token credential from [azure.identity](https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/identity/azure-identity) | ||
|
|
||
| Alternatively, you can authenticate with a storage connection string using the `from_connection_string` method. See example: [Client creation with a connection string](#client-creation-with-a-connection-string). | ||
|
|
||
| You can omit the credential if your account URL already has a SAS token. | ||
|
|
||
| #### Create client | ||
|
|
||
| Once you have your account URL and credentials ready, you can create the DataLakeServiceClient: | ||
|
|
||
| ```python | ||
| from azure.storage.file.datalake import DataLakeServiceClient | ||
|
|
||
| service = DataLakeServiceClient(account_url="https://<my-storage-account-name>.dfs.core.windows.net/", credential=credential) | ||
| ``` | ||
|
|
||
| ## Key concepts | ||
|
|
||
| DataLake storage offers four types of resources: | ||
| * The storage account | ||
| * A file system in the storage account | ||
| * A directory under the file system | ||
| * A file in a the file system or under directory | ||
|
|
||
| #### Clients | ||
|
|
||
| The DataLake Storage SDK provides four different clients to interact with the DataLake Service: | ||
| 1. **DataLakeServiceClient** - this client interacts with the DataLake Service at the account level. | ||
| It provides operations to retrieve and configure the account properties | ||
| as well as list, create, and delete file systems within the account. | ||
| For operations relating to a specific file system, directory or file, clients for those entities | ||
| can also be retrieved using the `get_file_client`, `get_directory_client` or `get_file_system_client` functions. | ||
| 2. **FileSystemClient** - this client represents interaction with a specific | ||
| file system, even if that file system need not exist yet. It provides operations to create, delete, or | ||
| configure file systems and includes operations to list paths under file system, upload, and delete file or | ||
| directory in the file system. | ||
| For operations relating to a specific file, the client can also be retrieved using | ||
| the `get_file_client` function. | ||
| For operations relating to a specific directory, the client can be retrieved using | ||
| the `get_directory_client` function. | ||
| 3. **DataLakeDirectoryClient** - this client represents interaction with a specific | ||
| directory, even if that directory need not exist yet. It provides directory operations create, delete, rename, | ||
| get properties and set properties operations. | ||
| 3. **DataLakeFileClient** - this client represents interaction with a specific | ||
| file, even if that file need not exist yet. It provides file operations to append data, flush data, delete, | ||
| create, and read file. | ||
| 4. **DataLakeLeaseClient** - this client represents lease interactions with a FileSystemClient, DataLakeDirectoryClient | ||
| or DataLakeFileClient. It provides operations to acquire, renew, release, change, and break leases on the resources. | ||
|
|
||
| ## Examples | ||
|
|
||
| The following sections provide several code snippets covering some of the most common Storage DataLake tasks, including: | ||
|
|
||
| * [Client creation with a connection string](#client-creation-with-a-connection-string) | ||
| * [Uploading a file](#uploading-a-file) | ||
| * [Downloading a file](#downloading-a-file) | ||
| * [Enumerating paths](#enumerating-paths) | ||
|
|
||
|
|
||
| ### Client creation with a connection string | ||
| Create the DataLakeServiceClient using the connection string to your Azure Storage account. | ||
|
|
||
| ```python | ||
| from azure.storage.file.datalake import DataLakeServiceClient | ||
|
|
||
| service = DataLakeServiceClient.from_connection_string(conn_str="my_connection_string") | ||
| ``` | ||
|
|
||
| ### Uploading a file | ||
| Upload a file to your file system. | ||
|
|
||
| ```python | ||
| from azure.storage.file.datalake import DataLakeFileClient | ||
|
|
||
| data = b"abc" | ||
| file = DataLakeFileClient.from_connection_string("my_connection_string", | ||
| file_system_name="myfilesystem", file_path="myfile") | ||
|
|
||
| file.append_data(data, offset=0, length=len(data)) | ||
| file.flush_data(len(data)) | ||
| ``` | ||
|
|
||
| ### Downloading a file | ||
| Download a file from your file system. | ||
|
|
||
| ```python | ||
| from azure.storage.file.datalake import DataLakeFileClient | ||
|
Contributor
There was a problem hiding this comment. Did we decide to go with file.datalake or filedatalake? |
||
|
|
||
| file = DataLakeFileClient.from_connection_string("my_connection_string", | ||
| file_system_name="myfilesystem", file_path="myfile") | ||
|
|
||
| with open("./BlockDestination.txt", "wb") as my_file: | ||
| file_data = file.read_file(stream=my_file) | ||
| ``` | ||
|
|
||
| ### Enumerating paths | ||
| List the paths in your file system. | ||
|
|
||
| ```python | ||
| from azure.storage.file.datalake import FileSystemClient | ||
|
|
||
| file_system = FileSystemClient.from_connection_string("my_connection_string", file_system_name="myfilesystem") | ||
|
|
||
| paths = file_system.get_paths() | ||
| for path in paths: | ||
| print(path.name + '\n') | ||
| ``` | ||
|
|
||
| ## Troubleshooting | ||
| DataLake Storage clients raise exceptions defined in [Azure Core](https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/core/azure-core/README.md). | ||
|
|
||
| All DataLake service operations will throw a StorageErrorException on failure with helpful [error codes](https://docs.microsoft.com/rest/api/storageservices/blob-service-error-codes). | ||
|
|
||
| ## Next steps | ||
|
|
||
| ### More sample code | ||
|
|
||
| Get started with our [Azure DataLake samples](https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/storage/azure-storage-file-datalake/samples). | ||
|
|
||
| Several DataLake Storage Python SDK samples are available to you in the SDK's GitHub repository. These samples provide example code for additional scenarios commonly encountered while working with DataLake Storage: | ||
|
|
||
| * [`datalake_samples_access_control.py`](https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/storage/azure-storage-file-datalake/samples/datalake_samples_access_control.py) - Examples for common DataLake Storage tasks: | ||
| * Set up a file system | ||
| * Create a directory | ||
| * Set/Get access control for the directory | ||
| * Create files under the directory | ||
| * Set/Get access control for each file | ||
| * Delete file system | ||
|
|
||
| * [`datalake_samples_upload_download.py`](https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/storage/azure-storage-file-datalake/samples/datalake_samples_upload_download.py) - Examples for authenticating and creating the client: | ||
| * Set up a file system | ||
| * Create file | ||
| * Append data to the file | ||
| * Flush data to the file | ||
| * Download the uploaded data | ||
| * Delete file system | ||
|
|
||
|
|
||
| ### Additional documentation | ||
|
|
||
| For more extensive REST documentation on Data Lake Storage Gen2, see the [Data Lake Storage Gen2 documentation](https://docs.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/filesystem) on docs.microsoft.com. | ||
|
|
||
|
|
||
| ## Contributing | ||
| This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com. | ||
|
|
||
| When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA. | ||
|
|
||
| This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| __path__ = __import__('pkgutil').extend_path(__path__, __name__) # type: str |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| __path__ = __import__('pkgutil').extend_path(__path__, __name__) # type: str |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| __path__ = __import__('pkgutil').extend_path(__path__, __name__) # type: str |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| # ------------------------------------------------------------------------- | ||
| # Copyright (c) Microsoft Corporation. All rights reserved. | ||
| # Licensed under the MIT License. See License.txt in the project root for | ||
| # license information. | ||
| # -------------------------------------------------------------------------- | ||
|
|
||
| from ._data_lake_file_client import DataLakeFileClient | ||
| from ._data_lake_directory_client import DataLakeDirectoryClient | ||
| from ._file_system_client import FileSystemClient | ||
| from ._data_lake_service_client import DataLakeServiceClient | ||
| from ._data_lake_lease import DataLakeLeaseClient | ||
| from ._models import ( | ||
| LocationMode, | ||
| ResourceTypes, | ||
| FileSystemProperties, | ||
| FileSystemPropertiesPaged, | ||
| DirectoryProperties, | ||
| PathProperties, | ||
| PathPropertiesPaged, | ||
| LeaseProperties, | ||
| ContentSettings, | ||
| AccountSasPermissions, | ||
| FileSystemSasPermissions, | ||
| DirectorySasPermissions, | ||
| FileSasPermissions, | ||
| UserDelegationKey, | ||
| PublicAccess | ||
| ) | ||
| from ._shared_access_signature import generate_account_sas, generate_file_system_sas, generate_directory_sas, \ | ||
| generate_file_sas | ||
|
|
||
| from ._shared.policies import ExponentialRetry, LinearRetry | ||
| from ._shared.models import StorageErrorCode | ||
| from ._version import VERSION | ||
|
|
||
|
|
||
| __all__ = [ | ||
| 'DataLakeServiceClient', | ||
| 'FileSystemClient', | ||
| 'DataLakeFileClient', | ||
| 'DataLakeDirectoryClient', | ||
| 'DataLakeLeaseClient', | ||
| 'ExponentialRetry', | ||
| 'LinearRetry', | ||
| 'LocationMode', | ||
| 'PublicAccess', | ||
| 'ResourceTypes', | ||
| 'StorageErrorCode', | ||
| 'UserDelegationKey', | ||
| 'FileSystemProperties', | ||
| 'FileSystemPropertiesPaged', | ||
| 'DirectoryProperties', | ||
| 'PathProperties', | ||
| 'PathPropertiesPaged', | ||
| 'LeaseProperties', | ||
| 'ContentSettings', | ||
| 'AccountSasPermissions', | ||
| 'FileSystemSasPermissions', | ||
| 'DirectorySasPermissions', | ||
| 'FileSasPermissions', | ||
| 'generate_account_sas', | ||
| 'generate_file_system_sas', | ||
| 'generate_directory_sas', | ||
| 'generate_file_sas', | ||
| 'VERSION', | ||
| ] |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove this extra minus sign and Doc Warden should be happier. @scbedd (saw an error here: https://dev.azure.com/azure-sdk/public/_build/results?buildId=181603&view=logs&j=b70e5e73-bbb6-5567-0939-8415943fadb9&t=9069f0a2-bd31-51b1-8b5c-08355cf80229&l=84)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just previously there was also a minus sign, it didn't fail the readme test, weird 😄