diff --git a/eng/code-quality-reports/src/main/resources/checkstyle/checkstyle-suppressions.xml b/eng/code-quality-reports/src/main/resources/checkstyle/checkstyle-suppressions.xml index e7baf627fd47..0e73a165b4af 100755 --- a/eng/code-quality-reports/src/main/resources/checkstyle/checkstyle-suppressions.xml +++ b/eng/code-quality-reports/src/main/resources/checkstyle/checkstyle-suppressions.xml @@ -126,7 +126,7 @@ diff --git a/sdk/storage/azure-storage-blob-nio/README.md b/sdk/storage/azure-storage-blob-nio/README.md index 2396ef3a19f5..bf9c0f7fb96e 100644 --- a/sdk/storage/azure-storage-blob-nio/README.md +++ b/sdk/storage/azure-storage-blob-nio/README.md @@ -1,5 +1,6 @@ -# Azure Storage Java NIO Blob plugin library for Java -## This README is not yet updated for this project and is a skeleton copied from blobs +# Azure Storage Blob NIO FileSystemProvider + +This package allows you to interact with Azure Blob Storage through the standard Java NIO Filesystem APIs. [Source code][source] | [API reference documentation][docs] | [REST API documentation][rest_docs] | [Product documentation][product_docs] | [Samples][samples] @@ -35,8 +36,8 @@ az storage account create \ ### Authenticate the client -In order to interact with the Storage Service (Blob, Queue, Message, MessageId, File) you'll need to create an instance of the Service Client class. -To make this possible you'll need the Account SAS (shared access signature) string of the Storage Account. Learn more at [SAS Token][sas_token] +The simplest way to interact with the Storage Service is to create an instance of the [FileSystem][file_system] class using the [FileSystems][file_systems] API. +To make this possible you'll need the Account SAS (shared access signature) string of the Storage Account or a Shared Key. Learn more at [SAS Token][sas_token] and [Shared Key][shared_key] #### Get credentials @@ -76,55 +77,185 @@ b. Alternatively, get the Account SAS Token from the Azure Portal. ##### **Shared Key Credential** -a. Use Account name and Account key. Account name is your Storage Account name. +Use Account name and Account key. Account name is your Storage Account name. 1. Go to your Storage Account 2. Select `Access keys` from the menu on the left 3. Under `key1`/`key2` copy the contents of the `Key` field -or +## Key concepts -b. Use the connection string. +NIO on top of Blob Storage is designed for: + +- Working with Blob Storage as though it were a local file system +- Random access reads on large blobs without downloading the entire blob +- Uploading full files as blobs +- Creating and navigating a directory structure within an account +- Reading and setting attributes on blobs + +## Design Notes +It is important to recognize that Azure Blob Storage is not a true FileSystem, nor is it the goal of this project to +force Azure Blob Storage to act like a full-fledged FileSystem. While providing FileSystem APIs on top of Azure Blob +Storage can offer convenience and ease of access in certain cases, trying to force the Storage service to work in +scenarios it is not designed for is bound to introduce performance and stability problems. + +To that end, this project will only offer APIs that can be sensibly and cleanly built on top of Azure Blob Storage APIs. +We recognize that this will leave some scenarios unsupported indefinitely, but we would rather offer a product that +works predictably and reliably in its well defined scenarios than eagerly support all possible scenarios at the expense +of quality. Even still, supporting some fundamentally required use cases, such as directories, can result in unexpected +behavior due to the difference between blob storage and a file system. The javadocs on each type and method should +therefore be read and understood for ways in which they may diverge from the standard specified by the JDK. + +Moreover, even from within a given application, it should be remembered that using a remote FileSystem introduces higher +latency. Because of this, particular care must be taken when managing concurrency. Race conditions are more likely to +manifest, network failures occur more frequently than disk failures, and other such distributed application scenarios +must be considered when working with this FileSystem. While the AzureFileSystem will ensure it takes appropriate steps +towards robustness and reliability, the application developer must also design around these failure scenarios and have +fallback and retry options available. + +The view of the FileSystem from within an instance of the JVM will be consistent, but the AzureFileSystem makes no +guarantees on behavior or state should other processes operate on the same data. The AzureFileSystem will assume that it +has exclusive access to the resources stored in Azure Blob Storage and will behave without regard for potential +interfering applications + +Finally, this implementation has currently chosen to always read/write directly to/from Azure Storage without a local +cache. Our team has determined that with the tradeoffs of complexity, correctness, safety, performance, debuggability, +etc. one option is not inherently better than the other and that this choice most directly addresses the current known +use cases for this project. While this has consequences for every API, of particular note is the limitations on writing +data. Data may only be written as an entire file (i.e. random IO or appends are not supported), and data is not +committed or available to be read until the write stream is closed. -1. Go to your Storage Account -2. Select `Access keys` from the menu on the left -3. Under `key1`/`key2` copy the contents of the `Connection string` field +## Examples -## Key concepts +The following sections provide several code snippets covering some of the most common Azure Storage Blob NIO tasks, including: -Blob Storage is designed for: +- [Create a `FileSystem`](#create-a-filesystem) +- [Create a directory](#create-a-directory) +- [Iterate over directory contents](#iterate-over-directory-contents) +- [Read a file](#read-a-file) +- [Write to a file](#write-to-a-file) +- [Copy a file](#copy-a-file) +- [Delete a file](#delete-a-file) +- [Read attributes on a file](#read-attributes-on-a-file) +- [Write attributes to a file](#write-attributes-to-a-file) -- Serving images or documents directly to a browser -- Storing files for distributed access -- Streaming video and audio -- Writing to log files -- Storing data for backup and restore, disaster recovery, and archiving -- Storing data for analysis by an on-premises or Azure-hosted service +### Create a `FileSystem` -## Examples +Create a `FileSystem` using the [`shared key`](#get-credentials) retrieved above. -The following sections provide several code snippets covering some of the most common Azure Storage Blob tasks, including: +Note that you can further configure the file system using constants available in `AzureFileSystem`. +Please see the docs for `AzureFileSystemProvider` for a full explanation of initializing and configuring a filesystem -- [Create a `BlobServiceClient`](#create-a-blobserviceclient) + +```java +Map config = new HashMap<>(); +config.put(AzureFileSystem.AZURE_STORAGE_ACCOUNT_KEY, ""); +config.put(AzureFileSystem.AZURE_STORAGE_FILE_STORES, ""); +FileSystem myFs = FileSystems.newFileSystem(new URI("azb://?account= +```java +Path dirPath = myFs.getPath("dir"); +Files.createDirectory(dirPath); +``` -### Create a `BlobServiceClient` +### Iterate over directory contents -Create a `BlobServiceClient` using the [`sasToken`](#get-credentials) generated above. +Iterate over a directory using a `DirectoryStream` + ```java -BlobServiceClient blobServiceClient = new BlobServiceClientBuilder() - .endpoint("") - .sasToken("") - .buildClient(); +for (Path p : Files.newDirectoryStream(dirPath)) { + System.out.println(p.toString()); +} ``` +### Read a file + +Read the contents of a file using an `InputStream`. Skipping, marking, and resetting are all supported. + + +```java +Path filePath = myFs.getPath("file"); +InputStream is = Files.newInputStream(filePath); +is.read(); +is.close(); +``` + +### Write to a file + +Write to a file. Only writing whole files is supported. Random IO is not supported. The stream must be closed in order +to guarantee that the data is available to be read. + + +```java +OutputStream os = Files.newOutputStream(filePath); +os.write(0); +os.close(); +``` + +### Copy a file + + +```java +Path destinationPath = myFs.getPath("destinationFile"); +Files.copy(filePath, destinationPath, StandardCopyOption.COPY_ATTRIBUTES); +``` + +### Delete a file + + +```java +Files.delete(filePath); +``` + +### Read attributes on a file + +Read attributes of a file through the `AzureBlobFileAttributes`. + + +```java +AzureBlobFileAttributes attr = Files.readAttributes(filePath, AzureBlobFileAttributes.class); +BlobHttpHeaders headers = attr.blobHttpHeaders(); +``` + +Or read attributes dynamically by specifying a string of desired attributes. This will not improve performance as a call +to retrieve any attribute will always retrieve all of them as an atomic bulk operation. You may specify "*" instead of a +list of specific attributes to have all attributes returned in the map. + + +```java +Map attributes = Files.readAttributes(filePath, "azureBlob:metadata,headers"); +``` + +### Write attributes to a file + +Set attributes of a file through the `AzureBlobFileAttributeView`. + + +```java +AzureBlobFileAttributeView view = Files.getFileAttributeView(filePath, AzureBlobFileAttributeView.class); +view.setMetadata(Collections.EMPTY_MAP); +``` + +Or set an attribute dynamically by specifying the attribute as a string. + + +```java +Files.setAttribute(filePath, "azureBlob:blobHttpHeaders", new BlobHttpHeaders()); +``` ## Troubleshooting -When interacting with blobs using this Java client library, errors returned by the service correspond to the same HTTP -status codes returned for [REST API][error_codes] requests. For example, if you try to retrieve a container or blob that -doesn't exist in your Storage Account, a `404` error is returned, indicating `Not Found`. +When using the NIO implementation for Azure Blob Storage, errors returned by the service are manifested as an +`IOException` which wraps a `BlobStorageException` having the same HTTP status codes returned for +[REST API][error_codes] requests. For example, if you try to read a file that doesn't exist in your Storage Account, a +`404` error is returned, indicating `Not Found`. ### Default HTTP Client All client libraries by default use the Netty HTTP client. Adding the above dependency will automatically configure @@ -137,12 +268,29 @@ operations. The Boring SSL library is an uber jar containing native libraries fo better performance compared to the default SSL implementation within the JDK. For more information, including how to reduce the dependency size, refer to the [performance tuning][performance_tuning] section of the wiki. -## Next steps - -Several Storage blob Java SDK samples are available to you in the SDK's GitHub repository. These samples provide example code for additional scenarios commonly encountered while working with Key Vault: - -## Next steps Samples -Samples are explained in detail [here][samples_readme]. +## Continued development + +This project is still actively being developed in an effort to move from preview to GA. Below is a list of features that +are not currently supported but are under consideration and may be added before GA. We welcome feedback and input on +which of these may be most useful and are open to suggestions for items not included in this list. While all of these +items are being considered, they have not been investigated and designed and therefore we cannot confirm their +feasibility within Azure Blob Storage. Therefore, it may be the case that further investigation reveals a feature may +not be possible or otherwise may conflict with established design goals and therefor will not ultimately be supported. + +- Symbolic links +- Hard links +- Hidden files +- Random writes +- File locks +- Watches on directory events +- Support for other Azure Storage services such as ADLS Gen 2 (Datalake) and Azure Files (shares) +- Token authentication +- Multi-account filesystems +- Delegating access to single files +- Normalizing directory structure of data upon loading a FileSystem +- Local caching +- Other `OpenOptions` such as append or dsync +- Flags to toggle certain behaviors such as FileStore (container) creation, etc. ## Contributing @@ -159,6 +307,7 @@ This project has adopted the [Microsoft Open Source Code of Conduct][coc]. For m [rest_docs]: https://docs.microsoft.com/rest/api/storageservices/blob-service-rest-api [product_docs]: https://docs.microsoft.com/azure/storage/blobs/storage-blobs-overview [sas_token]: https://docs.microsoft.com/azure/storage/common/storage-dotnet-shared-access-signature-part-1 +[shared_key]: https://docs.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key [jdk]: https://docs.microsoft.com/java/azure/jdk/ [azure_subscription]: https://azure.microsoft.com/free/ [storage_account]: https://docs.microsoft.com/azure/storage/common/storage-quickstart-create-account?tabs=azure-portal @@ -166,11 +315,13 @@ This project has adopted the [Microsoft Open Source Code of Conduct][coc]. For m [storage_account_create_portal]: https://docs.microsoft.com/azure/storage/common/storage-quickstart-create-account?tabs=azure-portal [identity]: https://github.com/Azure/azure-sdk-for-java/blob/master/sdk/identity/azure-identity/README.md [error_codes]: https://docs.microsoft.com/rest/api/storageservices/blob-service-error-codes -[samples]: src/samples +[samples]: https://docs.oracle.com/javase/tutorial/essential/io/fileio.html [cla]: https://cla.microsoft.com [coc]: https://opensource.microsoft.com/codeofconduct/ [coc_faq]: https://opensource.microsoft.com/codeofconduct/faq/ [coc_contact]: mailto:opencode@microsoft.com [performance_tuning]: https://github.com/Azure/azure-sdk-for-java/wiki/Performance-Tuning +[file_system]: https://docs.oracle.com/javase/7/docs/api/java/nio/file/FileSystem.html +[file_systems]: https://docs.oracle.com/javase/7/docs/api/java/nio/file/FileSystems.html ![Impressions](https://azure-sdk-impressions.azurewebsites.net/api/impressions/azure-sdk-for-java%2Fsdk%2Fstorage%2Fazure-storage-blob%2FREADME.png) diff --git a/sdk/storage/azure-storage-blob-nio/src/main/java/com/azure/storage/blob/nio/AzureBasicFileAttributeView.java b/sdk/storage/azure-storage-blob-nio/src/main/java/com/azure/storage/blob/nio/AzureBasicFileAttributeView.java index 4a0766ac4284..01ae0bd2e7a2 100644 --- a/sdk/storage/azure-storage-blob-nio/src/main/java/com/azure/storage/blob/nio/AzureBasicFileAttributeView.java +++ b/sdk/storage/azure-storage-blob-nio/src/main/java/com/azure/storage/blob/nio/AzureBasicFileAttributeView.java @@ -21,6 +21,8 @@ */ public final class AzureBasicFileAttributeView implements BasicFileAttributeView { + static final String NAME = "azureBasic"; + private final Path path; AzureBasicFileAttributeView(Path path) { @@ -33,7 +35,7 @@ public final class AzureBasicFileAttributeView implements BasicFileAttributeView */ @Override public String name() { - return "azureBasic"; + return NAME; } /** diff --git a/sdk/storage/azure-storage-blob-nio/src/main/java/com/azure/storage/blob/nio/AzureBlobFileAttributeView.java b/sdk/storage/azure-storage-blob-nio/src/main/java/com/azure/storage/blob/nio/AzureBlobFileAttributeView.java index 2761c54830b6..241d72e474ae 100644 --- a/sdk/storage/azure-storage-blob-nio/src/main/java/com/azure/storage/blob/nio/AzureBlobFileAttributeView.java +++ b/sdk/storage/azure-storage-blob-nio/src/main/java/com/azure/storage/blob/nio/AzureBlobFileAttributeView.java @@ -29,6 +29,7 @@ public final class AzureBlobFileAttributeView implements BasicFileAttributeView private final ClientLogger logger = new ClientLogger(AzureBlobFileAttributeView.class); static final String ATTR_CONSUMER_ERROR = "Exception thrown by attribute consumer"; + static final String NAME = "azureBlob"; private final Path path; @@ -74,7 +75,7 @@ static Map> setAttributeConsumers(AzureBlobFileAttribut */ @Override public String name() { - return "azureBlob"; + return NAME; } /** diff --git a/sdk/storage/azure-storage-blob-nio/src/main/java/com/azure/storage/blob/nio/AzureFileSystemProvider.java b/sdk/storage/azure-storage-blob-nio/src/main/java/com/azure/storage/blob/nio/AzureFileSystemProvider.java index 4be316c90c83..6fbfea7e2e70 100644 --- a/sdk/storage/azure-storage-blob-nio/src/main/java/com/azure/storage/blob/nio/AzureFileSystemProvider.java +++ b/sdk/storage/azure-storage-blob-nio/src/main/java/com/azure/storage/blob/nio/AzureFileSystemProvider.java @@ -750,9 +750,9 @@ public Map readAttributes(Path path, String s, LinkOption... lin state that "basic" must be supported, so we funnel to azureBasic. */ if (viewType.equals("basic")) { - viewType = "azureBasic"; + viewType = AzureBasicFileAttributeView.NAME; } - if (!viewType.equals("azureBasic") && !viewType.equals("azureBlob")) { + if (!viewType.equals(AzureBasicFileAttributeView.NAME) && !viewType.equals(AzureBlobFileAttributeView.NAME)) { throw LoggingUtility.logError(logger, new UnsupportedOperationException("Invalid attribute view: " + viewType)); } @@ -763,7 +763,7 @@ public Map readAttributes(Path path, String s, LinkOption... lin should at least validate that the attribute is available on a basic view. */ // TODO: Put these strings in constants - if (viewType.equals("azureBasic")) { + if (viewType.equals(AzureBasicFileAttributeView.NAME)) { if (!AzureBasicFileAttributes.ATTRIBUTE_STRINGS.contains(attributeName) && !attributeName.equals("*")) { throw LoggingUtility.logError(logger, new IllegalArgumentException("Invalid attribute. View: " + viewType @@ -781,7 +781,7 @@ public Map readAttributes(Path path, String s, LinkOption... lin // If "*" is specified, add all of the attributes from the specified set. if (attributeName.equals("*")) { Set attributesToAdd; - if (viewType.equals("azureBasic")) { + if (viewType.equals(AzureBasicFileAttributeView.NAME)) { for (String attr : AzureBasicFileAttributes.ATTRIBUTE_STRINGS) { results.put(attr, attributeSuppliers.get(attr).get()); } @@ -843,7 +843,7 @@ public void setAttribute(Path path, String s, Object o, LinkOption... linkOption state that "basic" must be supported, so we funnel to azureBasic. */ if (viewType.equals("basic")) { - viewType = "azureBasic"; + viewType = AzureBasicFileAttributeView.NAME; } // We don't actually support any setters on the basic view. @@ -851,7 +851,7 @@ public void setAttribute(Path path, String s, Object o, LinkOption... linkOption throw LoggingUtility.logError(logger, new IllegalArgumentException("Invalid attribute. View: " + viewType + ". Attribute: " + attributeName)); - } else if (viewType.equals("azureBlob")) { + } else if (viewType.equals(AzureBlobFileAttributeView.NAME)) { Map> attributeConsumers = AzureBlobFileAttributeView.setAttributeConsumers( this.getFileAttributeView(path, AzureBlobFileAttributeView.class, linkOptions)); if (!attributeConsumers.containsKey(attributeName)) { diff --git a/sdk/storage/azure-storage-blob-nio/src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java b/sdk/storage/azure-storage-blob-nio/src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java new file mode 100644 index 000000000000..004d5cfd1e1e --- /dev/null +++ b/sdk/storage/azure-storage-blob-nio/src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java @@ -0,0 +1,95 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. +package com.azure.storage.blob.nio; + +import com.azure.storage.blob.models.BlobHttpHeaders; + +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.net.URI; +import java.net.URISyntaxException; +import java.nio.file.FileSystem; +import java.nio.file.FileSystems; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.StandardCopyOption; +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; + +/** + * WARNING: MODIFYING THIS FILE WILL REQUIRE CORRESPONDING UPDATES TO README.md FILE. LINE NUMBERS + * ARE USED TO EXTRACT APPROPRIATE CODE SEGMENTS FROM THIS FILE. ADD NEW CODE AT THE BOTTOM TO AVOID CHANGING + * LINE NUMBERS OF EXISTING CODE SAMPLES. + * + * Code samples for the README.md + */ +public class ReadmeSamples { + + private FileSystem myFs = FileSystems.newFileSystem(new URI("azb://?account= config = new HashMap<>(); + config.put(AzureFileSystem.AZURE_STORAGE_ACCOUNT_KEY, ""); + config.put(AzureFileSystem.AZURE_STORAGE_FILE_STORES, ""); + FileSystem myFs = FileSystems.newFileSystem(new URI("azb://?account= attributes = Files.readAttributes(filePath, "azureBlob:metadata,headers"); + } + + public void writeAttributesToAFile() throws IOException { + AzureBlobFileAttributeView view = Files.getFileAttributeView(filePath, AzureBlobFileAttributeView.class); + view.setMetadata(Collections.EMPTY_MAP); + } + + public void writeAttributesToAFileString() throws IOException { + Files.setAttribute(filePath, "azureBlob:blobHttpHeaders", new BlobHttpHeaders()); + } +}