Azure Storage Gen2 (hierarchical namespaces) - use DFS endpoint to improve performance

[keen85]

Hi,
Azure Storage comes in different flavors. A "regular" Azure Storage account is a classic object store and it comes with blob endpoint (blob.core.windows.net, blob.fabric.windows.com).

Azure Storage Account with hierarchical namespaces enabled (aka. Azure Data Lake Gen2, Fabric OneLake) come with a second "DFS" (Distributed Filesystem) endpoint: dfs.core.windows.net (dfs.fabric.windows.com) that has it's own REST-API: https://learn.microsoft.com/en-us/rest/api/storageservices/data-lake-storage-gen2

This endpoint allows for potentially massive performance improvements in the following scenarios:

• recursive directory listings
• renaming or moving files or folders
• deleting directories with many files

These are single atomic operations via DFS. With the Blob endpoint they require many individual blob operations and are therefore slower.

To my understanding, DFS-URLs are already accepted but ultimately only the blob endpoint is used by object_store.

I propose implementing the DFS endpoint in object_store and introducing a feature-flag allowing user to specify if DFS endpoint should be used.

[tustvold]

In general object_store intentionally tries to abstract away the notion of directories as not only are they a portability headache, as many stores don't support them, the vast majority of workloads actively don't want to care about them - instead delegating catalog responsibilities to something better suited to this, e.g. an RDBMS, Deltalake, iceberg, etc...

I'm therefore not sure how useful this would be.

Edit: see also #284

recursive directory listings

Why would it provide better performance in this case?

[keen85]

Hi @tustvold

recursive directory listings

Why would it provide better performance in this case?

Because the DFS endpoint exposes a filesystem-like API that supports recursive listing in one call. The Blob endpoint is object-store oriented: it only lists a flat set of objects under a given prefix. To traverse a hierarchy recursively with the Blob API, the client must repeatedly enumerate each "folder level" and stitch results together. That means many network round-trips and higher latency.

The DFS endpoint, by contrast, is built for hierarchical namespaces. Its List Path API accepts a recursive=true flag, so the storage service itself performs the traversal server-side and streams all matching entries back in a single operation. This avoids multiple client-side iterations and accelerates recursive listings, especially in deep or large directory trees.

[tustvold]

Because the DFS endpoint exposes a filesystem-like API that supports recursive listing in one call. The Blob endpoint is object-store oriented: it only lists a flat set of objects under a given prefix. To traverse a hierarchy recursively with the Blob API, the client must repeatedly enumerate each "folder level" and stitch results together. That means many network round-trips and higher latency.

This isn't the case, if you don't specify a delimiter it lists recursively

See https://learn.microsoft.com/en-us/rest/api/storageservices/list-blobs?tabs=microsoft-entra-id

This is what is used by https://docs.rs/object_store/latest/object_store/trait.ObjectStore.html#tymethod.list with only https://docs.rs/object_store/latest/object_store/trait.ObjectStore.html#tymethod.list_with_delimiter specifying a delimiter.

[keen85]

@tustvold. thanks for the explanation. Probably you are right for this scenario.

Still, when having a container where there are 10 million files, scattered across 100 thousand folders, listing the content of a particular folder will be much faster using DFS endpoint, right?

When using Blob endpoint, Storage service would scans the entire container index to find blobs matching the desired folder prefix.
When using DFS endpoint, the Storage service can access the folder’s child entries directly - no search involved here.

[tustvold]

If anything I would expect the blob endpoint to be faster - it is literally just a range-scan on a sorted map of keys, most likely some sort of BTreeMap. It does not get simpler from a data-retrieval perspective.

The DFS endpoint by comparison is significantly more complicated as it has to do a recursive traversal, performing a separate range scan at each level. This is much more complicated to implement.

Now the reality is that when hierarchical namespaces are enabled, Azure is probably forced to store data in the hierarchical fashion, which will make recursive listing equally more expensive for both APIs. I would not expect either API to perform better than the other for this use-case, the fact directories exist in the data model will hurt both equally.