Support hierarchical namespace in Azure by dimas-b · Pull Request #3347 · apache/polaris

dimas-b · 2026-01-02T17:03:42Z

Add hierarchical to AzureStorageConfigInfo (the default is unset translating to current behaviour).
Use DataLakeDirectoryClient instead of DataLakeFileSystemClient for generating SAS tokens when hierarchical is set to true.
Add cloudTest classes for testing with credential vending in ADLS.

Checklist

🛡️ Don't disclose security issues! (contact security@apache.org)
🔗 Clearly explained why the changes are needed, or linked related issues: Fixes #
🧪 Added/updated tests with good coverage, or manually tested (and explained how)
💡 Added comments for complex logic
🧾 Updated CHANGELOG.md (if needed)
📚 Updated documentation in site/content/in-dev/unreleased (if needed)

evindj

Overall looks good to me, I do think we need to clarify the behavior when the feature is enabled but the account on azure does not have hierarchical namespace enabled.

evindj · 2026-01-02T17:57:06Z

spec/polaris-management-service.yml

+              description: >-
+                If set to `true`, instructs Polaris Servers to scope SAS tokens down to the most specific path
+                in the storage container (in most cases the table's base location). This flag should be set only
+                if hierarchical namespace is enabled in the Azure storage account.


What is the behavior if the flag is set but the Azure account does not have the feature enabled?
I am wondering if in a future iteration there will be a way to enable this feature based on whether or not the feature is enabled in Azure.

evindj · 2026-01-02T17:59:26Z

runtime/service/src/test/java/org/apache/polaris/service/entity/CatalogEntityTest.java

    CatalogEntity catalogEntity =
        new CatalogEntity.Builder()
-            .setName("testAwsConfigRoundTrip")
+            .setName("testStorageConfigRoundTrip")


+1 for removing AWS specific here.

evindj · 2026-01-02T19:06:21Z

.../src/main/java/org/apache/polaris/core/storage/azure/AzureCredentialsStorageIntegration.java

+            "Allowed read locations must not have more that one entry");
+        Preconditions.checkArgument(
+            allowedWriteLocations.size() <= 1,
+            "Allowed write locations must not have more that one entry");


N00b question for my own understanding, what is the use case for several allowedReadLocations and allowedWriteLocations?
Also why does this apply only to hierarchical use case?

what is the use case for several allowedReadLocations and allowedWriteLocations?

TBH, I do not really know 😅 Currently, only one location is passed in.

Azure can restrict SAS to only location (base directory or specific file).

dimas-b · 2026-01-05T15:32:39Z

@evindj :

do think we need to clarify the behavior when the feature is enabled but the account on azure does not have hierarchical namespace enabled.

Good point! Thanks for flagging it. I'll update the description in YAML.

snazy

The change looks good overall!

Please do not forget to update the CLI as well.

I was wondering whether we could implement this without introducing a configuration flag, but could not find a (cheap) way to figure out whether a file-system is hierarchical or not. So I guess there's no (performance neutral) way beside the introduced config flag.

.../src/main/java/org/apache/polaris/core/storage/azure/AzureCredentialsStorageIntegration.java

snazy · 2026-01-07T11:54:51Z

.../src/main/java/org/apache/polaris/core/storage/azure/AzureCredentialsStorageIntegration.java

+        return new DataLakePathClientBuilder()
+            .endpoint(endpoint)
+            .fileSystemName(fileSystemNameOrContainer)
+            .pathName(path) // TODO: drop authority part


jbonofre

LGTM, thanks !

snazy · 2026-01-09T07:55:08Z

...-core/src/main/java/org/apache/polaris/core/storage/azure/AzureStorageConfigurationInfo.java


+  /** The flag indicating whether the storage account supports hierarchical namespaces. */
+  @Nullable
+  public abstract Boolean isHierarchical();


Suggested change

public abstract Boolean isHierarchical();

@JsonInclude(JsonInclude.Include.NON_NULL)

public abstract Boolean isHierarchical();

Could actually also be (primitive):

Suggested change

public abstract Boolean isHierarchical();

@JsonInclude(JsonInclude.Include.NON_DEFAULT)

public abstract boolean isHierarchical();

Both help with serialization size and backwards compatiblity.

Not a blocker tho

An object Boolean is meant to allow distinguishing unset from false in the (generated) AzureStorageConfigInfo class. A simple boolean would be exposed to clients even it it was not explicitly set.

I'd rather not change AzureStorageConfigInfo serialization in this PR :)

Include.NON_NULL is implied:

polaris/polaris-core/src/main/java/org/apache/polaris/core/storage/PolarisStorageConfigurationInfo.java

Line 95 in 992a428

JsonInclude.Include.NON_NULL, JsonInclude.Include.NON_NULL))

* Add `hierarchical` to `AzureStorageConfigInfo` (the default is unset translating to current behaviour). * Use `DataLakeDirectoryClient` instead of `DataLakeFileSystemClient` for generating SAS tokens when `hierarchical` is set to `true`. * Add `cloudTest` classes for testing with credential vending in ADLS.

dimas-b · 2026-01-09T18:51:46Z

rebased to resolve CHANGELOG.md conflicts

snazy

+1, just don't forget the corresponding CLI changes.

Following up on apache#3347 this change adds the `--hierarchical` option to Polaris CLI in order to allow configuring this storage flag in Azure-based Catalogs.

dimas-b · 2026-01-13T01:55:34Z

CLI part: #3426

Following up on apache#3347 this change adds the `--hierarchical` option to Polaris CLI in order to allow configuring this storage flag in Azure-based Catalogs.

* Add `--hierarchical` to Polaris CLI Following up on #3347 this change adds the `--hierarchical` option to Polaris CLI in order to allow configuring this storage flag in Azure-based Catalogs.

* Support hierarchical namespace in Azure * Add `hierarchical` to `AzureStorageConfigInfo` (the default is unset translating to current behaviour). * Use `DataLakeDirectoryClient` instead of `DataLakeFileSystemClient` for generating SAS tokens when `hierarchical` is set to `true`. * Add `cloudTest` classes for testing with credential vending in ADLS.

* Add `--hierarchical` to Polaris CLI Following up on apache#3347 this change adds the `--hierarchical` option to Polaris CLI in order to allow configuring this storage flag in Azure-based Catalogs.

* feat: Add AWS STS Session Tags support for credential vending (apache#3327) * feat: Add AWS STS Session Tags support for credential vending Fixes apache#3325 * Minor fixes * Review comments * Review comments. * Remove request_id from the PR * Review comment: added activated roles * Review comments * Update AwsSessionTagsBuilder.java * Spotless fixes * Update StorageCredentialCacheKey.java * fix failing tests * Review comments. * Reverting last commit based on review comments for cache collisions. --------- Co-authored-by: Anand Kumar Sankaran <anand.sankaran@workday.com> * chore(deps): update registry.access.redhat.com/ubi9/openjdk-21-runtime docker tag to v1.24-2.1767639862 (apache#3383) * Added HttpRoute and Gateway to Helm Chart (apache#3314) * Added httproute and gateway * added to readme * updated helm site docs * updated changelog * Added tests * fixed broken test * fixed test part 2 * removed odd comment * added check for httproute and gateway * shuffled the gateway documentation * better gateway instructions * removed extra case in validateRouting * Errorprone: prepare for v2.46.0 (apache#3384) This tackles the current failure in apache#3382: `-XDaddTypeAnnotationsToSymbol=true is required by Error Prone on JDK 21` * NoSQL: Metastore maintenance (apache#3268) Implementation of the NoSQL meta-store maintenance implementation. It adds the meta-store specific handling to the existing NoSQL maintenance service to purge unreferenced and unneeded data from the database. * Update release workflows to use the new GPG private key (apache#3399) * [tech debt] Cleanup `gradle/libs.versions.toml` (apache#3394) This change removes unreferenced dependency references, inlines single usages of a version-reference and adds a dependency for checkstyle to get that version managed by renovate. * fix(deps): update dependency org.keycloak:keycloak-admin-client to v26.0.8 (apache#3405) * fix(deps): update dependency com.google.errorprone:error_prone_core to v2.46.0 (apache#3382) * [doc]: Add Minio OSS disclaimer (apache#3390) * [Releasy] Let Maven artifact publication propagate failures (apache#3407) The Gradle build to publish the Maven artifacts is invoked like `./gradlew ... | tee <log>`. The (overall) exit code of pipes is the exit code of the _last_ command. The exit codes of all pipe "parts" is available in bash's `PIPESTATUS` array and needs to be checked. * Fix Gradle up-to-date of jars (apache#3401) The change apache#1036 added the Maven pom.xml to all built jars. However, the `GenerateMavenPom` Gradle task type is never up-to-date, in consequence no jar can ever be up-to-date, leading to unnecessarily longer build times. This change ensures that the pom.xml is included in release builds, but not in developer/snapshot builds. * fix(deps): update dependency net.ltgt.gradle:gradle-errorprone-plugin to v4.4.0 (apache#3406) * fix(deps): update dependency com.puppycrawl.tools:checkstyle to v13 (apache#3403) * fix(deps): update dependency io.opentelemetry:opentelemetry-bom to v1.58.0 (apache#3408) * fix(deps): update dependency software.amazon.awssdk:bom to v2.41.5 (apache#3416) * chore(deps): update dependency jupyterlab to v4.5.2 (apache#3419) * Release workflows should retry svn checkout in case of failure (apache#3393) * Change parser option to be required (apache#3413) * Support hierarchical namespace in Azure (apache#3347) * Support hierarchical namespace in Azure * Add `hierarchical` to `AzureStorageConfigInfo` (the default is unset translating to current behaviour). * Use `DataLakeDirectoryClient` instead of `DataLakeFileSystemClient` for generating SAS tokens when `hierarchical` is set to `true`. * Add `cloudTest` classes for testing with credential vending in ADLS. * chore(test): Increase Authorization Test Coverage (apache#3332) * increase test coverage * update docstrings * clean up * make use of same helper method * remove duplicate tests from OpaIntegrationTest * use tempdir * fix(deps): update dependency io.micrometer:micrometer-bom to v1.16.2 (apache#3422) * chore(deps): update registry.access.redhat.com/ubi9/openjdk-21-runtime docker tag to v1.24-2.1767878250 (apache#3421) * Last merged commit 1996156 * Add free-disk-space action * Add free-disk-space action to regtest + spark_client_regtests --------- Co-authored-by: Anand K Sankaran <lists@anands.net> Co-authored-by: Anand Kumar Sankaran <anand.sankaran@workday.com> Co-authored-by: Mend Renovate <bot@renovateapp.com> Co-authored-by: cccs-cat001 <56204545+cccs-cat001@users.noreply.github.com> Co-authored-by: Pierre Laporte <pierre@pingtimeout.fr> Co-authored-by: Yong Zheng <yongzheng0809@gmail.com> Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@gmail.com> Co-authored-by: Sung Yun <107272191+sungwy@users.noreply.github.com>

* Flatten events hierarchy (apache#3293) Co-authored-by: Alexandre Dutra <adutra@apache.org> * (feat)Python CLI: Switch from Poetry to UV for python package management (apache#3410) * chore(deps): update dependency uv to v0.9.24 (apache#3430) * (doc): Fix Polaris getting started doc and docker-compose (apache#3425) * Fix Polaris getting started doc * Fix Polaris getting started doc * [Minor] [Site] fix scheduled meetings table (apache#3423) * NoSQL: add to config-docs (apache#3397) Add the NoSQL specific configurtion options to the configuration docs generation module. * Blog: Add blog for Lance-Polaris integration (apache#3424) * Add `--hierarchical` to Polaris CLI (apache#3426) * Add `--hierarchical` to Polaris CLI Following up on apache#3347 this change adds the `--hierarchical` option to Polaris CLI in order to allow configuring this storage flag in Azure-based Catalogs. * Use new Request Context for each realm during implicit bootstrap (apache#3411) * Use new Request Context for each realm during implicit bootstrap The implicit (auto) bootstrap calls used to share Request Context for potentially many realms. That used to work by coincidence because `RealmConfig`, for example, is a `RequestScoped` bean. With this change each realm will be bootstrapped in its own dedicated Request Context. This change lays down a foundation for future refactoring related to `RealmConfig`. * Change nested docs to use title case (apache#3432) * fix(deps): update dependency com.github.dasniko:testcontainers-keycloak to v4.1.1 (apache#3438) * Fix Helm doc note section under Gateway API (apache#3436) * Relax UV version (apache#3437) * fix(deps): update dependency org.jboss.weld.se:weld-se-core to v6.0.4.final (apache#3439) * Add free-disk-space action to regtest + spark_client_regtests (apache#3429) The "Spark Client Regression Tests" CI job requires some disk space to operate. With just a little bit of added "content", the job will fail to `no space left on device` during the `docker compose` invocation building an image. Such errors make it impossible to get the log from the workflow, unless you capture the log before the workflow runs into the `no space left on device` situation. With "no space left", GitHub workflow infra is unable to capture the logs. ``` #10 ERROR: failed to copy files: userspace copy failed: write /home/spark/polaris/v3.5/integration/build/2.13/quarkus-build/gen/quarkus-app/lib/main/com.google.http-client.google-http-client-1.47.1.jar: no space left on device ``` This change is a stop-gap solution to prevent this error from happening for now. * fix(deps): update dependency com.google.cloud:google-cloud-iamcredentials to v2.82.0 (apache#3449) * Update OPA docker image version (apache#3448) * Blog: Mapping Legacy and Heterogeneous Datalakes in Apache P… (apache#3417) * fix(deps): update dependency org.postgresql:postgresql to v42.7.9 (apache#3453) * chore(deps): update apache/spark docker tag to v3.5.8 (apache#3458) * fix(deps): update dependency org.apache.spark:spark-sql_2.12 to v3.5.8 (apache#3450) * site: add blog anchors (apache#3443) * render anchor * improve readme * RAT * fix(deps): update dependency com.google.cloud:google-cloud-storage-bom to v2.62.0 (apache#3455) * Update renovate to include docker file with suffix (apache#3454) * feat: Add trace_id to AWS STS session tags for end-to-end correlation (apache#3414) * feat: Add trace_id to AWS STS session tags for end-to-end correlation This change enables deterministic correlation between: - Catalog operations (Polaris events) - Credential vending (AWS CloudTrail via STS session tags) - Metrics reports from compute engines (Spark, Trino, etc.) Changes: 1. Add traceId field to CredentialVendingContext - Marked with @Value.Auxiliary to exclude from cache key comparison - Every request has unique trace ID, so including it in equals/hashCode would prevent all cache hits - Trace ID is for correlation/audit only, not authorization 2. Extract OpenTelemetry trace ID in StorageAccessConfigProvider - getCurrentTraceId() extracts trace ID from current span context - Populates CredentialVendingContext.traceId for each request 3. Add trace_id to AWS STS session tags - AwsSessionTagsBuilder includes trace_id in session tags - Appears in CloudTrail logs for correlation with catalog operations - Uses 'unknown' placeholder when trace ID is not available 4. Update tests to verify trace_id is included in session tags This enables operators to correlate: - Which catalog operation triggered credential vending - Which data access events in CloudTrail correspond to catalog operations - Which metrics reports correspond to specific catalog operations * Update AwsCredentialsStorageIntegrationTest.java * Review comments 1. Feature Flag to Disable Trace IDs in Session Tags Added a new feature configuration flag INCLUDE_TRACE_ID_IN_SESSION_TAGS in FeatureConfiguration.java: polaris-core/src/main/java/org/apache/polaris/core/config/FeatureConfiguration.java (EXCERPT) public static final FeatureConfiguration<Boolean> INCLUDE_TRACE_ID_IN_SESSION_TAGS = PolarisConfiguration.<Boolean>builder() .key("INCLUDE_TRACE_ID_IN_SESSION_TAGS") .description("If set to true (and INCLUDE_SESSION_TAGS_IN_SUBSCOPED_CREDENTIAL is also true), ...") .defaultValue(false) .buildFeatureConfiguration(); 2. Cache Key Correctness Solution The solution ensures cache correctness by including trace IDs in cache keys only when they affect the vended credentials: Key changes: 1. `StorageCredentialCacheKey` - Added a new traceIdForCaching() field that is populated only when trace IDs affect credentials: polaris-core/src/main/java/org/apache/polaris/core/storage/cache/StorageCredentialCacheKey.java (EXCERPT) @Value.Parameter(order = 10) Optional<String> traceIdForCaching(); 2. `StorageCredentialCache` - Reads both flags and includes trace ID in cache key only when both are enabled: polaris-core/src/main/java/org/apache/polaris/core/storage/cache/StorageCredentialCache.java (EXCERPT) boolean includeTraceIdInCacheKey = includeSessionTags && includeTraceIdInSessionTags; StorageCredentialCacheKey key = StorageCredentialCacheKey.of(..., includeTraceIdInCacheKey); 3. `AwsSessionTagsBuilder` - Conditionally includes trace ID based on the new flag. 4. Tests - Updated existing tests and added a new test testSessionTagsWithTraceIdWhenBothFlagsEnabled. How This Resolves the Cache Correctness vs. Efficiency Trade-off | Configuration | Trace ID in Session Tags | Trace ID in Cache Key | Caching Behavior | |---------------|--------------------------|----------------------|------------------| | Session tags disabled | No | No | Efficient caching | | Session tags enabled, trace ID disabled (default) | No | No | Efficient caching | | Session tags enabled, trace ID enabled | Yes | Yes | Correct but no caching across requests | This design ensures: • Correctness: When trace IDs affect credentials, they're included in the cache key • Efficiency: When trace IDs don't affect credentials, they're excluded from the cache key, allowing cache hits across requests * Update CHANGELOG.md Co-authored-by: Anand Kumar Sankaran <anand.sankaran@workday.com> * site: Update website for 1.3.0 (apache#3464) * site: Fix blog diagram with corrected architecture image (apache#3466) * Site: Add 20260108 Community Meeting (apache#3460) * CI: CLI Nightly build (apache#3457) * Fix Helm repository update after release vote (apache#3461) The Github workflow included a `svn add index.yaml` command which would be correct if it was a Git repository. But in SVN, this results in an error when the file is already under version control. This line is unnecessary and a simple `svn commit` results in pushing the changes to the SVN server. * Fix typo for the wrong reference (apache#3473) * chore(deps): update apache/ozone docker tag to v2.1.0 (apache#3364) * chore(deps): update docker.io/prom/prometheus docker tag to v3.9.1 (apache#3366) * chore(deps): update quay.io/keycloak/keycloak docker tag to v26.5.1 (apache#3362) * Last merged commit 1451ce4 --------- Co-authored-by: Oleg Soloviov <40199597+olsoloviov@users.noreply.github.com> Co-authored-by: Alexandre Dutra <adutra@apache.org> Co-authored-by: Yong Zheng <yongzheng0809@gmail.com> Co-authored-by: Mend Renovate <bot@renovateapp.com> Co-authored-by: Danica Fine <danica.fine@gmail.com> Co-authored-by: Jack Ye <jackye@apache.org> Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@gmail.com> Co-authored-by: Maninder <parmar.maninderjit@gmail.com> Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com> Co-authored-by: Anand K Sankaran <lists@anands.net> Co-authored-by: Anand Kumar Sankaran <anand.sankaran@workday.com> Co-authored-by: Pierre Laporte <pierre@pingtimeout.fr> Co-authored-by: JB Onofré <jbonofre@apache.org> Co-authored-by: Honah (Jonas) J. <honahx@apache.org>

github-project-automation bot added this to Basic Kanban Board Jan 2, 2026

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Jan 2, 2026

evindj reviewed Jan 2, 2026

View reviewed changes

dimas-b force-pushed the hierarchical-sas branch from 87e57ea to 35afeda Compare January 5, 2026 16:09

snazy reviewed Jan 7, 2026

View reviewed changes

jbonofre previously approved these changes Jan 7, 2026

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Jan 7, 2026

dimas-b dismissed jbonofre’s stale review via 2ea6475 January 8, 2026 16:42

dimas-b force-pushed the hierarchical-sas branch from 35afeda to 2ea6475 Compare January 8, 2026 16:42

dimas-b requested a review from snazy January 8, 2026 16:43

snazy reviewed Jan 9, 2026

View reviewed changes

dimas-b added 4 commits January 9, 2026 13:18

review: clarify behaviour with non-hierarchical storage

56ef418

review: remove local debugging log

a984fce

review: remove irrelevant TODO comment

c24fce3

dimas-b force-pushed the hierarchical-sas branch from 2ea6475 to c24fce3 Compare January 9, 2026 18:51

snazy approved these changes Jan 12, 2026

View reviewed changes

adutra approved these changes Jan 12, 2026

View reviewed changes

dimas-b merged commit e5b5122 into apache:main Jan 12, 2026
15 checks passed

github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board Jan 12, 2026

dimas-b deleted the hierarchical-sas branch January 12, 2026 22:41

dimas-b added a commit to dimas-b/polaris that referenced this pull request Jan 13, 2026

Add --hierarchical to Polaris CLI

aeec8fe

Following up on apache#3347 this change adds the `--hierarchical` option to Polaris CLI in order to allow configuring this storage flag in Azure-based Catalogs.

dimas-b mentioned this pull request Jan 13, 2026

Add --hierarchical to Polaris CLI #3426

Merged

6 tasks

dimas-b added a commit to dimas-b/polaris that referenced this pull request Jan 13, 2026

Add --hierarchical to Polaris CLI

ad881f2

Following up on apache#3347 this change adds the `--hierarchical` option to Polaris CLI in order to allow configuring this storage flag in Azure-based Catalogs.

	public abstract Boolean isHierarchical();
	@JsonInclude(JsonInclude.Include.NON_NULL)
	public abstract Boolean isHierarchical();

	public abstract Boolean isHierarchical();
	@JsonInclude(JsonInclude.Include.NON_DEFAULT)
	public abstract boolean isHierarchical();

Conversation

dimas-b commented Jan 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

evindj left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b commented Jan 5, 2026

Uh oh!

snazy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jbonofre left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b commented Jan 9, 2026

Uh oh!

snazy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dimas-b commented Jan 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dimas-b commented Jan 2, 2026 •

edited

Loading

dimas-b Jan 9, 2026 •

edited

Loading