Encryption for REST catalog #13225

smaheshwar-pltr · 2025-06-03T21:08:13Z

This PR implements client-side support for REST catalog encryption. With it, clients interacting with a REST catalog can read and write encrypted data.

It is similar to #13066, that integrates encryption with the Hive catalog.

cc @rdblue @RussellSpitzer @ggershinsky

smaheshwar-pltr · 2025-06-03T21:22:42Z

spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/sql/TestRestTableEncryption.java

+// TODO(smaheshwar): This test is taken from https://github.com/apache/iceberg/pull/13066, with the
+// exception of testCtas, but adapted for the REST catalog. When that merges, we can directly use
+// those tests for the REST catalog as well by  adding to the parameters method there, to have a
+// single test class for table encryption.


Highlighting this - this file is largely taken from #13066

smaheshwar-pltr · 2025-06-03T21:26:48Z

core/src/main/java/org/apache/iceberg/rest/RESTTableOperations.java

+              : Integer.parseInt(dekLength);
+    }
+
+    // Force re-creation of encryptingFileIO and encryptionManager


See #13066 (comment)

github-actions · 2025-07-07T00:19:57Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

smaheshwar-pltr · 2025-07-07T12:37:19Z

(Bump to remove staleness)

github-actions · 2025-08-07T00:19:41Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

smaheshwar-pltr · 2025-08-07T13:06:57Z

(Bump to remove staleness)

github-actions · 2025-09-08T00:17:56Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

smaheshwar-pltr · 2025-09-11T17:35:59Z

(Bump to remove staleness)

smaheshwar-pltr · 2025-09-23T15:29:31Z

core/src/main/java/org/apache/iceberg/encryption/StandardEncryptionManager.java

+   * @param dataKeyLength length of data encryption key (16/24/32 bytes)
+   * @param kmsClient Client of KMS used to wrap/unwrap keys in envelope encryption
+   */
+  public StandardEncryptionManager(


Could deprecate the old constructor

smaheshwar-pltr · 2025-09-23T15:32:52Z

spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/sql/TestRestTableEncryption.java

+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.TestTemplate;
+
+// TODO(smaheshwar-pltr): This test is taken from https://github.com/apache/iceberg/pull/13066, with


I added a CTAS test to the ones authored by @ggershinsky here. Perhaps there's a world in which this PR merges first in which case can undo this comment. There are also review comments on these tests in #13066 that are potentially relevant

smaheshwar-pltr · 2025-09-26T13:48:06Z

Given #7770 is merged, curious for thoughts on this PR.
REST integration sounds on the cards. Also happy for this PR to be superseded if other folks have worked on it.

cc @huaxingao @RussellSpitzer @ggershinsky @rdblue

RussellSpitzer · 2025-09-27T00:04:50Z

Could you elaborate on the api additions? I think it would help to have some more description on the general direction of this or

huaxingao · 2025-10-25T01:46:02Z

@smaheshwar-pltr Could you please resolve the conflicts?

ggershinsky · 2025-10-27T13:43:36Z

@huaxingao @smaheshwar-pltr Our team has a person who works on encryption with the REST catalog. If @smaheshwar-pltr does not object, we can follow up on this patch.

ggershinsky · 2025-10-27T13:50:30Z

core/src/main/java/org/apache/iceberg/rest/RESTTableOperations.java

+    return encryptionManager;
+  }
+
+  private void encryptionPropsFromMetadata(TableMetadata metadata) {


Is this method applied on the TableMetadata that is fetched directly from the REST catalog, and not from the metadata.json file? Both are possible, but the former must override (and check) the latter, to protect against the key removal and other attacks.

Yes this method will always be applied on a metadata field of a LoadTableResponse object received directly from the REST catalog (its only usage within this class is as such, and you can check the constructor usages within RESTSessionCatalog to confirm that the metadata coming in from there is as such too.

Question (not sure if there have been discussions here or if your team have thoughts): we want the key ID to come from the REST catalog service directly for security reasons.

It's typical for REST catalogs to provide metadata that corresponds to the metadata file in storage and not modify it apart from that. Given this, would it be preferable to have this field returned within the LoadTableResponse itself, to encourage catalogs to track it explicitly?

The concrete proposal here might be: ENCRYPTION_TABLE_KEY and ENCRYPTION_DEK_LENGTH become properties on the LoadTableResponse's config (mentioned in the REST spec here).

@ggershinsky @huaxingao LMKWYT!

Well I see two scenarios when thinking about this:

metadata.json is something that both the server and the clients can read (although clients wouldn't need to, given they get the metadata with the LoadTableResponse )

metadata.json can only be accessed on the server side and clients are not given FS credentials (either vended or not) to reach it

For case (1) I totally agree, we can't rely on just metadata.json to store these encryption properties, and the catalog should store it separately too, and eventually populating (i.e. doing the override logic referred by @ggershinsky) the properties in the LoadTableResponse to be created.
For case (2) I'm not 100% sure, but still leaning toward the catalog taking on this responsibility.

Either way, for the client side there's not much we can do other than recommending clients to consider the metadata from LoadTableResponse only. The rest (no pun intended) is on the server side to be decided and will be implementation-specific. For this code snippet above, irrelevant IMHO.

Let me know your thoughts.

Afaik, HMS is not optimized for JSON storage. But maybe someone in the community will take on storing the full metadata object there, to improve table security. Having only the table properties is barely sufficient. I think we should recommend REST catalogs for encrypted tables.

It's not. But would storing a hash suffice? If so we could generate the hash of the whole JSON content and store it via an additional (Hive) table property. Then during table loading we can verify that the TableMetadata we just read in from a potentially untrusted storage (and yet metadata.json is not encrypted) is original or has been tampered with.

One aspect of having an encrypted metadata.json is when the table schema is also considered a sensitive piece of information. I haven't found this in the discussions but do you know if this has ever been considered @ggershinsky ?

would storing a hash suffice? If so we could generate the hash of the whole JSON content and store it via an additional (Hive) table property. Then during table loading we can verify that the TableMetadata we just read in from a potentially untrusted storage (and yet metadata.json is not encrypted) is original or has been tampered with.

I think it's a good idea

One aspect of having an encrypted metadata.json is when the table schema is also considered a sensitive piece of information. I haven't found this in the discussions but do you know if this has ever been considered

Not sure. Though, it should be possible to have a REST implementation that hides the metadata.json file from the storage.

I think it's a good idea

Sounds good, I can take this on and will produce a PR shortly.

Not sure. Though, it should be possible to have a REST implementation that hides the metadata.json file from the storage.

Yes, with REST that's true, I just meant it in a general sense, e.g. it's not currently possible to hide the schema of an encrypted table with Hive catalog. It may just be one more thing to note/document as a limitation of encryption wrt. Hive catalog - just merely wanted to highlight this though.

ggershinsky · 2025-10-27T14:00:05Z

Also, it would be good to refactor (if possible) a code common to this PR and to #13066 , so that other catalogs will be able to re-use it.

smaheshwar-pltr · 2025-11-22T21:56:00Z

hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java


      if (removedProps.contains(TableProperties.ENCRYPTION_TABLE_KEY)) {
-        throw new RuntimeException("Cannot remove key in encrypted table");
+        throw new RuntimeException("Cannot remove encryption key ID from an encrypted table");


(Wanted to make this more descriptive on the REST side, changed here too for Hive in this PR so tests pass)

smaheshwar-pltr · 2025-12-02T23:47:28Z

core/src/main/java/org/apache/iceberg/rest/RESTTableOperations.java

+      TableMetadata.Builder builder = TableMetadata.buildFrom(metadata);
+      for (Map.Entry<String, EncryptedKey> entry :
+          EncryptionUtil.encryptionKeys(encryption()).entrySet()) {
+        builder.addEncryptionKey(entry.getValue());


This adds the required REST updates to the metadata to be committed

smaheshwar-pltr · 2025-12-04T01:57:50Z

core/src/main/java/org/apache/iceberg/rest/RESTTableOperations.java

+  // keys loaded from the latest metadata
+  private Optional<List<EncryptedKey>> encryptedKeysFromMetadata = Optional.empty();
+
+  // keys added to EM (e.g. as a result of a FileAppend) but not committed into the latest metadata
+  // yet
+  private Optional<List<EncryptedKey>> encryptedKeysPending = Optional.empty();


The flow here felt a bit complicated to me, so I put up #14752 separately

(I've now added those changes into this PR)

smaheshwar-pltr · 2025-12-05T19:47:01Z

core/src/main/java/org/apache/iceberg/rest/RESTTableOperations.java

+  private String tableKeyId;
+  private int encryptionDekLength;
+
+  private List<EncryptedKey> encryptedKeys = List.of();


This prevents key loss during transactions, see #14752

ggershinsky

Thanks @smaheshwar-pltr !

github-actions · 2026-01-09T00:19:06Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

github-actions bot added spark core labels Jun 3, 2025

smaheshwar-pltr commented Jun 3, 2025

View reviewed changes

smaheshwar-pltr marked this pull request as ready for review June 3, 2025 21:24

smaheshwar-pltr mentioned this pull request Jun 3, 2025

Encryption integration and test #13066

Merged

smaheshwar-pltr commented Jun 3, 2025

View reviewed changes

smaheshwar-pltr mentioned this pull request Jun 8, 2025

AWS: KeyManagementClient implementation that works with AWS KMS #13136

Merged

github-actions bot added the stale label Jul 7, 2025

github-actions bot removed the stale label Jul 8, 2025

github-actions bot added the stale label Aug 7, 2025

github-actions bot removed the stale label Aug 8, 2025

github-actions bot added the stale label Sep 8, 2025

github-actions bot removed the stale label Sep 12, 2025

smaheshwar-pltr force-pushed the rest-encrypt-on-main-cherry-pick branch from d007aa2 to 889cbbf Compare September 23, 2025 15:23

smaheshwar-pltr mentioned this pull request Sep 23, 2025

Encryption for REST catalog #13224

Closed

smaheshwar-pltr commented Sep 23, 2025

View reviewed changes

ggershinsky reviewed Oct 27, 2025

View reviewed changes

github-actions bot added the build label Oct 27, 2025

smaheshwar-pltr changed the title ~~REST encryption integration~~ Encryption for REST catalog Nov 22, 2025

smaheshwar-pltr force-pushed the rest-encrypt-on-main-cherry-pick branch from ed1165b to da894ba Compare November 22, 2025 18:04

smaheshwar-pltr mentioned this pull request Nov 22, 2025

Hive encryption nits #14659

Open

smaheshwar-pltr force-pushed the rest-encrypt-on-main-cherry-pick branch 3 times, most recently from 49114e7 to 11ff326 Compare November 22, 2025 21:55

github-actions bot added the hive label Nov 22, 2025

smaheshwar-pltr commented Nov 22, 2025

View reviewed changes

szlta mentioned this pull request Nov 25, 2025

Hive: Metadata integrity check for encrypted tables #14685

Merged

smaheshwar-pltr commented Dec 2, 2025

View reviewed changes

smaheshwar-pltr force-pushed the rest-encrypt-on-main-cherry-pick branch from 088c5e4 to 64e958a Compare December 2, 2025 23:52

smaheshwar-pltr mentioned this pull request Dec 4, 2025

Encryption: Simplify Hive key handling and add transaction tests #14752

Merged

smaheshwar-pltr commented Dec 4, 2025

View reviewed changes

smaheshwar-pltr force-pushed the rest-encrypt-on-main-cherry-pick branch 3 times, most recently from 73c351b to 4d4fd55 Compare December 5, 2025 19:24

Encryption for REST catalog

af672ed

smaheshwar-pltr force-pushed the rest-encrypt-on-main-cherry-pick branch from 4d4fd55 to af672ed Compare December 5, 2025 19:33

Nit improvements

7ad846d

smaheshwar-pltr commented Dec 5, 2025

View reviewed changes

smaheshwar-pltr requested review from ggershinsky, huaxingao and singhpk234 December 5, 2025 19:47

Suppress test method length

9ed8d8e

ggershinsky approved these changes Dec 8, 2025

View reviewed changes

smaheshwar-pltr added 2 commits December 9, 2025 16:40

Merge branch 'main' into rest-encrypt-on-main-cherry-pick

2d18a6a

Spotless

745a959

singhpk234 mentioned this pull request Dec 29, 2025

Add Polaris blog about KMS apache/polaris#3331

Merged

6 tasks

github-actions bot added the stale label Jan 9, 2026

Encryption for REST catalog #13225

Are you sure you want to change the base?

Encryption for REST catalog #13225

Conversation

smaheshwar-pltr commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 7, 2025

Uh oh!

smaheshwar-pltr commented Jul 7, 2025

Uh oh!

github-actions bot commented Aug 7, 2025

Uh oh!

smaheshwar-pltr commented Aug 7, 2025

Uh oh!

github-actions bot commented Sep 8, 2025

Uh oh!

smaheshwar-pltr commented Sep 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smaheshwar-pltr commented Sep 26, 2025

Uh oh!

RussellSpitzer commented Sep 27, 2025

Uh oh!

huaxingao commented Oct 25, 2025

Uh oh!

ggershinsky commented Oct 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ggershinsky commented Oct 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ggershinsky left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

smaheshwar-pltr commented Jun 3, 2025 •

edited

Loading