Manifest list encryption #7770

ggershinsky · 2023-06-05T06:09:39Z

No description provided.

ggershinsky · 2024-03-26T05:38:55Z

api/src/main/java/org/apache/iceberg/Snapshot.java

@@ -162,6 +162,15 @@ default Iterable<DeleteFile> removedDeleteFiles(FileIO io) {
   */
  String manifestListLocation();

+  /**
+   * Return the size of this snapshot's manifest list. For encrypted tables, a verified plaintext


fix comment

core/src/main/java/org/apache/iceberg/BaseSnapshot.java

RussellSpitzer · 2024-03-28T16:03:42Z

api/src/main/java/org/apache/iceberg/Snapshot.java

@@ -162,6 +162,25 @@ default Iterable<DeleteFile> removedDeleteFiles(FileIO io) {
   */
  String manifestListLocation();

+  /**
+   * Return the size of this snapshot's manifest list file. Must be a verified value, taken from a


Confused here, we have a default of -1 as well set in base Snapshot which seemed to also be allowed as an "unset". Should we mention that here or is it always required?

We can define this field to be required only for encrypted tables. It will be not set in the snapshot file for unencrypted tables - where this method can return 0 (or -1, I'll make it consistent across all implementation classes).

RussellSpitzer · 2024-03-28T16:08:06Z

core/src/main/java/org/apache/iceberg/BaseSnapshot.java

+      if (manifestListKeyMetadata != null) { // encrypted manifest list file
+        Preconditions.checkArgument(
+            fileIO instanceof EncryptingFileIO,
+            "No encryption in FileIO class " + fileIO.getClass());


Cannot read manifest list (%s) because it is encrypted but the configured FileIO (%s) does not implement EncryptingFileIO)

RussellSpitzer · 2024-03-28T16:09:21Z

core/src/main/java/org/apache/iceberg/BaseSnapshot.java

+        EncryptingFileIO encryptingFileIO = (EncryptingFileIO) fileIO;
+        Preconditions.checkArgument(
+            encryptingFileIO.encryptionManager() instanceof StandardEncryptionManager,
+            "Encryption manager for encrypted manifest list files can currently only be an instance of "


Cannot decrypt manifest list (%s) because the encryption manager (%s) does not implement StandardEncryptionManager

RussellSpitzer · 2024-03-28T16:23:26Z

core/src/main/java/org/apache/iceberg/SnapshotParser.java

+      generator.writeStringField(MANIFEST_LIST_KEY_METADATA, snapshot.manifestListKeyMetadata());
+    }
+
+    // TODO discuss: do we need to sign the size value? Or sign the whole snapshot?


How would this attack work? Wouldn't the user also need the key to encrypt the replacement files? I thought we were storing the metadata.json key in the catalog so an attacker could replace everything but still not be able to trick a client using the catalog.

This is the question. Some thoughts on the scenarios and protection options:

currently, we don't have a metadata.json key. We have only a key for snapshot's manifest list file. Besides using it for encrypting the manifest list file, we can also use this key for signing snapshot's sensitive parts like the manifest list size field. Or for signing the whole metadata.json file (should be possible with some effort) - then we also protect the integrity of e.g. the table properties (like the table key id).

snapshot (metadata.json file) doesn't keep secret values, so encrypting it might not be required. The signatures, mentioned above, would be kept in added snapshot fields - sufficient for detecting the file modification attacks.

these protection techniques are not required with the REST catalog - because we trust the catalog service (we don't trust the storage service). Since the whole snapshot is stored in the REST catalog, we don't need to sign anything.

the manifest list key is not stored in the catalog. Instead, it is wrapped in a KMS with the table master key, and stored in the snapshot MANIFEST_LIST_KEY_METADATA field. Only the KMS-authorized (for the table key) users/processes will be able to get the manifest list key.

In catalogs other than the REST, the signatures provide a partial protection - because the metadata.json is kept in the untrusted storage. With the signatures, it can't be modified. But the whole folder can be replaced (e.g. a replay attack - where all table files are removed, and replaced with files of an older version of the table). To prevent this attack in non-REST catalogs, we will have to update the catalog per each table snapshot (setting eg the latest table version/sequence number, or a random AAD prefix)

RussellSpitzer · 2024-03-28T16:24:44Z

core/src/main/java/org/apache/iceberg/SnapshotProducer.java

+        && encryptedManifestList.keyMetadata().buffer() != null) {
+      Preconditions.checkArgument(
+          encryptionManager instanceof StandardEncryptionManager,
+          "Encryption manager for encrypted manifest list files can currently only be an instance of "


similar comment as above. "Cannot X because Y"

RussellSpitzer · 2024-03-28T16:28:44Z

core/src/test/java/org/apache/iceberg/TestManifestEncryption.java

@@ -85,7 +85,7 @@ public class TestManifestEncryption {

  private static final DataFile DATA_FILE =
      new GenericDataFile(
-          0,
+          SPEC.specId(),


completing the previous patch #8252 (comment)

RussellSpitzer · 2024-03-28T16:30:08Z

core/src/test/java/org/apache/iceberg/TestManifestListEncryption.java

+  private static final long EXISTING_ROWS = 857273L;
+  private static final int DELETED_FILES = 1;
+  private static final long DELETED_ROWS = 22910L;
+  private static final List<ManifestFile.PartitionFieldSummary> PARTITION_SUMMARIES =


We probably need a test example which has a non-empty list of partition field summaries

RussellSpitzer · 2024-03-28T16:31:23Z

core/src/test/java/org/apache/iceberg/TestManifestListEncryption.java

+
+  @Test
+  public void testV2Write() throws IOException {
+    ManifestFile manifest = writeAndReadManifestList();


nit: writeAndReadEncryptedManifestList

RussellSpitzer · 2024-03-28T16:33:32Z

core/src/test/java/org/apache/iceberg/TestManifestListEncryption.java

+  public void testV2Write() throws IOException {
+    ManifestFile manifest = writeAndReadManifestList();
+
+    // all v2 fields should be read correctly


Assert J has some helper for this, Not sure if it is correct

assertThat(actual).usingRecursiveComparison().isEqualTo(expected);

This unitest/code is mostly copy/paste from https://github.com/apache/iceberg/blob/main/core/src/test/java/org/apache/iceberg/TestManifestListVersions.java#L137

RussellSpitzer · 2024-03-28T21:34:00Z

core/src/main/java/org/apache/iceberg/SnapshotParser.java

+
+    // TODO discuss: do we need to sign the size value? Or sign the whole snapshot?
+    // Or rely on REST catalog? - the only option that prevents "full folder replacement" attack.
+    if (snapshot.manifestListSize() >= 0) {


another small question here, we essentially are doing a transform here

manifestlists sizes < 0 become 0.

Also nit: we are also ignoring 0's that get passed through although we will read this as 0 if it is missing.

Just wondering what the intent here is. I think it may be better to have a defined missing value? Not sure

Yep, part of this thread #7770 (comment)

RussellSpitzer

Is it important that we store the manifest list size? Won't the encryption be enough to prove the file is the right one?

ggershinsky · 2024-04-01T08:59:02Z

Yep, this is due to https://github.com/apache/iceberg/blob/main/format/gcm-stream-spec.md#file-length . There are options for table modification attacks if this field is not (safely) stored.

RussellSpitzer · 2024-04-08T17:16:47Z

core/src/main/java/org/apache/iceberg/BaseSnapshot.java

    this.v1ManifestLocations = v1ManifestLocations;
+    this.manifestListKeyMetadata = null;


Same as comment above (group manifest vars)

rdblue · 2024-04-11T15:51:22Z

api/src/main/java/org/apache/iceberg/Snapshot.java

+   * In encrypted tables, return the size of this snapshot's manifest list file. Must be a verified
+   * value, taken from a trusted source. In unencrypted tables, can return 0.
+   */
+  default long manifestListSize() {


Rather than adding new methods for each piece of new information we want to pass, what about adding a ManifestList object that contains the location, size, and key metadata?

anuragmantri · 2024-06-11T11:41:22Z

api/src/main/java/org/apache/iceberg/Snapshot.java

@@ -162,6 +162,16 @@ default Iterable<DeleteFile> removedDeleteFiles(FileIO io) {
   */
  String manifestListLocation();


Is String manifestListLocation() same as what is stored in ManifestListFile? If so, do we still need it as a separate field?

@rdblue @RussellSpitzer what do you think?

There are dozens of calls to this method today (tests etc), so we'll likely need to keep it for a while.

Yeah, I saw that. We can consolidate in a separate PR.

Yeah, I agree. We need to keep this because it is part of the public API and used in many cases.

anuragmantri · 2024-06-11T11:49:31Z

core/src/main/java/org/apache/iceberg/SnapshotParser.java

+          ByteBuffer.wrap(
+              Base64.getDecoder().decode(snapshot.manifestListFile().wrappedKeyMetadata()));
+
+      NativeEncryptionKeyMetadata wrappedKeyMetadata =


Is this unused?

Good catch. I'll remove these lines.

anuragmantri · 2024-06-11T11:53:03Z

core/src/main/java/org/apache/iceberg/SnapshotParser.java

+      ByteBuffer manifestListKeyMetadata = null;
+      ByteBuffer wrappedManifestListKeyMetadata = null;
+      String wrappedKeyEncryptionKey = null;


Nit: Declare these inside if (node.has(MANIFEST_LIST_KEY_METADATA)) ?

They are used later, in the BaseSnapshot constructor. We can use a different constructor (if no manifest list encryption), but the code won't be more compact.

ggershinsky · 2024-06-11T07:14:43Z

api/src/main/java/org/apache/iceberg/ManifestListFile.java

+    return size;
+  }
+
+  public String wrappedKeyEncryptionKey() {


Add javadoc: "Manifest list keys are encrypted with a table "key encryption key". This function returns a KMS wrap of the key encryption key.

ggershinsky · 2024-08-14T12:19:51Z

core/src/main/java/org/apache/iceberg/BaseSnapshot.java

@@ -143,7 +172,13 @@ private void cacheManifests(FileIO fileIO) {

    if (allManifests == null) {
      // if manifests isn't set, then the snapshotFile is set and should be read to get the list
-      this.allManifests = ManifestLists.read(fileIO.newInputFile(manifestListLocation));
+      InputFile manifestListInputFile = fileIO.newInputFile(manifestListFile.location());


I'll move this under if else (called only for unencrypted)

api/src/main/java/org/apache/iceberg/ManifestListFile.java

ggershinsky · 2024-09-09T06:13:07Z

@rdblue Thanks for the patch, I've merged it into this PR. We still need to sync on caching the unwrapped keys, I've added a commit that implements one way of doing this, will appreciate your review.

rdblue · 2024-09-10T21:50:06Z

core/src/main/java/org/apache/iceberg/encryption/WrappedEncryptionKey.java

+    return keyBytes;
+  }
+
+  public void setUnwrappedKey(ByteBuffer key) {


This should not contain a key. It's fine to cache, but cache in unwrapKey instead.

update snapshot producer update the patch clean up fix for previous patch address review comments move key wrapping to metadata encryption encrypt manifest list key metadata new aad util null key needs no encryption comment; clearer method/var names use key encryption key for manifest list keys add encryption util changes update EncryptionTestHelpers handle api change remove unused lines revert revapi.yml KEK cache unitest update rename var address review comments fix timeout default change writer kek timeout default Updates from review. cache unwrapped keys

Co-authored-by: Ryan Blue <[email protected]>

ggershinsky · 2024-09-12T10:11:22Z

@rdblue thanks for the caffeine-based cache for unwrapped keys, I've applied the patch.

ggershinsky · 2024-09-12T11:55:30Z

Regarding the caching limit - 1 minute might be too short. If a reader (or writer) refreshes a table each few minutes, there could be many KMS unwrap calls. In the new commit, I've changed this to a combined limit of 1000 entries or 1 day. Each entry is comprised of a "wrapped key" (could be up to a few hundred bytes, depending on KMS), and "unwrapped key" (a few dozen bytes), so the 1000 entries / max size is something like 0.5MB.
@rdblue what do you think?

EDT: actually, we might be able to make the cache ~x10 smaller, if we use the "key id" as the cache key. Its size is a few dozen bytes.

rdblue · 2024-09-16T17:48:39Z

.palantir/revapi.yml

+      new: "method java.lang.Long org.apache.iceberg.encryption.NativeEncryptionKeyMetadata::fileLength()"
+      justification: "New method in interface"
+    - code: "java.method.addedToInterface"
+      new: "method org.apache.iceberg.encryption.NativeEncryptionKeyMetadata org.apache.iceberg.encryption.NativeEncryptionKeyMetadata::copyWithLength(long)"


Rather than revapi exceptions, we should add default implementations.

rdblue · 2024-09-16T17:49:28Z

core/src/main/java/org/apache/iceberg/BaseManifestListFile.java

+  }
+
+  @Override
+  public ByteBuffer decryptKeyMetadata(EncryptionManager em) {


When was this added? I don't think this should be here because it just calls a method on EncryptionManager.

It was added in PR11 (these lines: interface, implementation ). I think the main is reason is, the class EncryptingFileIO is in the api module, so it cannot access StandardEncryptionManager and EncryptionUtil classes.

rdblue · 2024-09-16T17:50:29Z

core/src/main/java/org/apache/iceberg/CatalogProperties.java

@@ -160,4 +160,10 @@ private CatalogProperties() {}

  public static final String ENCRYPTION_KMS_TYPE = "encryption.kms-type";
  public static final String ENCRYPTION_KMS_IMPL = "encryption.kms-impl";
+  public static final String WRITER_KEK_TIMEOUT_SEC = "encryption.kek-timeout-sec";


Avoid using kek because it is not clear. Is there a good term for the key you're referring to here?

Since we have a reasonably well defined value of this parameter in the NIST spec, maybe we don't need to make it configurable in an initial version of table encryption. This parameter is also not easy to explain. So I'll remove this from the CatalogProperties for now. If, for some reason, a requirement comes later to make this configurable, we can add this back, with a proper name and documentation.

Sounds great!

rdblue · 2024-09-16T17:51:09Z

core/src/main/java/org/apache/iceberg/ManifestListWriter.java

  }

  protected abstract ManifestFile prepare(ManifestFile manifest);

  protected abstract FileAppender<ManifestFile> newAppender(
-      OutputFile file, Map<String, String> meta);
+      OutputFile outputFile, Map<String, String> meta);


Looks like file was accidentally renamed to outputFile. Can you revert that change?

reverting it triggers the following build message:

> Task :iceberg-core:checkstyleMain [ant:checkstyle] [ERROR] /<path>/iceberg/core/src/main/java/org/apache/iceberg/ManifestListWriter.java:66:18: 'file' hides a field. [HiddenField] > Task :iceberg-core:checkstyleMain FAILED FAILURE: Build failed with an exception.

rdblue · 2024-09-16T17:57:53Z

core/src/main/java/org/apache/iceberg/encryption/NativeEncryptionKeyMetadata.java

@@ -27,4 +27,15 @@ public interface NativeEncryptionKeyMetadata extends EncryptionKeyMetadata {

  /** Additional authentication data as a {@link ByteBuffer} */
  ByteBuffer aadPrefix();
+
+  /** Encrypted file length */
+  Long fileLength();


From revapi changes, I think these need default implementations.

rdblue · 2024-09-16T18:00:04Z

core/src/main/java/org/apache/iceberg/encryption/WrappedEncryptionKey.java

+import java.io.Serializable;
+import java.nio.ByteBuffer;
+
+public class WrappedEncryptionKey implements Serializable {


Minor: needs Javadoc to explain what "Wrapped" means in this context.

rdblue · 2024-09-16T18:16:18Z

@ggershinsky, I'm not too concerned with the size of the cache. I'm okay with 1 day, but that seems like a long time to have unencrypted key material in memory. I'll defer to your judgement here.

ggershinsky · 2024-09-17T06:59:37Z

Ok. We don't have clear guidelines on key caching in memory (key copies are spread all over the process memory - cache, plug-in KMS client code, an HTTP library in the KMS client code; the Java GC - so there are no guarantees for when an uncached key is deleted from memory, if ever, before the process stops). But I agree a day could be too long. I'll change it to 1 hour - might be a reasonable trade-off between performance requirements (KMS call overhead), and safety requirements (there is a chance a key will be deleted from the memory within a business day).

ggershinsky · 2024-09-27T13:33:18Z

Hi @rdblue , I've built the integration code with the latest version of this patch, works ok. Can we merge this PR?

RussellSpitzer · 2024-10-31T22:01:33Z

@rdblue Did you have any more comments on this one? I can do another pass as well but I'd like to finish this up as well soon

ggershinsky · 2024-11-07T13:41:02Z

@rdblue @RussellSpitzer All current comments should have been addressed in this thread and in the last commit. An additional review round is always welcome, I too would like to complete this feature.

github-actions bot added API core labels Jun 5, 2023

ggershinsky marked this pull request as draft June 5, 2023 06:10

ggershinsky force-pushed the manifest-list-encryption branch from aba7650 to 3162f9e Compare March 26, 2024 05:37

ggershinsky commented Mar 26, 2024

View reviewed changes

ggershinsky force-pushed the manifest-list-encryption branch from 3162f9e to a2d7b10 Compare March 27, 2024 06:38

ggershinsky marked this pull request as ready for review March 27, 2024 07:07

RussellSpitzer reviewed Mar 28, 2024

View reviewed changes

core/src/main/java/org/apache/iceberg/BaseSnapshot.java Outdated Show resolved Hide resolved

RussellSpitzer reviewed Mar 28, 2024

View reviewed changes

RussellSpitzer reviewed Mar 29, 2024

View reviewed changes

RussellSpitzer reviewed Apr 8, 2024

View reviewed changes

rdblue reviewed Apr 11, 2024

View reviewed changes

ggershinsky force-pushed the manifest-list-encryption branch from 6c7f5a1 to 5486f6f Compare May 26, 2024 06:41

ggershinsky closed this May 28, 2024

ggershinsky reopened this May 28, 2024

anuragmantri reviewed Jun 11, 2024

View reviewed changes

ggershinsky commented Jun 11, 2024

View reviewed changes

ggershinsky reopened this Aug 4, 2024

ggershinsky commented Aug 14, 2024

View reviewed changes

rdblue reviewed Sep 2, 2024

View reviewed changes

api/src/main/java/org/apache/iceberg/ManifestListFile.java Show resolved Hide resolved

rdblue mentioned this pull request Sep 2, 2024

Updates from review. ggershinsky/iceberg#11

Merged

rdblue reviewed Sep 10, 2024

View reviewed changes

ggershinsky force-pushed the manifest-list-encryption branch from 1ffa46c to 684e5e3 Compare September 12, 2024 05:56

ggershinsky force-pushed the manifest-list-encryption branch from 684e5e3 to e39e740 Compare September 12, 2024 05:59

ggershinsky and others added 3 commits September 12, 2024 10:03

compile and ci fixes

cc27c3f

spotless apply

0988f6f

re-apply unwrapped cache patch

6e7de37

Co-authored-by: Ryan Blue <[email protected]>

caching limit

1c03cd1

rdblue reviewed Sep 16, 2024

View reviewed changes

ggershinsky added 2 commits September 17, 2024 14:05

address review comments

e637adc

spotless apply

1fbdaa6

		this.v1ManifestLocations = v1ManifestLocations;
		this.manifestListKeyMetadata = null;

		@@ -162,6 +162,16 @@ default Iterable<DeleteFile> removedDeleteFiles(FileIO io) {
		*/
		String manifestListLocation();

Manifest list encryption #7770

Are you sure you want to change the base?

Manifest list encryption #7770

Conversation

ggershinsky commented Jun 5, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RussellSpitzer Mar 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RussellSpitzer left a comment

Choose a reason for hiding this comment

ggershinsky commented Apr 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ggershinsky commented Sep 9, 2024

Choose a reason for hiding this comment

ggershinsky commented Sep 12, 2024

ggershinsky commented Sep 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ggershinsky Sep 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rdblue commented Sep 16, 2024

ggershinsky commented Sep 17, 2024

ggershinsky commented Sep 27, 2024

RussellSpitzer commented Oct 31, 2024

ggershinsky commented Nov 7, 2024

RussellSpitzer Mar 28, 2024 •

edited

Loading

ggershinsky commented Sep 12, 2024 •

edited

Loading

ggershinsky Sep 17, 2024 •

edited

Loading