fix: fix object metadata and bucket label change tracking to be fine-grained

[BenWhitehead]

Context

Historically, when calling storage#update(BlobInfo) or storage#update(BucketInfo) the whole info would be packed and sent to the backend as a PATCH request. The backend would then perform some server side diffing against what it already had and make modifications. Of note here, for Object#metadata and Bucket#labels the PATCH behavior meant that any value provided in the request was always treated as an append.

Code sample

Specifically, the following call sequence will actually append key2: value2 to the object metadata despite nothing in the method names communicating this.

try (Storage storage = StorageOptions.http().build().getService()) {
  BlobInfo info = BlobInfo.newBuilder("bucket", "object")
      .setMetadata(ImmutableMap.of("key1", "value1"))
      .build();
  Blob gen1 = storage.create(info);

  BlobInfo modified = gen1.toBuilder().setMetadata(ImmutableMap.of("key2", "value2")).build();
  Blob gen2 = storage.update(modified, BlobTargetOption.metagenerationMatch());

  // gen2.metadata = {"key1": "value1", "key2": "value2"}
}

Additionally, this now leaves the internal state of modified such that if you were to call modified.getMetadata() it would only contain the new key2: value2. However, after calling storage.update gen2.getMetadata() both keys are present.

This is confusing, and can be remedied now that we have the capability of tracking field level modification.

Another confusing thing about this: if you want to clear the metadata on an object, the intuitive .setMetadata(Collections.emptyMap()) does not work. Instead, you would need to do the following:

try (Storage storage = StorageOptions.http().build().getService()) {
  Blob gen1 = storage.get("bucket", "object");

  // Can't use ImmutableMap here, as we need null values
  HashMap<String, String> clearedKeys = new HashMap<>();
  for (String key : gen1.getMetadata().keySet()) {
    clearedKeys.put(key, null);
  }

  BlobInfo modified = gen1.toBuilder().setMetadata(clearedKeys).build();
  Blob gen2 = storage.update(modified, BlobTargetOption.metagenerationMatch());
}

The Change

Update the behavior of BlobInfo#setMetadata and BucketInfo#setLabels to always explicitly set to the specified value, whether adding, removing or updating a key.

Add new methods BlobInfo#addAllMetadata(Map) and BucketInfo#addAllLabels(Map) to allow performing an upsert without the need to first query and reconcile.

Clearing all object metadata

try (Storage storage = StorageOptions.http().build().getService()) {
  Blob gen1 = storage.get("bucket", "object");
  BlobInfo modified = gen1.toBuilder().setMetadata(Collections.emptyMap()).build();
  Blob gen2 = storage.update(modified, BlobTargetOption.metagenerationMatch());
}

Adding/modifying object metadata

try (Storage storage = StorageOptions.http().build().getService()) {
  Blob gen1 = storage.get("bucket", "object");
  BlobInfo modified = gen1.toBuilder().addAllMetadata(ImmutableMap.of("key1", "value2", "key3", "value3").build();
  Blob gen2 = storage.update(modified, BlobTargetOption.metagenerationMatch());
}

Removing an individual metadata key

try (Storage storage = StorageOptions.http().build().getService()) {
  Blob gen1 = storage.get("bucket", "object");
  HashMap<String, String> map = new HashMap<>();
  map.put("key1", null);
  BlobInfo modified = gen1.toBuilder().addAllMetadata(map).build();
  Blob gen2 = storage.update(modified, BlobTargetOption.metagenerationMatch());
}

Implementation Details

1. Field modification now tracks nested metadata/label keys
   • New UnifiedOpts.NestedNamedField has been added to track nested keys
   • UpdateMaskTest has been updated to expect nested field names
2. StorageImpl#update(BlobInfo) and StorageImpl#update(BucketInfo) have both been updated to take field modification into consideration.
3. New test class ITNestedUpdateMaskTest has been added to validate many permutations of modifying metadata/labels
4. Each of the metadata/label related methods have had @NonNull/@Nullable annotations to their signatures to communicate acceptable arguments, and to set expectations.
5. All Data.isNull/Data.nullOf has been moved away from Blob metadata and Bucket labels into ApiaryConversions as they only apply to Apiary usage not gRPC/Protobuf.
6. Several easymock test in StorageImplTest have been wholly deleted as they weren't testing behavior, only wiring mocks to dance to the current implementation.
7. Update BucketInfoTest deep equals assertions to use TestUtils.assertAll

Samples

Update samples to use new addAll* methods for their upsert use cases rather than set*.

[BenWhitehead]

I'm not tied to this name, would upsertMetadataWith be better?

[BenWhitehead]

I'm not tied to this name, would upsertLabelsWith be better?

[BenWhitehead]

This was forcing Data.nullOf into the BucketInfo internals leaking apiary specifics. Those apiary specific have been pushed down to ApiaryConversions leaving BucketInfo able to do java standard things.

[BenWhitehead]

Make sure all assertions run

[BenWhitehead]

Make sure all assertions run

[BenWhitehead]

Before adding this test class, only 1 case was being explicitly tested in the existing suite - Object add new key. Now we're testing permutations of add/update/remove for both objects and buckets.

[frankyn]

Hey @BenWhitehead thanks for the summary.

Changing the behavior of setMetadata is a behavior breaking change, this can impact anyone that upgrades based on minor and patch versions. I agree that this is confusing and could be seen as a bug but it has been in the library for years.

Are you planning on lumping this into the next major version release and just didn't call it out in the PR?

[burkedavison]

Intentional use of both javax.annotation.Nonnull and org.checkerframework.checker.nullness.qual.NonNull?

[BenWhitehead]

Closing in favor of maintaining the existing setMetadata behavior.