feat: Add datastore mode data transforms

[danieljbruce]

Summary:

This PR adds support for datastore mode data transforms. A new option is provided for the first argument in the save method and code is added that will build the entity proto so that the backend receives requests that will do Datastore Mode data transforms.

API change

Here is an example of a call that the user can now make with the save method. They can now provide transforms that will do data transforms for a commit operation:

const result = await datastore.save({
  key: key,
  data: {
    name: 'test',
    p1: 3,
    p2: 4,
    p3: 5,
    a1: [3, 4, 5],
  },
  transforms: [
    {
      property: 'p1',
      setToServerValue: true,
    },
    {
      property: 'p2',
      increment: 4,
    },
    {
      property: 'p3',
      maximum: 9,
    },
    {
      property: 'p2',
      minimum: 6,
    },
    {
      property: 'a1',
      appendMissingElements: [5, 6],
    },
    {
      property: 'a1',
      removeAllFromArray: [3],
    },
  ],
});

Notes about the API surface:

• maximum, minimum, increment, setToServerValue are encoded to google.datastore.v1.IValue so they can represent any value since google.datastore.v1.IValue is an encoded version of any value type. Likewise, appendMissingElements and removeAllFromArray can accept any array of any values since they are encoded into the google.datastore.v1.IArrayValue type.
• In the proto there is a one to many relationship between a mutation and property transforms. In the save function the user provides a list of entities which translates to a list of mutations for the request so we should have one list of transforms per entity.

Alternatives:

We could support an API where transforms are indexed by the property:

const result = await datastore.save({
  key: key,
  data: {
    name: 'test',
    p1: 3,
    p2: 4,
    p3: 5,
    a1: [3, 4, 5],
  },
  transforms: {
    p1: {
      setToServerValue: true, // Set transforms
    },
    p2: {
      increment: 4,
    }
  },
});

Pros:

• Less code

Cons:

• Doesn't match the structure of the proto
• Order can be important for transforms so this gives the user less control

Changes:

src/entity.ts: These are just interface changes. Technically they would affect users that currently supply a transform property, but I doubt any users currently use a transform property. We could remove this file and the PR would still be good.

src/index.ts: The save method is modified to attach propertyTransforms to the entity proto

src/utils/entity/buildPropertyTransforms.ts: This file is added to build the propertyTransforms from the user's transforms array.

system-test/datastore: A test is added that performs a save operation with transforms and ensures the encoded entity proto is correct, the transforms returned are correct and the backend data after the update is correct.

Next Steps:

We should add support for users that wish to do transforms with transactions.

Requests for the reviewer:

• Be sure to take a look at the API and ensure that it is the preferred choice
• Consider the suggestion of removing the src/entity.ts file.

[JU-2094]

LGTM !

[daniel-sanche]

do these have to be any?

[danieljbruce]

maximum/minimum/increment/appendMissingElements/removeAllFromArray proto values can technically accept any type so there really are tons of combinations. In this feature we use encodeValue function to take the user supplied value and encode it into the generic proto IValue format. The proto supports the generic IValue type so to match we allow users to supply any type to support all the different input possibilities.

[daniel-sanche]

What is this TODO here for?

[danieljbruce]

I think this TODO was just a note for me to mention this in the PR. The idea here is if the user provides a transforms property for the entities they pass in then the compiler will force this property to be a PropertyTransform[] which is technically a breaking change.

The alternative is that we could not change this interface, but instead cast the transforms property as soon as it is passed into the save function.

[daniel-sanche]

nit: this would benefit from some comments

[danieljbruce]

Added some comments.

[daniel-sanche]

Does this single test really cover the whole feature? (I dont see anything with setToServerValue: false for example. And aren't non-int values supported?)

[danieljbruce]

We can add tests to cover setToServerValue: false and minimum/maximum etc. for strings. There are tons of permutations/combinations and I don't think we can cover them all, but we can definitely cover these.

maximum/minimum/increment/appendMissingElements/removeAllFromArray proto values can technically accept any type so there really are tons of combinations. In this feature we use encodeValue function to take the user supplied value and encode it into the generic proto IValue format. The proto supports the generic IValue type so to match we allow users to supply any type to support all the different input possibilities.

[danieljbruce]

Ok. These edge cases have been added now

[daniel-sanche]

LGTM