Skip to content

Conversation

danieljbruce
Copy link
Contributor

@danieljbruce danieljbruce commented Mar 18, 2025

Summary:

This PR adds support for datastore mode data transforms. A new option is provided for the first argument in the save method and code is added that will build the entity proto so that the backend receives requests that will do Datastore Mode data transforms.

API change

Here is an example of a call that the user can now make with the save method. They can now provide transforms that will do data transforms for a commit operation:

const result = await datastore.save({
  key: key,
  data: {
    name: 'test',
    p1: 3,
    p2: 4,
    p3: 5,
    a1: [3, 4, 5],
  },
  transforms: [
    {
      property: 'p1',
      setToServerValue: true,
    },
    {
      property: 'p2',
      increment: 4,
    },
    {
      property: 'p3',
      maximum: 9,
    },
    {
      property: 'p2',
      minimum: 6,
    },
    {
      property: 'a1',
      appendMissingElements: [5, 6],
    },
    {
      property: 'a1',
      removeAllFromArray: [3],
    },
  ],
});

Notes about the API surface:

  • maximum, minimum, increment, setToServerValue are encoded to google.datastore.v1.IValue so they can represent any value since google.datastore.v1.IValue is an encoded version of any value type. Likewise, appendMissingElements and removeAllFromArray can accept any array of any values since they are encoded into the google.datastore.v1.IArrayValue type.
  • In the proto there is a one to many relationship between a mutation and property transforms. In the save function the user provides a list of entities which translates to a list of mutations for the request so we should have one list of transforms per entity.

Alternatives:

We could support an API where transforms are indexed by the property:

const result = await datastore.save({
  key: key,
  data: {
    name: 'test',
    p1: 3,
    p2: 4,
    p3: 5,
    a1: [3, 4, 5],
  },
  transforms: {
    p1: {
      setToServerValue: true, // Set transforms
    },
    p2: {
      increment: 4,
    }
  },
});

Pros:

  • Less code

Cons:

  • Doesn't match the structure of the proto
  • Order can be important for transforms so this gives the user less control

Changes:

src/entity.ts: These are just interface changes. Technically they would affect users that currently supply a transform property, but I doubt any users currently use a transform property. We could remove this file and the PR would still be good.

src/index.ts: The save method is modified to attach propertyTransforms to the entity proto

src/utils/entity/buildPropertyTransforms.ts: This file is added to build the propertyTransforms from the user's transforms array.

system-test/datastore: A test is added that performs a save operation with transforms and ensures the encoded entity proto is correct, the transforms returned are correct and the backend data after the update is correct.

Next Steps:

We should add support for users that wish to do transforms with transactions.

Requests for the reviewer:

  • Be sure to take a look at the API and ensure that it is the preferred choice
  • Consider the suggestion of removing the src/entity.ts file.

@product-auto-label product-auto-label bot added size: m Pull request size is medium. api: datastore Issues related to the googleapis/nodejs-datastore API. labels Mar 18, 2025
@product-auto-label product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Mar 19, 2025
@danieljbruce danieljbruce changed the title 404540305 datastore mode data transforms feat: Add datastore mode data transforms Mar 24, 2025
@danieljbruce danieljbruce marked this pull request as ready for review March 24, 2025 19:18
@danieljbruce danieljbruce requested review from a team as code owners March 24, 2025 19:18
@JU-2094
Copy link
Member

JU-2094 commented Mar 25, 2025

LGTM !

export type PropertyTransform = {
property: string;
setToServerValue: boolean;
increment: any;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do these have to be any?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maximum/minimum/increment/appendMissingElements/removeAllFromArray proto values can technically accept any type so there really are tons of combinations. In this feature we use encodeValue function to take the user supplied value and encode it into the generic proto IValue format. The proto supports the generic IValue type so to match we allow users to supply any type to support all the different input possibilities.

image

src/entity.ts Outdated

// eslint-disable-next-line @typescript-eslint/no-explicit-any
export type Entity = any;
// TODO: Call out this interface change

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this TODO here for?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this TODO was just a note for me to mention this in the PR. The idea here is if the user provides a transforms property for the entities they pass in then the compiler will force this property to be a PropertyTransform[] which is technically a breaking change.

The alternative is that we could not change this interface, but instead cast the transforms property as soon as it is passed into the save function.

import IValue = google.datastore.v1.IValue;
import ServerValue = google.datastore.v1.PropertyTransform.ServerValue;

export function buildPropertyTransforms(transforms: PropertyTransform[]) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this would benefit from some comments

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some comments.

assert.strictEqual(entity, undefined);
});
});
describe('Datastore mode data transforms', () => {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this single test really cover the whole feature? (I dont see anything with setToServerValue: false for example. And aren't non-int values supported?)

Copy link
Contributor Author

@danieljbruce danieljbruce May 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can add tests to cover setToServerValue: false and minimum/maximum etc. for strings. There are tons of permutations/combinations and I don't think we can cover them all, but we can definitely cover these.

maximum/minimum/increment/appendMissingElements/removeAllFromArray proto values can technically accept any type so there really are tons of combinations. In this feature we use encodeValue function to take the user supplied value and encode it into the generic proto IValue format. The proto supports the generic IValue type so to match we allow users to supply any type to support all the different input possibilities.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. These edge cases have been added now

@product-auto-label product-auto-label bot added size: xl Pull request size is extra large. and removed size: l Pull request size is large. labels May 30, 2025
@product-auto-label product-auto-label bot added size: l Pull request size is large. and removed size: xl Pull request size is extra large. labels May 30, 2025
@product-auto-label product-auto-label bot added size: xl Pull request size is extra large. and removed size: l Pull request size is large. labels Sep 11, 2025
Copy link

@daniel-sanche daniel-sanche left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@danieljbruce danieljbruce added automerge Merge the pull request once unit tests and other checks pass. kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Sep 15, 2025
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 15, 2025
@danieljbruce danieljbruce merged commit 8ab6209 into main Sep 15, 2025
19 of 20 checks passed
@danieljbruce danieljbruce deleted the 404540305-datastore-mode-data-transforms branch September 15, 2025 20:22
@gcf-merge-on-green gcf-merge-on-green bot removed the automerge Merge the pull request once unit tests and other checks pass. label Sep 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: datastore Issues related to the googleapis/nodejs-datastore API. size: xl Pull request size is extra large.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants