Cloud Datastore Emulator V1 - Generated Ids Are Too Small

[devng]

Our team at eGym GmbH is using Google Cloud Datastore and currently we are finally moving from v1beta3 to v1. However, we noticed some issues with the local testing environment. Previously we used gcd version gcd-v1beta3-1.0.1 for local testing, now we switched to cloud-datastore-emulator-1.2.1, because the java library datastore-v1-proto-client version 1.3.0 does not work with gcd-v1beta3-1.0.1. This is not a problem in itself, but we noticed some differences between gcd and thecloud-datastore-emulator. First, there is no more local admin console under _ah/admin, like before with gcd, but we saw that there is already a ticket for that (Data Viewer #148). However, the biggest issue for us right now is the fact that previously gcd generated entity ids similar to the one on the real datastore, e.g., 5629499534213120. Now the cloud-datastore-emulator does not do that and uses ids starting from 1. This is a problem since we are using the cloud datastore only for new data and old data is still stored on a legacy MySQL database, in order to speed thing up and not to query both systems for a result, we see if an ID is big enough and then only query the datastore, otherwise we query the data from our old MySQL. I know this is not ideal, but it works for us so far, and it works on production with the real datastore, however, our integration tests are now not working properly due to this bug. Can you please switch to the old gcd logic for generating ids on the new cloud-datastore-emulator. Thank you.

[elharo]

I'm afraid I can't see us doing this unless you can point to some documentation where datastore promises IDs in a certain range. I looked but I couldn't find anything like that.

[spiqueras]

Hi,

I know this is an old issue but just stumbled upon this.

I'm afraid I can't see us doing this unless you can point to some documentation where datastore promises IDs in a certain range. I looked but I couldn't find anything like that.

There are no promises as far as I know but the new Datastore autogenerated ID policy was announced here https://cloudplatform.googleblog.com/2013/05/update-on-datastore-auto-ids.html . From the announcement text:

These IDs are large, well-distributed integers, but are guaranteed to be small enough to be completely represented as 64-bit floats so they can be stored as Javascript numbers or JSON

As OP said, this behavior was already in place in the previous local emulators. I'd be great if this decision was reconsidered, as take advantage of the fact that Datastore never allocates ids in the lower range, and the current emulator behavior makes local testing harder.