GCP persistent disks

[pkaminski]

Is LMDB compatible with GCP's persistent disks? The docs warn against using LMDB on remote filesystems, and while persistent disks aren't that, they're not precisely local either. If the answer is yes, does that extend to regional persistent disks that automatically replicate writes to another zone, or is it possible for the replica to end up with a corrupted copy of the database?

[kriszyp]

Is LMDB compatible with GCP's persistent disks

I am not sure, I don't know how closely these disks stick to normal disk behavior.

automatically replicate writes to another zone, or is it possible for the replica to end up with a corrupted copy of the database?

This depends on if the replication is using snapshots of the disk system. There are certainly filesystems like btrfs that can provide snapshots of a file, and at any given point, the on-disk version of the database is stable, a server can crash, and the data is safe. However, file copying tools are typically not snapshot-based and instead are just incrementally pulling data from the incremental locations of the file at different points in time as the copying takes place. Since this is copying from the file at multiple points in times with multiple versions of the database state, this does not reliably lead to a safe copy and will frequently lead to database corruption.

I don't know if this is also intended to provide multiple LMDB processes access to the same database remotely. In order for two LMDB processes to safely access the same database, they must be able to mutually lock regions of the database (for exclusive write access).

[pkaminski]

Thanks, that helps. From what I've read about persistent disks so far I'm cautiously optimistic but obviously will need to try things out. Is there a particular stress-test workload pattern that would be most likely to reveal whether replication is atomic?

Understood on the multi-process limitation of LMDB. My intention here is to construct a low-RPO cross-zone disaster recovery option: the database runs in a single VM in zone A, the disk is synchronously replicated in zones A and B, and to fail over we just shut down the VM in zone A and bring up another in zone B.