How to fix "mmap too large"?

[scjane]

Problem description:
We have a self-developed application management platform based on Kubernetes, which has been running on three VMs for about six months. Our platform provides a GUI for easy operation and management. Then, it suddenly fails to log into the GUI this week. We haved checked some components. The etcd container on the VM that is the master machine of the Kubernetes cluster displays "cannot open database at /var/lib/etcd/manager/snap/db (mmap too large)" in its logs.

As show in the following figure, the command output shows that the db file in /var/lib/etcd/manager/snap is nearly 800 TB.

However, the disk space of this VM is only 50 GB.

Version info:
OS of the three VMs: 7.3.1611 (Core)
etcd version: 3.1.7
docker version: 1.12.6
Kubernetes version: v1.6.4

How to fix:
Has anyone encountered this problem or similar ones? Cloud anyone give me some clues for troubleshooting it? Thanks a lot.

[xiang90]

we have never seen this. probably you triggered a boltdb bug.

only one member in the cluster has a huge db file, right?

[scjane]

@xiang90 Yes, in fact, only the master node has etcd deployed in a container. The other two worker nodes don't run etcd.

[xiang90]

If you have no backup and do not run etcd in HA manner, then the best option for you is to rebuild the cluster from scratch at this point.

[xiang90]

closing due to low activity. also etcd now uses its forked bboltdb. this problem is probably already got fixed.