etdc-manager overwriting existing /etc/hosts entries

[wskulley]

What steps did you take and what happened:
Running etcd-manager via kops in AWS on kope.io/k8s-1.11-debian-stretch-amd64-hvm-ebs-2018-08-17. We've observed the issue below in the following configurations (which is not intended as an exhaustive list of affected configurations, just the configurations we've tried):

• kops-1.11.0 , kopeio/etcd-manager:1.0.20181001, kubernetes 1.11.6, 3 masters
• kops-1.11.1, kopeio/etcd-manager:3.0.20190224, kubernetes 1.11.6, 3 masters

Kops with etcd-manager enabled appears to by default start two instances of etcd-manager on each master, one for "main" and one for events.

The master images have manage_etc_hosts set which means at boot time a handful of lines are placed into /etc/hosts, i.e.:

# Your system has configured 'manage_etc_hosts' as True.
# As a result, if you wish for changes to this file to persist
# then you will need to either 
# a.) make changes to the master file in /etc/cloud/templates/hosts.tmpl
# b.) change or remove the value of 'manage_etc_hosts' in
#     /etc/cloud/cloud.cfg or cloud-config from user-data
#
127.0.1.1 your-ec2-fqdn your-ec2-shortname
127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

Each instance of etcd-manager (main/events) writes records about the etcd-manager cluster into /etc/hosts, apparently every 10 seconds:

# Begin host entries managed by etcd-manager[etcd-events] - do not edit
your-master1-ip	your-master1-name
your-master2-ip	your-master2-name
your-master3-ip	your-master3-name
# End host entries managed by etcd-manager[etcd-events]

# Begin host entries managed by etcd-manager[etcd] - do not edit
your-master1-ip	your-master1-name
your-master2-ip	your-master2-name
your-master3-ip	your-master3-name
# End host entries managed by etcd-manager[etcd]

At some indeterminate time after boot (hours or days), we are seeing the manage_etc_hosts entries disappear from /etc/hosts, including localhost entries, leaving only the etcd-manager entries. Per auditd logging no other processes are writing to this file, so etcd-manager appears to be the cause of the disappearing entries.

What did you expect to happen:
Existing entries in /etc/hosts to remain undisturbed.

Anything else you would like to add:
A reboot of the node will (temporarily) restore the records, and the entries can of course be (temporarily) re-added by hand.

[canadiannomad]

We have encountered this problem as well with kops 1.12.2 and 1.12.3 etcd-manager:3.0.20190801

I believe it has to do with how https://github.com/kopeio/etcd-manager/blob/master/pkg/hosts/hosts.go does its pseudoAtomicWrite.

The series of events I think are happening are: (remember main and events pods are on same server, started at the same time)

• main pod reads /etc/hosts
• main pod writes /etc/hosts -- simultaneously events pod reads /etc/hosts and sees an empty file
• events pod writes /etc/hosts
• main pod detects concurrent write, and writes again
• events pod detects concurrent write, and writes again
• ... until it settles with a file that is missing any previously existing entries.

Note that this series of events would only happen if one pod reads just as the other is writing.