[cabpk] Write "/etc/hosts" and "/etc/hostname"

[akutz]

/kind feature

Describe the solution you'd like
The bootstrap data should include the files /etc/hosts and /etc/hostname with the values derived from the host. This ensures these files reflect the correct values without requiring a reboot before the systemd-hostnamed service updates them.

[akutz]

Related to kubernetes-sigs/cluster-api-provider-vsphere#474

[detiber]

@akutz I'm not sure I agree that the bootstrap data should always include /etc/hosts and /etc/hostname. Shouldn't we rely on the set_hostname, update_hostname, and update_etc_hosts modules to handle this?

[vincepri]

In addition to what @detiber mentioned, this functionality should be offered as an opt-in behavior that only some providers should use. AWS for example already sets these values internally on boot and doesn't need further intervention.

[akutz]

Hi @detiber / @vincepri,

Some background on this:

• The kubelet in CAPV images was using the static host name instead of the transient (DHCP) host name
• This is because the return value from the system call gethostname() is not always in sync with the value in /etc/hostname
• This drift occurs when the /etc/hostname file has not yet been updated by systemd-hostnamed, a service that starts after cloud-init
• The result was sudo calls that take obscenely long and kubelet node resolution that failed due to the value of /etc/hostname and /etc/hosts not reflecting the current, transient host name

All this was a problem on CAPV because of the fact that the host name provided to a guest may be affected by a DHCP server's advertised domain, causing the static host name to change to the transient, fully-qualified host name. This would be a problem on any cloud-init system that uses DHCP and is subject to transient host names.

However

It turns out all of this may have been avoided in the first place if the bug resolved by kubernetes-sigs/cluster-api-provider-vsphere#476 had not existed. It turns out that the cloud-provider was never actually resolving VMs via their UUIDs, and so the node resolution wasn't occurring against vSphere. This was a really subtle issue, and quite frankly, I'm as equally shocked that the bug wasn't caught sooner as I am that the bug was caught at all.

@ykakarap is building a custom version of CAPV that reverts kubernetes-sigs/cluster-api-provider-vsphere#474. Since CAPV now includes the cloud-provider fix (kubernetes-sigs/cluster-api-provider-vsphere#476), we think the host name fix in kubernetes-sigs/cluster-api-provider-vsphere#474 is no longer going to be necessary since the kubelet will be able to resolve the node by it's UUID.

I will close this issue if @ykakarap 's testing reveals the issue no longer occurs.

[ykakarap]

Test results:

By using a CAPV image that reverts kubernetes-sigs/cluster-api-provider-vsphere#474 clusterctl is NOT able to create a management cluster on vSphere.
A VM is created on vSphere but looks like there is an issue with kubelet on that VM.

Relevant kubelet logs:

Unable to register node "sc-rdops-vm06-dhcpv6-16d0" with API server: nodes "sc-rdops-vm06-dhcpv6-16d0" is forbidden: node "sc-rdops-vm06-dhcpv6-16d0.eng.vmware.com" cannot modify node "sc-rdops-vm06-dhcpv6-16d0"

Relevant section of /var/run/cloud-init/instance-data.json:

"ds": {
  "meta_data": {
   "hostname": "sc-rdops-vm06-dhcpv6-16d0.eng.vmware.com",
   "instance-id": "management-cluster-controlplane-1",
   "local-hostname": "sc-rdops-vm06-dhcpv6-16d0.eng.vmware.com",
   "local-ipv4": "10.160.169.10",
    ...

Relevant section of /tmp/kubeadm.yaml:

nodeRegistration:
  criSocket: /var/run/containerd/containerd.sock
  kubeletExtraArgs:
    cloud-provider: vsphere
  name: 'sc-rdops-vm06-dhcpv6-16d0.eng.vmware.com'

Content of /var/lib/cloud/data/set-hostname:

{
 "fqdn": "sc-rdops-vm06-dhcpv6-16d0.eng.vmware.com",
 "hostname": "sc-rdops-vm06-dhcpv6-16d0"
}

Results of hostname and hostnamectl commands:

ubuntu@sc-rdops-vm06-dhcpv6-16d0:~$ hostname
sc-rdops-vm06-dhcpv6-16d0
ubuntu@sc-rdops-vm06-dhcpv6-16d0:~$ hostnamectl
   Static hostname: sc-rdops-vm06-dhcpv6-16d0
   Pretty hostname: localhost.local
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 19e6d06bff3744ce8fd7713ca9420d7f
           Boot ID: 3bc8c93b690b44858a7fa6e3e960134a
    Virtualization: vmware
  Operating System: Ubuntu 18.04.2 LTS
            Kernel: Linux 4.15.0-54-generic
      Architecture: x86-64

[detiber]

hmm, that output leads me to believe that the hostname is initially set to the fqdn (or possibly the ds.meta_data.local-hostname value) when the kubelet bootstrapping was initially started, and at some point later changes to the hostname defined in /var/lib/cloud/data/set-hostname.

@akutz @ykakarap are there possibly two processes trying to set the local hostname based on those different values?

[akutz]

Hi @detiber,

It's absolutely the case that ds.meta_data.local-hostname is modified once the network is online. Initially it would just be the host name sans domain, but once the networking is online, the network info is discovered again and then persisted back to instance-data.json to ensure the local IPv4 and IPv6 addresses are included.

[akutz]

What's interesting to me is that the instance ID is management-cluster-controlplane-1. That's also the machine name, and thus the host name. The DHCP server is completely overriding the entire host name with a new host FQDN instead of just appending an advertised domain. In other words:

The host name changes from

management-cluster-controlplane-1

to

sc-rdops-vm06-dhcpv6-16d0.eng.vmware.com

It was never just sc-rdops-vm06-dhcpv6-16d0.

[detiber]

@akutz based on the output of the kubelet logs, it looks like it is indeed changed to sc-rdops-vm06-dhcpv6-16d0, which is also the value returned from hostname and hostnamectl as indicated in the output above.

That appears to be happening after the initial change from management-cluster-controlplane-1 to sc-rdops-vm06-dhcpv6-16d0.eng.vmware.com, though.

[akutz]

Hi @detiber,

I'm sorry, I meant it wasn't ever just sc-rdops-vm06-dhcpv6-16d0 before knowing about the FQDN.

The way the host name commands work and are invoked in the cloud-init sources do show how the host name is set multiple ways in https://github.com/cloud-init/cloud-init/blob/2f3bb764626b9065f4102c7c0a67998a9c174444/cloudinit/distros/__init__.py#L218-L347.

What's happening is the hostname is set to sc-rdops-vm06-dhcpv6-16d0.eng.vmware.com, and then a call gethostname is used and just sc-rdops-vm06-dhcpv6-16d0 is returned because the call defaults to short instead of long.

[akutz]

I think what we actually need to do is update the vSphere cloud-init datasource here to ask cloud-init to update the hostname after DHCP is enabled to ensure the host files are updated before kubeadm is run. This ensures that calls to gethostname return the correct values because they will no longer be out of sync with the host files on disk.

[akutz]

Hmm, that or do not let this line overwrite the hostname that was provided via the metadata.

(which @ykakarap already told me we should do a long time ago)

[ykakarap]

Hmm, that or do not let this line overwrite the hostname that was provided via the metadata.

This will make sure that the hostname of the VM is always the machine name (like management-cluster-controlplane-1).
(FYI, I already tried this fix and it works)

Or, we can make sure to update the hostname properly before kubeadm runs as suggested here

update the vSphere cloud-init datasource here to ask cloud-init to update the hostname after DHCP is enabled

I think the choice boils down to if we want the machine's hostname to be the machine name or the FQDN.

I would vote for the hostname being set to the machine name as it would be consistent with the behavior in environments like VMC.

[akutz]

Hi @ykakarap,

I would vote for the hostname being set to the machine name

The problem with this, and correct me if I'm wrong @detiber, is when a DHCP server sets the host name option, the client (machine), unless configured not to, will use that name. This is what is happening in the Nimbus situation.

it would be consistent with the behavior in environments like VMC.

VMC's DHCP server simply doesn't set the host name option, so it's not a VMC specific issue. It's specific to DHCP.

This is once again a problem unique to the on-prem providers since the AWS instance data sets the host name to the actual name the host will have since AWS knows that in advance and includes it in the metadata for the AWS cloud-init datasource to persist as the instance data.

Whereas in a true DHCP environment, CAPV won't know the host name of the machine in advance.

[detiber]

I'm not sure how cloud provider integration works with CAPV, but in at least AWS and OpenStack we are limited in what the Node "Name" can be based on what is expected by the cloud provider integration to match with queries against the remote API.

[akutz]

I'm not sure how cloud provider integration works with CAPV, but in at least AWS and OpenStack we are limited in what the Node "Name" can be based on what is expected by the cloud provider integration to match with queries against the remote API.

That's not the issue here. It's that the DHCP server is overriding the host name.

[chuckha]

@akutz can we close this?

[chuckha]

Trying to figure this one out, but this sentence has me a bit stumped

The bootstrap data should include the files /etc/hosts and /etc/hostname with the values derived from the host.

It sounds like the files /etc/hosts and /etc/hostname need to be correctly populated with data only known by the host. If there is an external thing that can provide the data these files need, then perhaps the thing responsible for making the bootstrap data can get the necessary data from that external system.

However, if only the host knows what values need to be there, then either the host needs to exist before CABPK generates data and the user adds these files to the KubeadmConfigSpec or the user could inject a script to generate these files before kubeadm init runs.

not really sure where to put this, so i'm leaving it low priority in the current milestone. Please adjust if necessary, this is a best guess.

/milestone v0.3.0
/priority important-longterm

[akutz]

cc @yastij

We need to kick this out of the v1a3 milestone if this is no longer an issue. Yuvaraj had been previously working on this.

[yastij]

/assign

[joonas]

@yastij did you have any updates on this?

[vincepri]

@yastij Any updates?

Bumping to next for now.

/milestone Next
/priority awaiting-more-evidence

[vincepri]

Closing this due to lack of updates

/close

[k8s-ci-robot]

@vincepri: Closing this issue.

In response to this:

Closing this due to lack of updates

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.