vSphere deploy does not connect NIC by default

[ryanl-ee]

Hi! I'm having an issue where deploying a VM in vSphere doesn't connect the NIC by default. Terraform will successfully clone, customize, and boot the VM, but editing the VM settings I can see the "Network adapter 1" does not have the "Connected" or "Connect at power on" boxes checked. The source VM does have "Connect at power on" checked.

• vSphere 6.0.0, 3018523
• Terraform 0.6.7 (I would have tried again using master but make test fails even in the Vagrant box)
• Source box (extracted and deployed using Workstation): https://atlas.hashicorp.com/boxcutter/boxes/ubuntu1404/versions/2.0.10
• Gist: https://gist.github.com/ryanl-ee/04a19dddacf5ecf70137 (this includes logs from after I manually connected the NIC and let the Terraform process continue - which failed, but I think that is a separate issue)

I've tried:

• Removing the NIC and adding a new NIC to the source VM
• Upgrading its virtual hardware version to 11
• Removing it from the inventory, creating a new VM, and attaching the old disk

Let me know if there is anything I can do to help test.

[chrislovecnm]

@ryan-ee can you test master? What do the customization logs in the VM say? How about the events on the VM? I am guessing that VM customization is failing, and vmtools and deploypkg have caused me fits in the past.

[talarczykco]

I'm having the same issue as @ryanl-ee. VM is created, powers up, has a NIC on the specified VLAN, but network adapter is disconnected. This happens with 0.6.7 as well as 0.6.8-dev as of yesterday afternoon. My main.tf follows the same basic format, specifying the network interface label and omitting the IP address. My Ubuntu template is configured for DHCP; as soon as I edit the VM manually in vSphere to connect the network, the DHCP kicks in and assigns an IP.

I think this is progress from a few weeks ago when the vSphere provider would crash or would conflict with network settings I already had on the template, but makes me wonder if this is working for anyone? It doesn't look like Ryan or I are using custom specifications - from the docs, I don't even see how one would be selected. Are working values documented for custom specifications and package names / versions for open-vm-tools and the like? Is there an chunk of code that can be added to explicitly connect the network adapter, perhaps becoming an optional argument defaulting to true?

[talarczykco]

Note: my VM template is a powered down Ubuntu image with DHCP enabled and 1 network interface with "Power on at connect" checked. Works with right-click deploy or convert from template. Open-vm-tools is installed.

Does this vSphere provider require 0 or 1 interfaces on the template? mkuzmin's was working for me. I know different third-party providers did this differently, but I don't know which one was used as the basis for the "official" version. Some asked you to remove the NIC entirely from the template.

[chrislovecnm]

Here are some instructions that I wrote for another project on how I built the image. Centos:

Create a template for the microservices cluster virtual machines. You will able to change CPU and RAM parameters while provisioning a virtual machine from template with Terraform. It's recommended to disable SELinux. Create user and add public RSA keys for SSH into the $HOME/.ssh/authorized_keys.
It is required to have VMware tools and deployPkg in the template, because we need to populate resulting .tfstate file with IP addresses of provisioned machines. With CentOS it is recommended to use open-vm-tools::

yum install open-vm-tools -y

It is recommended to setup yum to install open-vm-tools-deploypkg, as deployPkg is required to customize such things as hostname. Download the VMware packaging public keys from http://packages.vmware.com/tools/keys for yum.

Install the KEYS::

rpm –import VMWARE-PACKAGING-GPG-DSA-KEY.pub

rpm –import /tmp/VMWARE-PACKAGING-GPG-RSA-KEY.pub

mkdir /etc/yum.repos.d/vmware-tools.repo

echo -e "[vmware-tools]\nname = VMware Tools\nbaseurl = http://packages.vmware.com/packages/rhel7/x86_64/\nenabled = 1\ngpgcheck = 1 >> /etc/yum.repos.d/vmware-tools.repo

yum install open-vm-tools-deploypkg

sudo systemctl restart vmtoolsd

Heads down on other stuff. If someone wants to get this converted to both Centos and Ubuntu, and update the docs ... 😄

[chrislovecnm]

Guys .. please work off of master :)

Already asked:

What does the events on the vm say?
What does the customization logs say on the vm?

Also APPRECIATE you breaking it. Through this we will make the code better!!!!

[talarczykco]

I tested the 0.6.7 release via two different binary distributions (Mac Homebrew and direct Hashicorp download.) I tested 0.6.8 off master. Not sure what would suggest otherwise - as I understand it, "make dev" applies the "-dev" tag even when building with master, but correct me if I'm wrong. VM events showed nothing beyond a successful deployment, but the network was not connected at power on as is specified in the template. Custom specifications were not used; regardless, I'm not aware where such logs might be kept. open-vm-tools is installed, I don't know about deployPkg. How would that affect Terraform's communication with the hypervisor? I can't use this provider to do anything until it connects the NIC. Seems to me like there could be an explicit chunk of code that does so. Also, I'm still unclear whether this plugin expects 0 or 1 NIC configured. As mentioned, the three different plugins I used before all did things differently, and I'm not sure which of their behavior this plugin is expected to emulate (namely, does it create a new NIC or configure one that is already there.)

[chrislovecnm]

@mijit the provider on the master branch will handle both existing or no nics on the template. Pull requests have resolved issues with nics, and I am not certain when the pull requests were merged. I am testing on the master branch on git 'git checkout master'.

What does your tf file look like? Do you have deployPkg installed? DeployPkg should not matter, but I would like to get your ovf like mine.

[dariusjs]

I've also come across this where an ubuntu-14.04 vmware template fails to bring up the network using terraform 0.6.9. Bringing up an ubuntu-15.10 template works as expected.

edit:
A centos_7 image comes up correctly as well with the IP I am setting so only my ubuntu 14.04 is having issues. It has open-vm-tools installed and is updated with the latest packages. Don't get a link up if the network interface is present or missing in the template I get the same behaviour with both.

Below is a dmesg log of the vmxnet driver. The last log is me manually bringing up the network interface on sphere. Terraform then says the task was successful even though it did not setup the network and the incorrect IP address is configured as a DHCP address is getting set. I would love to find out what terraform is doing during this, it just waits for the network interface to come up.

root@ubuntu:# dmesg | grep -i vmxnet
[ 1.205648] VMware vmxnet3 virtual NIC driver - version 1.2.1.0-k-NAPI
[ 1.205919] vmxnet3 0000:03:00.0: # of Tx queues : 2, # of Rx queues : 2
[ 1.232205] vmxnet3 0000:03:00.0 eth0: NIC Link is Down
[ 8.306229] vmxnet3 0000:03:00.0 eth0: intr type 3, mode 0, 3 vectors allocated
[ 8.306563] vmxnet3 0000:03:00.0 eth0: NIC Link is Down
[ 163.471387] vmxnet3 0000:03:00.0 eth0: NIC Link is Up 10000 Mbps
root@ubuntu:#

[dariusjs]

On my side the issue is resolved with regards to ubuntu 14.04. I have not used ubuntu much in the past and after digging around I found some reports of people also struggling with this in general unrelated to terraform. It turns out that for these releases you need the open-vm-tools-debloypkg which allows interaction with network interfaces. open-vm-tools on its own doesn't give drivers towards NIC's.

Further info on how to install it is here http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2075048

[diamond29]

I followed your instructions @dariusjs and was able to get terraform to work. But it seems that the interface is only connected after the machine boots and the network configuration fails. You could make the timer shorter, but if there's a good way to make the interface connect before that maybe that would be desirable?

[Illydth]

I wanted to add a "Me Too" into this. I was having the same problem (Oracle Linux 7, VSphere/VMWare 5.1) with the NIC not being turned on.

After upgrade to the most recent vmware tools version (9.10) all works fine. The system comes up initially with the NIC off, and then gets turned on by the running scripts.

My main.tf file does have a Network_interface section defined with only the VLAN in it.

NOTE: The information above about adding open-vm-tools-deploypkg ONLY applies if you are using VMTools 9.4.x or before. Tools v. 9.10 merges the deploypkg stuff into the base open-vm-tools package, so for 9.10 there is no open-vm-tools-deploypkg.

[chrislovecnm]

Guys what is a status on this? I know we are waiting on #4283 to get merged, which will help some of this.

We probably should document a couple of the items on how to setup a image correctly.

Does anyone have packer instructions on setting up a vsphere vm to be used by terraform?

[talarczykco]

I can confirm Terraform 0.6.14 with open-vm-tools-deploypkg-9.4.x correctly connects the NIC and allows the VMware custom spec to set static IP, DNS settings, etc. Here's a gist if you're stuck on the older VM tools:

https://gist.github.com/mijit/8689b2b53391d903eea308566ed906f8

[chrislovecnm]

@mijit do you think it is a timing issue? have you experienced the problem before? More load on vSphere can slow down ip address allocation, and since we are not doing an IPWait ...

[talarczykco]

Not a timing issue as far as I was seeing. The VM and NIC came up, but the NIC was disconnected and never became connected. I am stuck on the older open-vm-tools without deploypkg, and after seeing Illydth's comment, I hunted down the instructions I needed to install it that I put in the gist. That seems to be what I was missing - just confirming it for inclusion in documentation or Packer template.

[chrislovecnm]

@xantheran the branch that you are testing helps some of this. Part of this bug is older vmtools, not usin deploypkg, and the other parts was just a timing problem. @tkak has a PR in, but I like where I put the hook sooner in the code. I have not put in a PR, because we already sorta have one in!

Frankly this has been a bitch to recreate, but here is my branch https://github.com/chrislovecnm/terraform/tree/vsphere-ip-wait-issue

Thanks!!! 👍

[chrislovecnm]

@ryanl-ee @mijit @Illydth and many others. @tkak's PR was merged. Any more word on this, or can we change this to documentation? I know there are some gems in here that need to be documented 😄

[chrislovecnm]

@ryanl-ee This may be fixed in #6377 .. can you compile the latest and test drive??

[chrislovecnm]

@stack72 @jen20 this should be addressed, can we close this issue.

@ryanl-ee please let us know if it is not fixed!!

[stack72]

Fixed by #6377

[chrislovecnm]

Thanks Paul

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.