Certificate not removed from puppet master when destroying machine

[fduranti]

The unprovision task try to remove the machine from puppet using the vmname but if the server is joined to a domain or have a domain configured this task fail because the host is registered with fqdn. Trying to redeploy a machine will result in a failure.
Not sure if this is related or can be solved by #106

This is the destroy log and the machine is not purged because it search for hostname and not fqdn

fduranti@itrompmd15 ostest [fduranti_devel] $ vagrant destroy -f win2012r2pmd15.q8inttest.com
==> win2012r2pmd15.q8inttest.com: Running triggers before destroy ...
==> win2012r2pmd15.q8inttest.com: Running trigger...
==> win2012r2pmd15.q8inttest.com: Should Removing machine win2012r2pmd15 win2012r2pmd15.q8inttest.com from domain.
==> win2012r2pmd15.q8inttest.com: Running cleanup tasks for 'reload' provisioner...
==> win2012r2pmd15.q8inttest.com: Running cleanup tasks for 'pe_agent' provisioner...
==> pmaster.q8inttest.com: No certificate for win2012r2pmd15 on pmaster.q8inttest.com.
==> pmaster.q8inttest.com: Skipping purge of agent data.
==> win2012r2pmd15.q8inttest.com: Forcing shutdown of VM...
==> win2012r2pmd15.q8inttest.com: Destroying VM and associated drives...
==> win2012r2pmd15.q8inttest.com: [vagrant-hostmanager:guests] Updating hosts file on active guest virtual machines...

This is the cert list on puppet server, machine are configured with fqdn.

[root@pmaster ~]# puppet cert list -a
+ "pe-internal-mcollective-servers"        (SHA256) FB:B5:5B:D8:9E:77:80:B0:69:22:C2:47:88:56:DE:30:74:52:11:D7:51:9E:AA:9C:C1:5E:C7:68:50:56:B4:B6 **
+ "pe-internal-peadmin-mcollective-client" (SHA256) DF:CB:95:95:24:2E:CE:C0:DC:97:43:4A:FF:BA:A9:75:74:87:CB:A7:A8:4B:55:1C:B9:5C:7E:14:11:2D:3D:39 **
+ "pmaster.q8inttest.com"                  (SHA256) E3:71:16:4A:5F:41:14:01:72:DB:0A:1C:3C:F7:F7:3F:B3:AE:E4:05:4B:66:79:4B:AB:CC:B3:D2:93:7D:E7:5F (alt names: "DNS:pmaster.q8inttest.com", "DNS:puppet") **
+ "win2012r2pmd15.q8inttest.com"           (SHA256) 1F:4B:67:59:CC:82:2A:05:81:C3:43:24:B4:3B:78:E8:ED:A6:FB:BC:BB:98:9C:2E:CA:72:5F:24:76:26:F9:8C
+ "win2016pmd15.q8inttest.com"             (SHA256) 44:DD:B9:BE:F7:55:B7:D8:3D:15:00:06:9D:19:90:29:4A:24:8E:38:10:96:C6:C7:7D:EC:95:4C:46:35:2A:9A

[Sharpie]

If the master_vm config option is set on a pe_agent provisioner, then the cleanup method of that provisioner should run puppet node purge on the master:

https://github.com/oscar-stack/vagrant-pe_build#pe_agent-provisioner-settings

[fduranti]

the master_vm is configured and during the provision the node is correclty added and autosigned. the problem is during the destroy... the node is not removed because the plugin try to use the hostname but the agent is registered with the fqdn so puppet cannot do the purge

==> pmaster.q8inttest.com: No certificate for win2012r2pmd15 on pmaster.q8inttest.com.
==> pmaster.q8inttest.com: Skipping purge of agent data.

[Sharpie]

Ah, this is the issue:

==> pmaster.q8inttest.com: No certificate for win2012r2pmd15 on pmaster.q8inttest.com.
==> pmaster.q8inttest.com: Skipping purge of agent data.

The cleanup action tried to purge a node named win2012r2pmd15. However, the certname for the node is actually:

win2012r2pmd15.q8inttest.com

Puppet defaults to using the output of facter fqdn when assigning the certname. However, Vagrant has no access to that info as there are several cases where the machine is not running when the settings for provision or cleanup are locked in.

The cleanup routine for pe_agent will use the first of the following items that is set to a value when deciding which certname to purge from the master:

• config.vm.hostname
• machine.name (what you see in vagrant status)

In this case, vm.hostname likely has to be set. However, Windows does not allow FQDNs to be assigned as the hostname so it will be a bit odd. Haven't found a good workaround for that one that doesn't involve assuming the existence of a DNS server, so I usually just set it to the machine name for consistency.

[fduranti]

Yes vm.hostname is set and it's just the hotsname. machine.name is correct (it's the fqdn).
I'm using DNS on both linux and windows vagrant machine on my test domain as I need a domain (and dns) to really test everything.

vagrant status win2016pmd15.q8inttest.com
Connection to 127.0.0.1 closed.
Current machine states:

win2016pmd15.q8inttest.com running (virtualbox)

I'm not a vagrant expert, it seems something that could be solved by the #106 is this correct? Any idea if and when it will be merged/published as new version?

Do you think i can not set config.vm.hostname or this will cause other problems?

[Sharpie]

If vm.hostname is set than the likely explanation is that DNS is providing an answer for facter fqdn, so that is getting used instead. #106 would solve this issue, but it's a too narrowly scoped. Especially since both the Linux and Windows installers support setting arbitrary puppet.conf options during install these days.

At the moment I'm focused on getting existing functionality tested out against Vagrant 2.x, so I likely won't be looking to build anything new for the next .z release. Managing puppet.conf entries during install would definitely be at the top of my list for a .y release though.

[fduranti]

Tomorrow I’ll do some checks trying to nit define hostname in the vagrantfile, I’ll let you know how it goes. If you need to test anything particular let me know if I can help. the #106 would be nice when released :)

[fduranti]

Done some checks. I've removed the config.vm.hostname from the Vagrantfile...
I had to setup manually the hostname both on windows and on linux machine via a new script and on windows i now need 1 more reboot (one for host rename, one after putting the machine at domain) but it seems that now the certificate removal is working correctly.
It's not so good as it take more time to provision and sometimes this method is prone to error on windows (sometimes after host rename or domain join the winrm protocol/vagrant fail and exit with an error) so the #106 will really be appreciated when merged :)

fduranti@itrompmd15 ostest [fduranti_devel] $ vagrant destroy win2012r2pmd15.q8inttest.com
==> win2012r2pmd15.q8inttest.com: Running triggers before destroy ...
==> win2012r2pmd15.q8inttest.com: Running trigger...
==> win2012r2pmd15.q8inttest.com: Should Removing machine win2012r2pmd15 win2012r2pmd15.q8inttest.com from domain.
    win2012r2pmd15.q8inttest.com: Are you sure you want to destroy the 'win2012r2pmd15.q8inttest.com' VM? [y/N] y
==> win2012r2pmd15.q8inttest.com: Running cleanup tasks for 'reload' provisioner...
==> win2012r2pmd15.q8inttest.com: Running cleanup tasks for 'reload' provisioner...
==> win2012r2pmd15.q8inttest.com: Running cleanup tasks for 'pe_agent' provisioner...
==> pmaster.q8inttest.com: Purging agent data for win2012r2pmd15.q8inttest.com from pmaster.q8inttest.com.
    pmaster.q8inttest.com: Running: inline script
    pmaster.q8inttest.com: Notice: Revoked certificate with serial 10
    pmaster.q8inttest.com: Notice: Removing file Puppet::SSL::Certificate win2012r2pmd15.q8inttest.com at '/etc/puppetlabs/puppet/ssl/ca/signed/win2012r2pmd15.q8inttest.com.pem'
    pmaster.q8inttest.com: Node "win2012r2pmd15.q8inttest.com" was purged.
    pmaster.q8inttest.com: 
    pmaster.q8inttest.com: To ensure this node can not check into any additional compile masters, run puppet on all compile masters.
    pmaster.q8inttest.com: 
    pmaster.q8inttest.com: - If you plan to re-add a node to your Puppet infrastructure:
    pmaster.q8inttest.com:    1. Clear the agent certificate from the node.
    pmaster.q8inttest.com:       On *nix, run `rm -rf /etc/puppetlabs/puppet/ssl`.
    pmaster.q8inttest.com:       On Windows, delete the `$confdir\ssl` directory.
    pmaster.q8inttest.com:    2. On the agent node, run Puppet.
==> win2012r2pmd15.q8inttest.com: Forcing shutdown of VM...
==> win2012r2pmd15.q8inttest.com: Destroying VM and associated drives...
==> win2012r2pmd15.q8inttest.com: [vagrant-hostmanager:guests] Updating hosts file on active guest virtual machines...