Node/client cleanup never succeeds

[ShibaBandit]

chef-client run succeeds, but running the knife node/client delete -y always fails with output:

amazon-instance: Chef Client finished, 0/0 resources updated in 6.707980715 seconds
==> amazon-instance: Cleaning up chef node...
amazon-instance: Response: node 'packer-build-api' not found
amazon-instance:
==> amazon-instance: Terminating the source AWS instance...
==> amazon-instance: Deleting temporary security group...
Build 'amazon-instance' errored: Error cleaning up chef node: exit status 100

Source AMI OS: ubuntu 12.04

packer.json:

{
  "description": "Build AMIs for API",
  "builders": [
    {
      "type": "amazon-instance",
      "ami_description": "Ubuntu64 API Node",
      "bundle_vol_command": "sudo /usr/local/ec2/bin/ec2-bundle-vol -k {{.KeyPath}} -c {{.CertPath}} -u {{.AccountId}} -r {{.Architecture}} -e {{.PrivatePath}}/* -d {{.Destination}} -p {{.Prefix}} --batch --no-filter --exclude /tmp,/mnt,/home/ubuntu/cert",
      "bundle_upload_command": "sudo /usr/local/ec2/bin/ec2-upload-bundle -b {{.BucketName}} -m {{.ManifestPath}} -a {{.AccessKey}} -s {{.SecretKey}} -d {{.BundleDirectory}} --batch --url {{.S3Endpoint}} --retry",

      "access_key": "<REDACTED>",
      "secret_key": "<REDACTED>",

      "source_ami": "<AMI>",
      "instance_type": "m1.small",
      "ssh_username": "ubuntu",
      "ami_virtualization_type": "paravirtual",

      "region": "us-west-2",
      "s3_bucket":"<BUCKET_NAME>",
      "ami_name": "packer api node {{timestamp}}",

      "vpc_id": "<REDACTED>",
      "subnet_id": "<REDACTED>",
      "associate_public_ip_address": true,
      "ssh_private_key_file": "oregon-staging-vpc.pem",

      "account_id": "<REDACTED>",
      "x509_cert_path": "cert.pem",
      "x509_key_path": "key.pem",
      "x509_upload_path": "/home/ubuntu/cert",

      "run_tags": {"Name": "packer build tmp api {{timestamp}}"},
      "tags": {"Name": "packer api node {{timestamp}}"}
    }
  ],
  "provisioners": [
    {
        "type": "shell",
        "remote_path": "/home/ubuntu/packer_prov.sh",
        "inline": ["sudo mkdir /tmp/packer-chef-client", "sudo chmod 777 /tmp/packer-chef-client",
                      "sudo chmod -R 777 /etc/chef", "mkdir -p /home/ubuntu/cert"]
    },
    {
      "type": "chef-client",
      "node_name": "packer-build-api",
      "server_url": "<SERVER_REDACTED>",
      "run_list": [],
      "validation_key_path":"validation.pem",
      "config_template": "chef_server_config.template"
    }
  ]
}

Is packer using the -c option for knife when it tries to clean up?

[matt-smith]

Same problem for us, also it appears as though the knife command is not using sudo (is this correct?)

[ShibaBandit]

I don't believe it is.

https://github.com/mitchellh/packer/blob/master/provisioner/chef-client/provisioner.go#L345
https://github.com/mitchellh/packer/blob/master/provisioner/chef-client/provisioner.go#L361

[mitchellh]

I think we need to mirror more of the knife output. Tagged as bug.

[scott2449]

The knife command runs on the same machine as packer. I finally got it working by adding ~/.chef/knife.rb with all the right config for the server. I am going to make sure my PR contains some new language for the docs. Even if knife commands were executed remotely sudo should not be required. Also see #1520

[mfischer-zd]

The current Chef-client provisioner implementation requires that a Chef user private key with admin privileges be accessible from the host running Packer (i.e., not the build target). But in fact, this is not required, and leads to unnecessary problems, especially in CI environments where sharing a user's private key may not be desired.

Instead, I'd propose that the provisioner run knife inside the build environment with the following arguments:

sudo knife node delete NODE_NAME -s SERVER_URL -k /etc/chef/client.pem -u NODE_NAME -y
sudo knife client delete NODE_NAME -s SERVER_URL -k /etc/chef/client.pem -u NODE_NAME -y

Notes:

1. The order is important (client must be deleted last)
2. The only key that's needed is the client key that's placed in /etc/chef/client.pem by the initial bootstrap process. This key has permissions to delete itself.

[mfischer-zd]

Here is what we're doing now, since the Chef-client provisioner is still not fully-baked:

  "provisioners": [
    {
      "type": "shell",
      "inline": [
        "set -e",
        "curl -L https://www.opscode.com/chef/install.sh | sudo bash -s -- -v {{user `chef_version`}}",
        "install -d -m 755 -o ubuntu /tmp/packer-chef-client",
        "sudo install -d -m 755 -o root -g root /etc/chef"
      ]
    },
    {
      "type": "file",
      "source": "/tmp/validation.pem",
      "destination": "/tmp/validation.pem"
    },
    {
      "type": "shell",
      "inline": [
        "set -e",
        "sudo chef-client --no-color -S {{user `chef_server`}} -K /tmp/validation.pem -N {{user `node_name`}} -r 'cookbook::recipe'",
        "sudo knife node delete {{user `node_name`}} -s {{user `chef_server`}} -k /etc/chef/client.pem -u {{user `node_name`}} -y",
        "sudo knife client delete {{user `node_name`}} -s {{user `chef_server`}} -k /etc/chef/client.pem -u {{user `node_name`}} -y"
      ]
    },
    ....

[dev-head]

I am running into this problem as well, so far I have not found a viable solution when using chef-client. I've searched high and low to no avail, so i am here as a humble user who desperately needs your help in having a solution in this provisioner.

Update:
I did some testing with the source and found that the clean up commands are being executed on your host OS and not the guest OS. I successfully tested passing these commands to the guest OS but need to do some more testing before I take another step.

I would bet that the current logic to use the host OS was a purposeful decision however there may be value in adding code to support both routines and expose a configuration option to the user.

[venkat566]

This worked for me.
{
"description": "Builds AMI with chef-client installed",
"variables": {
"aws_access_key": "aws key",
"aws_secret_key": "secret key",
"server_url": "chef-server URL",
"validation_client_name": "validator Name",
"validation_key_path": "path of validator.pem"
},
"builders": [{
"type": "amazon-ebs",
"access_key": "{{user aws_access_key}}",
"secret_key": "{{user aws_secret_key}}",
"region": "us-east-1",
"source_ami": "ami-de0d9eb7",
"instance_type": "t1.micro",
"ssh_username": "ubuntu",
"ami_name": "packer-example {{timestamp}}"
}],
"provisioners": [{
"type": "shell",
"inline": [
"sleep 30",
"sudo apt-get update",
"sudo apt-get install -y redis-server"
]
},
{
"type": "chef-client",
"server_url": "{{user server_url}}",
"node_name": "node",
"validation_client_name": "{{user validation_client_name}}",
"validation_key_path": "{{user validation_key_path}}"
}]
}

[adhodgson1]

+1 here for doing the cleanup inside the build environment. In my environment I have several Chef orgs with different parameters, and I don't want my default Knife config to be the same as the configuration I am using for building AWS AMIs.

The other issue (not sure if opened elsewhere), is this cleanup fails on Windows machines running Packer, it assumes a Linux host with the Sudo command, which always fails on Windows:

==> amazon-ebs: Cleaning up chef node...
amazon-ebs: 'sudo' is not recognized as an internal or external command,
amazon-ebs: operable program or batch file.

Thanks.
Andrew.