Elkstack logstash-forwarder (AKA lumberjack) SSL auth

[martinb3]

@hhoover It didn’t generate my certificates properly, and after manually putting certificates in place (on the shipper and elk server) would still not work. I ended up writing a very small cookbook to upload a cert and key to the filesystem, which even then could not be verified by logstash-forwarder.

[mrjcleaver]

I might be experiencing something similar... Is this what you saw?

[2014-12-31T14:34:09-05:00] WARN: Could not find encrypted data bag item lumberjack/secrets
[2014-12-31T14:34:09-05:00] WARN: Could not find un-encrypted data bag item lumberjack/secrets
[2014-12-31T14:34:09-05:00] WARN: Generating a new lumberjack keypair and data bag item lumberjack/secrets
Recipe: elkstack::_secrets
  * openssl_x509[/var/chef/cache/lumberjack.crt] action create (up to date)
  * ruby_block[read generated keypair from disk] action run
    - execute the ruby block read generated keypair from disk

  ================================================================================
  Recipe Compile Error in /var/chef/cache/cookbooks/elkstack/recipes/single.rb
  ================================================================================

  Net::HTTPServerException
  ------------------------
  403 "Forbidden"

  Cookbook Trace:
  ---------------
    /var/chef/cache/cookbooks/elkstack/recipes/_secrets.rb:70:in `from_file'
    /var/chef/cache/cookbooks/elkstack/recipes/logstash.rb:43:in `from_file'
    /var/chef/cache/cookbooks/elkstack/recipes/single.rb:14:in `from_file'

  Relevant File Content:
  ----------------------
  /var/chef/cache/cookbooks/elkstack/recipes/_secrets.rb:

   63:      'key' => key_contents,
   64:      'certificate' => certificate_contents
   65:    }
   66:  
   67:    # unencrypted data bag if we just need a shared secret for ourselves
   68:    lumberjack_secrets_bag = Chef::DataBag.new
   69:    lumberjack_secrets_bag.name(lumberjack_data_bag)
   70>>   lumberjack_secrets_bag.save
   71:  
   72:    lumberjack_secrets = Chef::DataBagItem.new
   73:    lumberjack_secrets.data_bag(lumberjack_data_bag)
   74:    lumberjack_secrets.raw_data = secrets
   75:    lumberjack_secrets.save
   76:  end
   77:  
   78:  # now try to use the data bag
   79:  if !lumberjack_secrets.nil? && lumberjack_secrets['key'] && lumberjack_secrets['certificate']


  Running handlers:
[2014-12-31T14:34:09-05:00] ERROR: Running exception handlers
  Running handlers complete
[2014-12-31T14:34:09-05:00] ERROR: Exception handlers complete
[2014-12-31T14:34:09-05:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
  Chef Client failed. 2 resources updated in 12.689721249 seconds
[2014-12-31T14:34:09-05:00] ERROR: 403 "Forbidden"
[2014-12-31T14:34:09-05:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
root@elkstack:~# less /var/chef/cache/cookbooks/elkstack/recipes/_secrets.rb:70
/var/chef/cache/cookbooks/elkstack/recipes/_secrets.rb:70: No such file or directory
root@elkstack:~# less /var/chef/cache/cookbooks/elkstack/recipes/_secrets.rb
root@elkstack:~# sed -i '70p' /var/chef/cache/cookbooks/elkstack/recipes/_secrets.rb
root@elkstack:~# sed -n '70p' /var/chef/cache/cookbooks/elkstack/recipes/_secrets.rb
  lumberjack_secrets_bag.save
root@elkstack:~# sed -n '65,70p' /var/chef/cache/cookbooks/elkstack/recipes/_secrets.rb
  }

  # unencrypted data bag if we just need a shared secret for ourselves
  lumberjack_secrets_bag = Chef::DataBag.new
  lumberjack_secrets_bag.name(lumberjack_data_bag)
  lumberjack_secrets_bag.save
root@elkstack:~# sed -n '39,43p' /var/chef/cache/cookbooks/elkstack/recipes/logstash.rb
  input_syslog_port: 5959,
  chef_environment: node.chef_environment
}

include_recipe 'elkstack::_secrets'

[hhoover]

Looks about right, although that was a while ago. Due to frustration with the SSL problem and impending deliverable dates I went with Fluentd instead of logstash/lumberjack.

http://engineering.objectrocket.com/2014/11/07/logging-hybrid-cloud-fluentd-objectrocket-2/

[martinb3]

@mrjcleaver Your error is from a change in the default behavior of chef itself. Recently, chef stopped allowing clients to write data bags back to the server by default. I think @hhoover's issue of the cert and key not working even when he added them manually is something else.

Link to the chef announcement about the change:
https://www.chef.io/blog/2014/11/10/security-update-hosted-chef/

[mrjcleaver]

Okay, so my issue is continued at #75