Impossible to upload cookbooks if non-standard port is used

[oleg-z]

My chef-server.rb:

nginx['non_ssl_port'] = 8081
nginx['ssl_port'] = 4000

After running chef-server-ctl reconfigure I can't upload cookbook:

INFO: Uploading /var/default/chef-data/cookbooks/squeeze64-1.13.0/yum/templates/default/main.erb (checksum hex = e5ab84fe45a83c038ff442722be03dbd) to https://127.0.0.1:4000/bookshelf/organization-55800a42d41ca4067b8d9cc3f9d1ab51/checksum-e5ab84fe45a83c038ff442722be03dbd?AWSAccessKeyId=e737796ac367dbe4a94c96ad3ed439d9a3099d17&Expires=1419376039&Signature=clYmIZ0ViZA3dXgosSXDTiH9LfA%3D
INFO: HTTP Request Returned 500 Internal Server Error: internal service error
ERROR: Server returned error for https://127.0.0.1:4000/organizations/default/sandboxes/9cc3f9d1ab51f248e5ce1ccd3913b6da, retrying 1/5 in 3s
INFO: HTTP Request Returned 500 Internal Server Error: internal service error
ERROR: Server returned error for https://127.0.0.1:4000/organizations/default/sandboxes/9cc3f9d1ab51f248e5ce1ccd3913b6da, retrying 2/5 in 8s                                              
INFO: HTTP Request Returned 500 Internal Server Error: internal service error                                                                                                                
ERROR: Server returned error for https://127.0.0.1:4000/organizations/default/sandboxes/9cc3f9d1ab51f248e5ce1ccd3913b6da, retrying 3/5 in 10s

Reason:
incorrect erchef template which assumes that default protocol port is used. Attempt to specify vip parameter with port (e.g. 1.1.1.1:4000) causes issue because normalize_host method parses specified string as IPv6

{chef_objects, [
                  {s3_access_key_id, "<%= node['private_chef']['bookshelf']['access_key_id'] %>"},
                  {s3_secret_key_id, "<%= node['private_chef']['bookshelf']['secret_access_key'] %>"},
                  {s3_url, "<%= node['private_chef']['nginx']['x_forwarded_proto'] %>://<%= @helper.vip_for_uri('bookshelf') %>"},

[stevendanna]

@oleg-z Thanks for reporting this. I've confirmed the behavior you are seeing and your root cause is spot on. Below is a slightly more verbose description of what is happening for someone not as familiar with how opscode-omnibus is all tied together.

With the following placed in private-chef.rb:

nginx['non_ssl_port'] = 8081
nginx['ssl_port'] = 4000

nginx will listen on port 4000 for HTTPS connections and not the default port of 443.

During cookbook uploads, the opscode-erchef service talks to bookshelf via the s3_url in its configuration file (/var/opt/opscode/opscode-erchef/etc/app.config). This configuration file is rendered via a template(opscode-omnibus/files/private-chef-cookbooks/private-chef/templates/default/oc_erchef.config.erb), a portion of which looks like:

{s3_url, "<%= node['private_chef']['nginx']['x_forwarded_proto'] %>://<%= @helper.vip_for_uri('bookshelf') %>"},

Thus, the rendered configuration file will have an s3_url like:

{s3_url, "https://private-chef.opscode.piab"},

Given this configuration, erchef will attempt to contact erchef on port 443, the default HTTPS port. Unfortunately, nothing is listening on 443, the request to bookshelf fails and erchef returns a 500 to the user.

An astute user may attempt to set bookshelf['vip'] in private-chef.rb to something like:

bookshelf['vip'] = 'private-chef.opscode.piab:4000'

Unfortunately, this does not work either, since our IPv6 handling code will mangle this into:

{s3_url, "https://[private-chef.opscode.piab:4000]"},

which also fails. If you manually edit /var/opt/opscode/opscode-erchef/etc/app.config to make the s3_url line as follows:

{s3_url, "https://private-chef.opscode.piab:4000"},

and restart opscode-erchef, cookbook uploads will be successful.

The code that interprets anything with an ":" as an IPv6 address can be found in opscode-omnibus/files/private-chef-cookbooks/private-chef/libraries/helper.rb:

  def self.normalize_host(host_part)
    # Make this simple: if ':' is detected at all, it is assumed
    # to be a valid ipv6 address. We don't do data validation at this
    # point, and ':' is only valid in an URL if it is quoted by brackets.
    if host_part =~ /:/
      "[#{host_part}]"
    else
      host_part
    end
  end

  def normalize_host(host_part)
    self.class.normalize_host(host_part)
  end

  def vip_for_uri(service)
    normalize_host(node['private_chef'][service]['vip'])
  end

[tokenrain]

The problem with this solution, as I am sure you already know, is that running chef-server-ctl reconfigure will wipe out the custom modification. Are there any plans to fix this. It seems to me that in reality Chef Server does not really support any other port than 443 in reality

[shuoli84]

I met this and it really hurts. This is a bug, right?
Either we should able to configure bookshelf port or bookshelf should return the redirect url with the same port as chef server.

[jeremiahsnapp]

I also hope this gets fixed soon. In the meantime you can also manually edit the template that creates the erchef config file so the modification will survive a chef-server-ctl reconfigure.

Look around line 163 of the file /opt/opscode/embedded/cookbooks/private-chef/templates/default/oc_erchef.config.erb.

For example, if you want to set s3_url to "https://private-chef.opscode.piab:4000"

then replace the following line:

{s3_url, "<%= node['private_chef']['nginx']['x_forwarded_proto'] %>://<%= @helper.vip_for_uri('bookshelf') %>"},

with this:

{s3_url, "https://private-chef.opscode.piab:4000"},

Then run chef-server-ctl reconfigure.

[neutralrockets]

This is not only a problem with uploading cookbooks. I have found the same sort of issue with trying to get the Chef 12 reporting working for a Chef server on a non-standard port.

For reporting, had to resort to manually editing pedant_config.rb.erb and oc_reporting.config.erb under /opt/opscode-reporting/embedded/cookbooks/opscode-reporting/templates/default/ and added

:<%= node['private_chef']['nginx']['ssl_port'] %>

so that the reconfiguration generates the port in the config files. After a reconfigure and restart, reporting seems to be working. Before this, the chef-client would take a while to run on the remote node because it would be getting "500 Internal server errors" and retrying to connect to the reporting server several times before failing.

[rottenbytes]

Note for people who would stumble on this : this bug also hits when you make nginx listen on a particular IP

bookshelf["vip"] = "a.b.c.d"

will allow to change the IP that erchef will use to contact bookshelf

[diegows]

Same problem here, I have to revert back non_ssl_port to 80 to upload cookbooks.

[jessehu]

I also met this problem in open source chef server 12.1.2.

In my /etc/opscode/chef-server.rb :
nginx['ssl_port'] = 9443
nginx['non_ssl_port'] = 9080

After add the ssl port number into s3_url in /var/opt/opscode/opscode-erchef/sys.config and run 'sudo chef-server-ctl restart', the cookbooks are uploaded:

              {s3_url, "https://hostname:9443"},

[liku]

+1 faced same issue.

[bluejaguar]

+1 , bumping as appears to be same issue we are hitting. chef-server-core-12.1.2-1.el5.x86_64.rpm
I had to hack /var/opt/opscode/opscode-erchef/sys.config to insert the port number into the end of {s3_url,"https://machinename"},
As this issue was raised back on Dec 23, 2014 can we please get this fixed before Dec 23, 2015?

[jstange]

Can corroborate here with chef-server-core-12.2.0-1.el6.x86_64.

[kkeane]

Still happening with chef-server-core-12.3.0-1.el6

[marcparadise]

This tests out locally via both manual cookbook upload tests and pedant run: #637

[wduncanfraser]

Still seeing this with Chef server 12.4.1.

As workaround after reading the above comments, changed line 220 in /opt/opscode/embedded/cookbooks/private-chef/templates/default/oc_erchef.config.erb from

{s3_url, "<%= node['private_chef']['nginx']['x_forwarded_proto'] %>://<%= @helper.vip_for_uri('bookshelf') %>"},

to

{s3_url, "<%= node['private_chef']['nginx']['x_forwarded_proto'] %>://<%= @helper.vip_for_uri('bookshelf') %>:<%= node['private_chef']['nginx']['ssl_port'] %>"},

Additionally, to configure reporting properly, I changed line 36 in /opt/opscode-reporting/embedded/cookbooks/opscode-reporting/templates/default/pedant_config.rb.erb from

chef_server "https://<%= node['private_chef']['lb']['api_fqdn'] %>"

to

chef_server "https://<%= node['private_chef']['lb']['api_fqdn'] %>:<%= node['private_chef']['nginx']['ssl_port'] %>"

and line 91 in /opt/opscode-reporting/embedded/cookbooks/opscode-reporting/templates/default/oc_reporting.config.erb from

{chef_service, [{root_url, "https://<%= node['private_chef']['lb']['vip'] %>"},

to

{chef_service, [{root_url, "https://<%= node['private_chef']['lb']['vip'] %>:<%= node['private_chef']['nginx']['ssl_port'] %>"},

[zbuttram]

Just a note for others that might end up here, I had this problem but the fixes weren't fixing the issue completely. If you end up there, make sure your chef-server can resolve its own FQDN as well, that was the extra roadblock in my case.

[marcparadise]

I believe #833 has resolved this oleg-z, would you be able to confirm?

[kkeane]

Is this pull request in chef-server-core 12.7.0? The problem still exists in that version.

I do notice the changes in oc_erchef.config.erb

I suspect the problem now is that the attribute default['private_chef']['bookshelf']['vip_port'] defaults to 443 and as far as I can tell is never set to match the custom nginx port.

[zooda56]

This is absolutely correct. The attribute default['private_chef']['bookshelf']['vip_port'] defaults to 443.
Defaults can be overwritten, so in order to keep chef on non-standard post, two attributes must be configured in /etc/opscode/chef-server.rb

nginx['ssl_port'] = 4000
bookshelf['vip_port'] = 4000

Everything runs fine and cookbooks can be uploaded.

[dixon1234]

Not able to login if set custom nginx ssl port number
nginx['ssl_port']=8444
bookshelf['vip_port'] = 8444

When i login in chef UI i get this error

500Smell something burning?Sorry we have a small fire in the kitchen.An unexpected error has occurred. Our staff are manning the fire extinguishers and have been alerted.

[ChadScott]

Just wasted hours on this. Can someone please add the port into the s3_url line in the template?

[yonatane]

I hope this helps with the original issue and others coming here. After trying everything mentioned here without success (while playing with chef-server installed in a VirtualBox), I was able to fix the issue with this /etc/opscode/chef-server.rb:

bookshelf['external_url'] = 'https://ubuntu-xenial:8443'
opscode_erchef['base_resource_url'] = 'https://ubuntu-xenial:8443'

erchef is the REST API server that knife sends to. When uploading a cookbook it sends erchef a command to create a sandbox, and erchef responds with uri's to call next. By default, these uri's get the host name from the request (not sure why it isn't aware of the custom port), but if opscode_erchef['base_resource_url'] is configured it will use that.

More configuration options can be found here: https://docs.chef.io/config_rb_server_optional_settings.html#opscode-erchef

I found the solution by adding the very verbose -VVV option when uploading cookbook:
knife cookbook upload learn_chef_apache2 -VVV
Clipped output:

TRACE: Initiating GET to https://ubuntu-xenial:8443/organizations/4thcoffee/cookbooks?num_versions=all
TRACE: Initiating POST to https://ubuntu-xenial:8443/organizations/4thcoffee/sandboxes
INFO: Uploading /Users/ye/dev/learn-chef/cookbooks/learn_chef_apache2/LICENSE (checksum hex = 1d1c9ae2289377a52e9dcec0a6d255b6) to https://ubuntu-xenial:8443/bookshelf/organization-2320edd03d16313960a528c34aa85b18/checksum-1d1c9ae228…
TRACE: Initiating PUT to https://ubuntu-xenial:8443/bookshelf/organization-2320edd03d16313960a528c34aa85b18/checksum-1d1c9...
TRACE: Committing sandbox
TRACE: Initiating PUT to https://ubuntu-xenial/organizations/4thcoffee/sandboxes/28c34aa85b183a81f302aed64e23b2f9
ERROR: Connection refused connecting to https://ubuntu-xenial/organizations/4thcoffee/sandboxes/28c34aa85b18c48bd69f00f9507252ed, retry 1/5

and then searching for the code that's responsible for the sanboxes route in knife and chef-server:

Knife cookbook uploader:
https://github.com/chef/chef/blob/afb519b0a35ca36d6762b300d55911a4dc39b1e1/lib/chef/cookbook_uploader.rb#L80

Response with uri's:

chef-server/src/oc_erchef/apps/oc_chef_wm/src/chef_wm_sandboxes.erl

Line 111 in 9369149

{<<"uri">>, oc_chef_wm_routes:route(sandbox, Req, [{id, Id}])},

Building the uri:

chef-server/src/oc_erchef/apps/oc_chef_wm/src/oc_chef_wm_routes.erl

Line 107 in 9369149

render_template(Template, Req, TemplateArgs);

render_template function:

chef-server/src/oc_erchef/apps/oc_chef_wm/src/oc_chef_wm_routes.erl

Line 34 in 9369149

render_template(Template, chef_wm_util:base_uri(Req), Args).

base_uri function which uses base_resource_url config:

chef-server/src/oc_erchef/apps/oc_chef_wm/src/chef_wm_util.erl

Line 58 in 9369149

case application:get_env(oc_chef_wm, base_resource_url) of

[dgige]

Upload and login work for me:

CHEF_VERSION="15.3.14"
CHEFDK_VERSION="4.3.13"
CHEF_SERVER_VERSION="13.0.17"
CHEF_MANAGE_VERSION="2.5.16"

/etc/opscode/chef-server.rb

nginx['non_ssl_port'] = 8080
nginx['ssl_port'] = 8443
nginx['url'] = 'https://chef-server-01.vb.local:8443/'
nginx['server_name'] = 'chef-server-01.vb.local'
bookshelf['vip_port'] = 8443
bookshelf['external_url'] = 'https://chef-server-01.vb.local:8443'
opscode_erchef['base_resource_url'] = 'https://chef-server-01.vb.local:8443'

then

chef-server-ctl reconfigure
chef-manage-ctl reconfigure

[pkazi]

In my case, it was related to fips compliance.
My server's kernel was enabled with FIPS mode, and because of that chef-server was running on FIPS mode.

$ sudo grep fips /etc/opscode/chef-server-running.json
    "fips_enabled": true,

$ sudo cat  /var/opt/opscode/opscode-erchef/sys.config | grep s3_url
        {s3_url, "http://127.0.0.1:4321"},

To make it work, added fips false parameter to /etc/opscode/chef-server.rb and ran reconfigure.

Ref - https://docs.chef.io/config_rb_server_optional_settings.html