nginx: [emerg] unknown directive "proxy_ssl_server_name" when using proxies on Heroku-20

[edmorley]

When using proxies on Heroku-20, nginx aborts with:
nginx: [emerg] unknown directive "proxy_ssl_server_name"

STR:

$ mkdir testapp-static && cd $_ && git init && h create --stack heroku-20
$ h buildpacks:add https://github.com/heroku/heroku-buildpack-static
$ cat > static.json <<EOF
{
  "proxies": {
    "/api/": {
      "origin": "https://example-app-that-does-not-exist.herokuapp.com/"
    }
  }
}
EOF
$ git add -A; git commit -m '.' && git push heroku main
...
$ curl -sSI https://infinite-forest-04844.herokuapp.com/api/ | head -n1
HTTP/1.1 503 Service Unavailable
$ h logs
...
2020-11-19T13:44:20.084652+00:00 heroku[web.1]: Starting process with command `bin/boot`
2020-11-19T13:44:22.800187+00:00 app[web.1]: Starting log redirection...
2020-11-19T13:44:22.800411+00:00 app[web.1]: Starting nginx...
2020-11-19T13:44:22.875465+00:00 app[web.1]: nginx: [emerg] unknown directive "proxy_ssl_server_name" in ./config/nginx.conf:77
2020-11-19T13:44:22.879244+00:00 app[web.1]: Process exited unexpectedly: nginx
2020-11-19T13:44:22.879338+00:00 app[web.1]: Going down, terminating child processes...

bin/nginx -V reports:

nginx version: nginx/1.19.0
built by gcc 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)
configure arguments: --add-module=/tmp/ngx_mruby.9Ikl --add-module=/tmp/ngx_mruby.9Ikl/dependence/ngx_devel_kit --prefix=/tmp/ngx_mruby.9Ikl/build/nginx --with-http_stub_status_module --with-stream --without-stream_access_module --with-cc-opt=-fno-common

And a h run bash followed by:

~ $ bin/boot &
~ $ cat /app/config/nginx.conf

Shows the generated config to be:

`nginx.conf` (click to expand)

daemon off;
worker_processes auto;

events {
  use epoll;
  accept_mutex on;
  worker_connections 2048;
}

http {
  gzip on;
  gzip_comp_level 6;
  gzip_min_length 512;
  gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
  gzip_vary on;
  gzip_proxied any;

  server_tokens off;


  access_log logs/access.log;



  error_log stderr error;


  include mime.types;
  default_type application/octet-stream;
  sendfile on;

  #Must read the body in 5 seconds.
  client_body_timeout 5;

  server {
    listen 36736 reuseport;
    charset UTF-8;
    port_in_redirect off;
    keepalive_timeout 5;
    root public_html/;


    resolver 172.16.0.23 8.8.8.8;




    mruby_post_read_handler /app/bin/config/lib/ngx_mruby/headers.rb cache;

    location / {
      mruby_set $fallback /app/bin/config/lib/ngx_mruby/routes_fallback.rb cache;

      try_files $uri $uri/ $fallback;

    }









  # need this b/c setting $fallback to =404 will try #{root}=404 instead of returning a 404
  location @404 {
    return 404;
  }

  # fallback proxy named match

    set $upstream_endpoint_0 https://hone-ember-todo-rails.herokuapp.com;
    location @/api/ {
      rewrite ^/api//?(.*)$ /$1 break;
      # can reuse variable set above
      proxy_pass $upstream_endpoint_0;
      proxy_ssl_server_name on;

      proxy_redirect http://hone-ember-todo-rails.herokuapp.com/ /api/;

      proxy_redirect https://hone-ember-todo-rails.herokuapp.com/ /api/;

    }


  # fallback redirects named match


  }
}

[edmorley]

The buildpack generates the proxy section of the nginx config here:

heroku-buildpack-static/scripts/config/templates/nginx.conf.erb

Lines 105 to 117 in 2ffb4f1

	# fallback proxy named match
	<% proxies.each do |location, hash| %>
	set $<%= hash['name'] %> <%= hash['host'] %>;
	location @<%= location %> {
	rewrite ^<%= location %>/?(.*)$ <%= hash['path'] %>/$1 break;
	# can reuse variable set above
	proxy_pass $<%= hash['name'] %>;
	proxy_ssl_server_name on;
	<% %w(http https).each do |scheme| %>
	proxy_redirect <%= hash["redirect_#{scheme}"] %> <%= location %>;
	<% end %>
	}
	<% end %>

Comparing this generated content between an app on heroku-18 and heroku-20 I see no differences.

The nginx docs say that the proxy_ssl_server_name directive should be available in all nginx versions since 1.7.0:
https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_server_name

My only thought is that perhaps SSL/TLS isn't enabled correctly in this build, and that nginx has an awful UX that doesn't say "this directive isn't supported when SSL isn't enabled".

[valentinoli]

I encountered this problem with heroku-20, had to temporarily downgrade to heroku-18. My static.json is pretty basic:

{
  "root": "dist",
  "clean_urls" :false,
  "https_only" :true,
  "routes": {
    "/**": "/app.html"
  },
  "proxies": {
    "/api/": {
      "origin": "https://someapi.com"
    }
  }
}

[edmorley]

Re-running the nginx compile locally (using make build-heroku-20) I see this in the output:

Configuration summary
  + using system PCRE library
  + OpenSSL library is not used
  + using system zlib library

This is surprising, since the build docs for ngx_ruby suggested openssl was a requirement for building, so I hadn't imagined it would be off by default.

Now trying a build locally that passes --with-http_ssl_module.

[edmorley]

I have a fix in #186 - after that's approved/merged I'll generate/upload new binaries to S3.

[edmorley]

The STR no longer reproduce for me now that I've regenerated the binaries in #186.

[valentinoli]

I redeployed my app with heroku-20 and now everything works as expected. Thank you for this quick fix!

[edmorley]

@valentinoli You're welcome! :-)