From 2fc0df2532b5b8a6da442966feabf51dbabc5d7b Mon Sep 17 00:00:00 2001 From: kevprice83 Date: Wed, 2 Feb 2022 17:31:59 +0100 Subject: [PATCH 1/5] removes resolve of hostname and adds host/port to target server connect request adds comments linking to RFC and defines host and port values to remaining conditions adds CHANGELOG entry patches http proxy upstream requests adds comments for future refactor --- CHANGELOG.md | 42 ++++++++++++++++++------------ gateway/src/apicast/http_proxy.lua | 6 ++--- gateway/src/resty/http/proxy.lua | 15 ++++++----- 3 files changed, 38 insertions(+), 25 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a941f2dca..f8edb9f58 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,5 @@ # Change Log + All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](http://keepachangelog.com/) @@ -6,6 +7,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/). ## [3.11.0] 2021-09-03 +## [Unreleased] ### Fixed @@ -34,6 +36,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/). - Fixed warning messages [PR #1318](https://github.com/3scale/APIcast/pull/1318) [THREESCALE-7906](https://issues.redhat.com/browse/THREESCALE-7906) - Fixed dirty context [PR #1328](https://github.com/3scale/APIcast/pull/1328) [THREESCALE-8000](https://issues.redhat.com/browse/THREESCALE-8000) [THREESCALE-8007](https://issues.redhat.com/browse/THREESCALE-8007) - Fixed jwk alg confusion [PR #1329](https://github.com/3scale/APIcast/pull/1329) [THREESCALE-8249](https://issues.redhat.com/browse/THREESCALE-8249) +- Fixed issue with resolving target server hostnames to IP when using CONNECT method [PR #1323](https://github.com/3scale/APIcast/pull/1323) [THREESCALE-7967](https://issues.redhat.com/browse/THREESCALE-7967) ### Added @@ -54,7 +57,6 @@ and this project adheres to [Semantic Versioning](http://semver.org/). - Added on_failed policy [PR#1286](https://github.com/3scale/APIcast/pull/1286) [THREESCALE-6705](https://issues.redhat.com/browse/THREESCALE-6705) - Master branch containers builds are now latest tag on quay.io [PR#1289](https://github.com/3scale/APIcast/pull/1289) [THREESCALE-7251](https://issues.redhat.com/browse/THREESCALE-7251) - ## [3.10.0] 2021-01-04 Beta1 is stable and moved to final release. @@ -74,6 +76,7 @@ Beta1 is stable and moved to final release. ## [3.10.0-alpha1] 2020-10-13 ### Added + - Support Proxy Protocol [PR #1211](https://github.com/3scale/APIcast/pull/1211) [THREESCALE-5366](https://issues.redhat.com/browse/THREESCALE-5366) - Enable support to log credentials on logging policy [PR #1217](https://github.com/3scale/APIcast/pull/1217) [THREESCALE-5273](https://issues.redhat.com/browse/THREESCALE-5273) - Add a way to support more than 1000 services in a single instance [PR #1222](https://github.com/3scale/APIcast/pull/1222) [THREESCALE-5308](https://issues.redhat.com/browse/THREESCALE-5308) @@ -82,21 +85,21 @@ Beta1 is stable and moved to final release. - Add response/request content size limits [PR #1227](https://github.com/3scale/APIcast/pull/1227) [THREESCALE-5244](https://issues.redhat.com/browse/THREESCALE-5244) - Add HTTP codes policy [PR #1236](https://github.com/3scale/APIcast/pull/1236) [THREESCALE-6255](https://issues.redhat.com/browse/THREESCALE-6255) - - ### Fixed + - Fixed issues with allow caching mode and 3scale batcher [PR #1216](https://github.com/3scale/APIcast/pull/1216) [THREESCALE-5753](https://issues.redhat.com/browse/THREESCALE-5753) - Fixed issues when Auth Caching is disabled [PR #1225](https://github.com/3scale/APIcast/pull/1225) [THREESCALE-4464](https://issues.redhat.com/browse/THREESCALE-4464) - Fixed issues with service filter and OIDC [PR #1229](https://github.com/3scale/APIcast/pull/1229) [THREESCALE-6042](https://issues.redhat.com/browse/THREESCALE-6042) - Increased size of dictionaries used by the Batching policy to 20 MB. Users -with many services might have experienced issues with this policy because the -size of those dictionaries was not enough to store everything the policy needs -to function correctly. [PR #1231](https://github.com/3scale/APIcast/pull/1231) + with many services might have experienced issues with this policy because the + size of those dictionaries was not enough to store everything the policy needs + to function correctly. [PR #1231](https://github.com/3scale/APIcast/pull/1231) - Fixed issue with Camel service over HTTPs when Routing Policy [PR #1230](https://github.com/3scale/APIcast/pull/1230) [THREESCALE-5891](https://issues.redhat.com/browse/THREESCALE-5891) - Fixed doc issue on SERVICES_FILTER parameter [PR #1233](https://github.com/3scale/APIcast/pull/1233) [THREESCALE-5421](https://issues.redhat.com/browse/THREESCALE-5421) - Non-alphanumeric metric name in 3scale-batcher policy [PR #1234](https://github.com/3scale/APIcast/pull/1234) [THREESCALE-4913](https://issues.redhat.com/browse/THREESCALE-4913) ## [3.9.1] 2020-10-13 + - Fixed issues when using fully qualified DNS query [PR #1235](https://github.com/3scale/APIcast/pull/1235) [THREESCALE-4752](https://issues.redhat.com/browse/THREESCALE-4752) - Fixed issues with OIDC validation [PR #1239](https://github.com/3scale/APIcast/pull/1239) [THREESCALE-6313](https://issues.redhat.com/browse/THREESCALE-6313) - Fixed issues with Liquid body size [PR #1240](https://github.com/3scale/APIcast/pull/1240) [THREESCALE-6315](https://issues.redhat.com/browse/THREESCALE-6315) @@ -125,7 +128,6 @@ No issues found on beta1,so becames final release. - Fixed issues with path routing and query args [THREESCALE-5149](https://issues.redhat.com/browse/THREESCALE-5149) [PR #1190](https://github.com/3scale/APIcast/pull/1190) - Fixed issue with IPCheck policy when forwarder-for value contains port [THREESCALE-5258](https://issues.redhat.com/browse/THREESCALE-5258) [PR #1192](https://github.com/3scale/APIcast/pull/1192) - ### Added - Added upstream Mutual TLS policy [THREESCALE-672](https://issues.jboss.org/browse/THREESCALE-672) [PR #1182](https://github.com/3scale/APIcast/pull/1182) @@ -138,7 +140,6 @@ No issues found on beta1,so becames final release. - New content_caching Prometheus metric [THREESCALE-5439](https://issues.jboss.org/browse/THREESCALE-5439) [PR #1203](https://github.com/3scale/APIcast/pull/1203) - Added Camel policy [PR #1193](https://github.com/3scale/APIcast/pull/1193) [THREESCALE-4867](https://issues.jboss.org/browse/THREESCALE-4867) - ## [3.8.0] - 2020-03-24 `3.8.0-cr1` was considered final and became `3.8.0`. @@ -146,6 +147,7 @@ No issues found on beta1,so becames final release. ## [3.8.0-cr1] - 2020-03-07 ### Fixed + - Fixed naming issues in policies [THREESCALE-4150](https://issues.jboss.org/browse/THREESCALE-4150) [PR #1167](https://github.com/3scale/APIcast/pull/1167) - Fixed issues on invalid config in logging policy [THREESCALE-4605](https://issues.jboss.org/browse/THREESCALE-4605) [PR #1168](https://github.com/3scale/APIcast/pull/1168) - Fixed issues with routing policy and GRPC one [THREESCALE-4684](https://issues.jboss.org/browse/THREESCALE-4684) [PR #1177](https://github.com/3scale/APIcast/pull/1177) [PR #1179](https://github.com/3scale/APIcast/pull/1179) @@ -153,8 +155,8 @@ No issues found on beta1,so becames final release. ## [3.8.0-alpha2] - 2020-02-18 ### Fixed -- Check status is bigger than zero on caching policy [THREESCALE-4471](https://issues.jboss.org/browse/THREESCALE-4471) [PR #1163](https://github.com/3scale/APIcast/pull/1163) +- Check status is bigger than zero on caching policy [THREESCALE-4471](https://issues.jboss.org/browse/THREESCALE-4471) [PR #1163](https://github.com/3scale/APIcast/pull/1163) ## [3.8.0-alpha1] - 2020-01-31 @@ -167,7 +169,6 @@ No issues found on beta1,so becames final release. - Added Request_id on ngx.log function. [THREESCALE-3644](https://issues.jboss.org/browse/THREESCALE-3644) [PR #1156](https://github.com/3scale/APIcast/pull/1156) - Logging policy add the option to log JWT claims [THREESCALE-4326](https://issues.jboss.org/browse/THREESCALE-4326) [PR #1160](https://github.com/3scale/APIcast/pull/1160) - ### Fixed - When PATH routing was enabled the URL was not correctly escaped [THREESCALE-3468](https://issues.jboss.org/browse/THREESCALE-3468) [PR #1150](https://github.com/3scale/APIcast/pull/1150) @@ -176,7 +177,6 @@ No issues found on beta1,so becames final release. - Fix issues with non-alphanumeric variables in liquid [THREESCALE-3968](https://issues.jboss.org/browse/THREESCALE-3968) [PR #1158](https://github.com/3scale/APIcast/pull/1158) - Fix issues with double mapping rules [THREESCALE-3950](https://issues.jboss.org/browse/THREESCALE-3950) [PR #1159](https://github.com/3scale/APIcast/pull/1159) - ## [3.7.0] - 2019-11-27 `3.7.0-rc2` was considered final and became `3.7.0`. @@ -203,7 +203,6 @@ No issues found on beta1,so becames final release. - Fix issues with escaped characters in URI [THREESCALE-3468](https://issues.jboss.org/browse/THREESCALE-3468) [PR #1123](https://github.com/3scale/APIcast/pull/1123) - ## [3.7.0-beta1]- 2019-09-13 ### Added @@ -243,7 +242,6 @@ No issues found on beta1,so becames final release. - Extended variables in Liquid template operations [PR #1081](https://github.com/3scale/APIcast/pull/1081), [THREESCALE-2927](https://issues.jboss.org/browse/THREESCALE-2927) - ## [3.6.0-beta1] - 2019-06-18 ### Added @@ -374,11 +372,11 @@ Apart from the changes mentioned in this section, this version also includes the `3.3.0-cr2` was considered final and became `3.3.0`. - The configuration schema of the rate-limit policy has changed from `3.2.0` so -if you were using it, please adapt your configuration file accordingly. + if you were using it, please adapt your configuration file accordingly. - The Native OAuth 2.0 flow is deprecated. Please consider using the OIDC -integration instead. + integration instead. - The new conditional policy is considered experimental. The way conditions are -expressed might change in future releases. + expressed might change in future releases. ## [3.3.0-cr2] - 2018-09-25 @@ -670,6 +668,7 @@ expressed might change in future releases. - Live and ready endpoints now set correct Content-Type header in the response[PR #441](https://github.com/3scale/apicast/pull/441), [THREESCALE-377](https://issues.jboss.org/browse/THREESCALE-377) ## [3.1.0] - 2017-10-27 + - 3.1.0-rc2 was considered final and became 3.1.0. ## [3.1.0-rc2] - 2017-09-29 @@ -794,6 +793,7 @@ expressed might change in future releases. - Ability to Authenticate against API using RHSSO and OpenID Connect [PR #283](https://github.com/3scale/apicast/pull/283) ### Fixed + - `http_ng` client supports auth passsed in the url, and default client options if the request options are missing for methods with body (POST, PUT, etc.) [PR #310](https://github.com/3scale/apicast/pull/310) - Fixed lazy configuration loader to recover from failures [PR #313](https://github.com/3scale/apicast/pull/313) - Fixed undefined variable `p` in post\_action [PR #316](https://github.com/3scale/apicast/pull/316) @@ -818,6 +818,7 @@ expressed might change in future releases. ## [3.0.0-beta1] - 2017-03-03 ### Changed + - Lazy load DNS resolver to improve performance [PR #251](https://github.com/3scale/apicast/pull/251) - Execute queries to all defined nameservers in parallel [PR #260](https://github.com/3scale/apicast/pull/260) - `RESOLVER` ENV variable overrides all other nameservers detected from `/etc/resolv.conf` [PR #260](https://github.com/3scale/apicast/pull/260) @@ -849,9 +850,11 @@ expressed might change in future releases. ## [3.0.0-alpha2] - 2017-02-06 ### Added + - A way to override backend endpoint [PR #248](https://github.com/3scale/apicast/pull/248) ### Changed + - Cache all calls to `os.getenv` via custom module [PR #231](https://github.com/3scale/apicast/pull/231) - Bump s2i-openresty to 1.11.2.2-1 [PR #239](https://github.com/3scale/apicast/pull/239) - Use resty-resolver over nginx resolver for HTTP [PR #237](https://github.com/3scale/apicast/pull/237) @@ -859,24 +862,31 @@ expressed might change in future releases. - Internal change to reduce global state [PR #233](https://github.com/3scale/apicast/pull/233) ### Fixed + - [OAuth] Return correct state value back to client ### Removed + - Nginx resolver directive auto detection. Rely on internal DNS resolver [PR #237](https://github.com/3scale/apicast/pull/237) ## [3.0.0-alpha1] - 2017-01-16 + ### Added + - A CHANGELOG.md to track important changes - User-Agent header with APIcast version and system information [PR #214](https://github.com/3scale/apicast/pull/214) - Try to load configuration from V2 API [PR #193](https://github.com/3scale/apicast/pull/193) ### Changed + - Require openresty 1.11.2 [PR #194](https://github.com/3scale/apicast/pull/194) - moved development from `v2` branch to `master` [PR #209](https://github.com/3scale/apicast/pull/209) - `X-3scale-Debug` HTTP header now uses Service Token [PR #217](https://github.com/3scale/apicast/pull/217) ## [2.0.0] - 2016-11-29 + ### Changed + - Major rewrite using JSON configuration instead of code generation. [Unreleased]: https://github.com/3scale/apicast/compare/v3.10.0...HEAD diff --git a/gateway/src/apicast/http_proxy.lua b/gateway/src/apicast/http_proxy.lua index e4c36e821..6b5767f29 100644 --- a/gateway/src/apicast/http_proxy.lua +++ b/gateway/src/apicast/http_proxy.lua @@ -58,6 +58,7 @@ function _M.resolve(uri) return ip, port end +-- #TODO: This local function is no longer called as of PR#1323 and should be removed local function resolve(uri) local host = uri.host local port = uri.port @@ -70,12 +71,11 @@ local function resolve(uri) end local function absolute_url(uri) - local host, port = resolve(uri) return format('%s://%s:%s%s', uri.scheme, - host, - port, + uri.host, + uri.port, uri.path or '/' ) end diff --git a/gateway/src/resty/http/proxy.lua b/gateway/src/resty/http/proxy.lua index 4cc3ae583..1ed06218c 100644 --- a/gateway/src/resty/http/proxy.lua +++ b/gateway/src/resty/http/proxy.lua @@ -17,7 +17,7 @@ local function connect_direct(httpc, request) local uri = request.uri local host = uri.host local ip, port = httpc:resolve(host, nil, uri) - + -- #TODO: This logic may no longer be needed as of PR#1323 and should be reviewed as part of a refactor local options = { pool = format('%s:%s', host, port) } local ok, err = httpc:connect(ip, port or default_port(uri), options) @@ -72,8 +72,9 @@ local function _connect_proxy_https(httpc, request, host, port) end local function connect_proxy(httpc, request, skip_https_connect) + -- target server requires hostname not IP and DNS resolution is left to the proxy itself as specified in the RFC #7231 + -- https://httpwg.org/specs/rfc7231.html#CONNECT for the CONNECT method local uri = request.uri - local host, port = httpc:resolve(uri.host, uri.port, uri) local proxy_uri = request.proxy if proxy_uri.scheme ~= 'http' then @@ -94,15 +95,17 @@ local function connect_proxy(httpc, request, skip_https_connect) ngx.log(ngx.DEBUG, 'connection to ', proxy_uri.host, ':', proxy_uri.port, ' established', ', pool: ', options.pool, ' reused times: ', httpc:get_reused_times()) + ngx.log(ngx.DEBUG, 'targeting server ', uri.host, ':', uri.port) + if uri.scheme == 'http' then -- http proxy needs absolute URL as the request path - request.path = format('%s://%s:%s%s', uri.scheme, host, port, uri.path or '/') + request.path = format('%s://%s:%s%s', uri.scheme, uri.host, uri.port, uri.path or '/') return httpc elseif uri.scheme == 'https' and skip_https_connect then - request.path = format('%s://%s:%s%s', uri.scheme, host, port, request.path or '/') - return _connect_tls_direct(httpc, request, host, port) + request.path = format('%s://%s:%s%s', uri.scheme, uri.host, uri.port, request.path or '/') + return _connect_tls_direct(httpc, request, uri.host, uri.port) elseif uri.scheme == 'https' then - return _connect_proxy_https(httpc, request, host, port) + return _connect_proxy_https(httpc, request, uri.host, uri.port) else return nil, 'invalid scheme' From 1fa7dd8810df2685de36eb4b2c5ab617dd81413c Mon Sep 17 00:00:00 2001 From: kevprice83 Date: Mon, 18 Apr 2022 09:51:09 +0200 Subject: [PATCH 2/5] fixes integration tests 1-10 fixed integration tests from 10 to 20 Signed-off-by: Samuele Illuminati fixes integration tests & adds upstream http proxy patch fixes integration & unit tests --- gateway/src/apicast/http_proxy.lua | 11 +- gateway/src/resty/http/proxy.lua | 8 +- spec/resty/http/proxy_spec.lua | 46 +----- t/http-proxy.t | 230 ++++++++++++++++++----------- 4 files changed, 146 insertions(+), 149 deletions(-) diff --git a/gateway/src/apicast/http_proxy.lua b/gateway/src/apicast/http_proxy.lua index 6b5767f29..6439f9827 100644 --- a/gateway/src/apicast/http_proxy.lua +++ b/gateway/src/apicast/http_proxy.lua @@ -159,16 +159,9 @@ end function _M.request(upstream, proxy_uri) local uri = upstream.uri - if uri.scheme == 'http' then -- rewrite the request to use http_proxy - local err - local host = upstream:set_host_header() - upstream:use_host_header(host) - upstream.servers, err = resolve_servers(proxy_uri) - if err then - ngx.log(ngx.WARN, "HTTP proxy is set, but no servers have been resolved, err: ", err) - end - upstream.uri.path = absolute_url(uri) + if uri.scheme == 'http' then upstream:rewrite_request() + forward_https_request(proxy_uri, uri, upstream.skip_https_connect) return elseif uri.scheme == 'https' then upstream:rewrite_request() diff --git a/gateway/src/resty/http/proxy.lua b/gateway/src/resty/http/proxy.lua index 1ed06218c..a399d6c81 100644 --- a/gateway/src/resty/http/proxy.lua +++ b/gateway/src/resty/http/proxy.lua @@ -72,8 +72,6 @@ local function _connect_proxy_https(httpc, request, host, port) end local function connect_proxy(httpc, request, skip_https_connect) - -- target server requires hostname not IP and DNS resolution is left to the proxy itself as specified in the RFC #7231 - -- https://httpwg.org/specs/rfc7231.html#CONNECT for the CONNECT method local uri = request.uri local proxy_uri = request.proxy @@ -95,11 +93,9 @@ local function connect_proxy(httpc, request, skip_https_connect) ngx.log(ngx.DEBUG, 'connection to ', proxy_uri.host, ':', proxy_uri.port, ' established', ', pool: ', options.pool, ' reused times: ', httpc:get_reused_times()) - ngx.log(ngx.DEBUG, 'targeting server ', uri.host, ':', uri.port) - if uri.scheme == 'http' then -- http proxy needs absolute URL as the request path - request.path = format('%s://%s:%s%s', uri.scheme, uri.host, uri.port, uri.path or '/') + request.path = format('%s://%s:%s%s', uri.scheme, uri.host, uri.port, uri.path or request.path or '/') return httpc elseif uri.scheme == 'https' and skip_https_connect then request.path = format('%s://%s:%s%s', uri.scheme, uri.host, uri.port, request.path or '/') @@ -179,4 +175,4 @@ end _M.new = connect -return _M:reset() +return _M:reset() \ No newline at end of file diff --git a/spec/resty/http/proxy_spec.lua b/spec/resty/http/proxy_spec.lua index d76be8d04..fef47fdf0 100644 --- a/spec/resty/http/proxy_spec.lua +++ b/spec/resty/http/proxy_spec.lua @@ -50,55 +50,13 @@ describe('resty.http.proxy', function() it('connects to the #http_proxy', function() _M:reset({ http_proxy = 'http://127.0.0.1:1984' }) - local request = { url = 'http://127.0.0.1:1984/request', method = 'GET' } + local request = { url = 'http://upstream:8091/request', method = 'GET' } local proxy = assert(_M.new(request)) local res = assert(proxy:request(request)) assert.same(200, res.status) - assert.match('GET http://127.0.0.1:1984/request HTTP/1.1', res:read_body()) - end) - - -- Regression test. Ref: https://issues.jboss.org/browse/THREESCALE-2205 - context('when different subdomains resolve to the same IP', function() - local request_domain_1 = { url = 'http://test.example.com/', method = 'GET' } - local request_domain_2 = { url = 'http://prod.example.com/', method = 'GET' } - - before_each(function() - -- Make everything resolve to the same IP - local resty_resolver = require 'resty.resolver.http' - stub(resty_resolver, 'resolve', function() return "1.1.1.1", 80 end) - end) - - context('and it uses a http proxy', function() - before_each(function() - _M:reset({ http_proxy = 'http://127.0.0.1:1984' }) - end) - - it('does not reuse the connection', function() - local proxy = _M.new(request_domain_1) - proxy:request(request_domain_1):read_body() - proxy:set_keepalive() - - proxy = _M.new(request_domain_2) - assert.same(0, proxy:get_reused_times()) - end) - end) - - context('and it does not use an http proxy', function() - before_each(function() - _M:reset({}) - end) - - it('does not reuse the connection', function() - local proxy = _M.new(request_domain_1) - proxy:request(request_domain_1):read_body() - proxy:set_keepalive() - - proxy = _M.new(request_domain_2) - assert.same(0, proxy:get_reused_times()) - end) - end) + assert.match('GET http://upstream:8091/request HTTP/1.1', res:read_body()) end) end) end) diff --git a/t/http-proxy.t b/t/http-proxy.t index df293183b..657ed3ba0 100644 --- a/t/http-proxy.t +++ b/t/http-proxy.t @@ -15,7 +15,7 @@ $ENV{'LARGE_BODY'} = large_body(); require("policies.pl"); -repeat_each(3); +repeat_each(1); run_tests(); @@ -65,15 +65,17 @@ yay, api backend: test:$TEST_NGINX_SERVER_PORT -=== TEST 2: Downloading configuration uses http proxy + TLS +=== TEST 2: Downloading configuration uses http proxy --- env eval ( "http_proxy" => $ENV{TEST_NGINX_HTTP_PROXY}, - 'APICAST_CONFIGURATION' => "http://test:$ENV{TEST_NGINX_SERVER_PORT}", + 'APICAST_CONFIGURATION' => "http://test.lvh.me:$ENV{TEST_NGINX_SERVER_PORT}", 'APICAST_CONFIGURATION_LOADER' => 'lazy', 'THREESCALE_DEPLOYMENT_ENV' => 'production', ) --- upstream env + +server_name test.lvh.me; location = /admin/api/services.json { content_by_lua_block { ngx.say([[{ "services": [ { "service": { "id": 1337 } } ] }]]) @@ -94,8 +96,8 @@ content_by_lua_block { --- error_code: 200 --- error_log env -proxy request: GET http://127.0.0.1:$TEST_NGINX_SERVER_PORT/admin/api/services.json HTTP/1.1 -proxy request: GET http://127.0.0.1:$TEST_NGINX_SERVER_PORT/admin/api/services/1337/proxy/configs/production/latest.json HTTP/1.1 +proxy request: GET http://test.lvh.me:$TEST_NGINX_SERVER_PORT/admin/api/services.json HTTP/1.1 +proxy request: GET http://test.lvh.me:$TEST_NGINX_SERVER_PORT/admin/api/services/1337/proxy/configs/production/latest.json HTTP/1.1 --- no_error_log [error] @@ -105,11 +107,12 @@ proxy request: GET http://127.0.0.1:$TEST_NGINX_SERVER_PORT/admin/api/services/1 --- env random_port eval ( "https_proxy" => $ENV{TEST_NGINX_HTTPS_PROXY}, - 'APICAST_CONFIGURATION' => "https://test:$ENV{TEST_NGINX_RANDOM_PORT}", + 'APICAST_CONFIGURATION' => "https://test.lvh.me:$ENV{TEST_NGINX_RANDOM_PORT}", 'APICAST_CONFIGURATION_LOADER' => 'lazy', 'THREESCALE_DEPLOYMENT_ENV' => 'production', ) --- upstream env +server_name test.lvh.me; listen $TEST_NGINX_RANDOM_PORT ssl; ssl_certificate $TEST_NGINX_SERVER_ROOT/html/server.crt; @@ -134,13 +137,16 @@ content_by_lua_block { } --- error_code: 200 --- error_log env -proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT +proxy request: CONNECT test.lvh.me:$TEST_NGINX_RANDOM_PORT --- no_error_log [error] === TEST 4: 3scale backend connection uses proxy --- env eval -("http_proxy" => $ENV{TEST_NGINX_HTTP_PROXY}) +( + "http_proxy" => $ENV{TEST_NGINX_HTTP_PROXY}, + 'BACKEND_ENDPOINT_OVERRIDE' => "http://test_backend.lvh.me:$ENV{TEST_NGINX_SERVER_PORT}" +) --- configuration { "services": [ @@ -150,7 +156,7 @@ proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT "backend_authentication_type": "service_token", "backend_authentication_value": "token-value", "proxy": { - "api_backend": "http://test:$TEST_NGINX_SERVER_PORT/", + "api_backend": "http://test-upstream.lvh.me:$TEST_NGINX_SERVER_PORT/", "proxy_rules": [ { "pattern": "/", "http_method": "GET", "metric_system_name": "hits", "delta": 2 } ] @@ -159,6 +165,7 @@ proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT ] } --- backend +server_name test_backend.lvh.me; location /transactions/authrep.xml { content_by_lua_block { local expected = "service_token=token-value&service_id=42&usage%5Bhits%5D=2&user_key=value" @@ -166,16 +173,17 @@ proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT } } --- upstream +server_name test-upstream.lvh.me; location / { echo 'yay, api backend: $http_host'; } --- request GET /?user_key=value --- response_body_like -yay, api backend: test:.* +yay, api backend: test-upstream.lvh.me:.* --- error_code: 200 --- error_log env -proxy request: GET http://127.0.0.1:$TEST_NGINX_SERVER_PORT/transactions/authrep.xml? +proxy request: GET http://test_backend.lvh.me:$TEST_NGINX_SERVER_PORT/transactions/authrep.xml? --- no_error_log [error] @@ -185,7 +193,7 @@ proxy request: GET http://127.0.0.1:$TEST_NGINX_SERVER_PORT/transactions/authrep --- env random_port eval ( 'https_proxy' => $ENV{TEST_NGINX_HTTPS_PROXY}, - 'BACKEND_ENDPOINT_OVERRIDE' => "https://test_backend:$ENV{TEST_NGINX_RANDOM_PORT}" + 'BACKEND_ENDPOINT_OVERRIDE' => "https://test_backend.lvh.me:$ENV{TEST_NGINX_RANDOM_PORT}" ) --- configuration env { @@ -196,7 +204,7 @@ proxy request: GET http://127.0.0.1:$TEST_NGINX_SERVER_PORT/transactions/authrep "backend_authentication_type": "service_token", "backend_authentication_value": "token-value", "proxy": { - "api_backend": "http://test:$TEST_NGINX_SERVER_PORT/", + "api_backend": "http://test-upstream.lvh.me:$TEST_NGINX_SERVER_PORT/", "proxy_rules": [ { "pattern": "/", "http_method": "GET", "metric_system_name": "hits", "delta": 2 } ] @@ -205,6 +213,7 @@ proxy request: GET http://127.0.0.1:$TEST_NGINX_SERVER_PORT/transactions/authrep ] } --- backend env +server_name test_backend.lvh.me; listen $TEST_NGINX_RANDOM_PORT ssl; ssl_certificate $TEST_NGINX_SERVER_ROOT/html/server.crt; @@ -215,7 +224,7 @@ location /transactions/authrep.xml { assert = require('luassert') assert.equal('https', ngx.var.scheme) assert.equal('$TEST_NGINX_RANDOM_PORT', ngx.var.server_port) - assert.equal('test_backend', ngx.var.ssl_server_name) + assert.equal('test_backend.lvh.me', ngx.var.ssl_server_name) } content_by_lua_block { @@ -224,16 +233,18 @@ location /transactions/authrep.xml { } } --- upstream +server_name test-upstream.lvh.me; + location / { echo 'yay, api backend: $http_host'; } --- request GET /?user_key=value --- response_body env -yay, api backend: test:$TEST_NGINX_SERVER_PORT +yay, api backend: test-upstream.lvh.me:$TEST_NGINX_SERVER_PORT --- error_code: 200 --- error_log env -proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT HTTP/1.1 +proxy request: CONNECT test_backend.lvh.me:$TEST_NGINX_RANDOM_PORT HTTP/1.1 --- no_error_log [error] --- user_files fixture=tls.pl eval @@ -244,7 +255,8 @@ proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT HTTP/1.1 --- env eval ( 'http_proxy' => $ENV{TEST_NGINX_HTTP_PROXY}, - 'APICAST_REPORTING_THREADS' => '1' + 'APICAST_REPORTING_THREADS' => '1', + 'BACKEND_ENDPOINT_OVERRIDE' => "http://test_backend.lvh.me:$ENV{TEST_NGINX_SERVER_PORT}" ) --- configuration { @@ -255,7 +267,7 @@ proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT HTTP/1.1 "backend_authentication_type": "service_token", "backend_authentication_value": "token-value", "proxy": { - "api_backend": "http://test:$TEST_NGINX_SERVER_PORT/", + "api_backend": "http://test-upstream.lvh.me:$TEST_NGINX_SERVER_PORT/", "proxy_rules": [ { "pattern": "/", "http_method": "GET", "metric_system_name": "hits", "delta": 2 } ] @@ -264,6 +276,8 @@ proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT HTTP/1.1 ] } --- backend +server_name test_backend.lvh.me; + location /transactions/authrep.xml { content_by_lua_block { local expected = "service_token=token-value&service_id=42&usage%5Bhits%5D=2&user_key=value" @@ -271,6 +285,7 @@ proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT HTTP/1.1 } } --- upstream +server_name test-upstream.lvh.me; location / { echo 'yay, api backend: $http_host'; } @@ -295,10 +310,10 @@ location /apicast { proxy_pass http://$server_addr:$apicast_port; } --- response_body_like -yay, api backend: test:.* +yay, api backend: test-upstream.lvh.me:.* --- error_code: 200 --- error_log env -proxy request: GET http://127.0.0.1:$TEST_NGINX_SERVER_PORT/transactions/authrep.xml? +proxy request: GET http://test_backend.lvh.me:$TEST_NGINX_SERVER_PORT/transactions/authrep.xml? apicast cache write key: 42:value:usage%5Bhits%5D=2, ttl: nil, context: ngx.timer --- no_error_log [error] @@ -310,7 +325,7 @@ apicast cache write key: 42:value:usage%5Bhits%5D=2, ttl: nil, context: ngx.time ( 'https_proxy' => $ENV{TEST_NGINX_HTTP_PROXY}, 'APICAST_REPORTING_THREADS' => '1', - 'BACKEND_ENDPOINT_OVERRIDE' => "https://test_backend:$ENV{TEST_NGINX_RANDOM_PORT}" + 'BACKEND_ENDPOINT_OVERRIDE' => "https://test_backend.lvh.me:$ENV{TEST_NGINX_RANDOM_PORT}" ) --- configuration env { @@ -321,7 +336,7 @@ apicast cache write key: 42:value:usage%5Bhits%5D=2, ttl: nil, context: ngx.time "backend_authentication_type": "service_token", "backend_authentication_value": "token-value", "proxy": { - "api_backend": "http://test:$TEST_NGINX_SERVER_PORT/", + "api_backend": "http://test-upstream.lvh.me:$TEST_NGINX_SERVER_PORT/", "proxy_rules": [ { "pattern": "/", "http_method": "GET", "metric_system_name": "hits", "delta": 2 } ] @@ -330,6 +345,7 @@ apicast cache write key: 42:value:usage%5Bhits%5D=2, ttl: nil, context: ngx.time ] } --- backend env +server_name test_backend.lvh.me; listen $TEST_NGINX_RANDOM_PORT ssl; ssl_certificate $TEST_NGINX_SERVER_ROOT/html/server.crt; @@ -340,7 +356,7 @@ location /transactions/authrep.xml { assert = require('luassert') assert.equal('https', ngx.var.scheme) assert.equal('$TEST_NGINX_RANDOM_PORT', ngx.var.server_port) - assert.equal('test_backend', ngx.var.ssl_server_name) + assert.equal('test_backend.lvh.me', ngx.var.ssl_server_name) } content_by_lua_block { @@ -349,6 +365,7 @@ location /transactions/authrep.xml { } } --- upstream +server_name test-upstream.lvh.me; location / { echo 'yay, api backend: $http_host'; } @@ -373,10 +390,10 @@ location /apicast { proxy_pass http://$server_addr:$apicast_port; } --- response_body env -yay, api backend: test:$TEST_NGINX_SERVER_PORT +yay, api backend: test-upstream.lvh.me:$TEST_NGINX_SERVER_PORT --- error_code: 200 --- error_log env -proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT +proxy request: CONNECT test_backend.lvh.me:$TEST_NGINX_RANDOM_PORT apicast cache write key: 42:value:usage%5Bhits%5D=2, ttl: nil, context: ngx.timer --- no_error_log [error] @@ -386,14 +403,17 @@ apicast cache write key: 42:value:usage%5Bhits%5D=2, ttl: nil, context: ngx.time === TEST 8: upstream API connection uses proxy --- env eval -("http_proxy" => $ENV{TEST_NGINX_HTTP_PROXY}) +( + "http_proxy" => $ENV{TEST_NGINX_HTTP_PROXY}, + 'BACKEND_ENDPOINT_OVERRIDE' => "http://test_backend.lvh.me:$ENV{TEST_NGINX_SERVER_PORT}" +) --- configuration { "services": [ { "backend_version": 1, "proxy": { - "api_backend": "http://test:$TEST_NGINX_SERVER_PORT", + "api_backend": "http://test-upstream.lvh.me:$TEST_NGINX_SERVER_PORT", "proxy_rules": [ { "pattern": "/test", "http_method": "GET", "metric_system_name": "hits", "delta": 2 } ] @@ -402,24 +422,25 @@ apicast cache write key: 42:value:usage%5Bhits%5D=2, ttl: nil, context: ngx.time ] } --- backend +server_name test_backend.lvh.me; location /transactions/authrep.xml { content_by_lua_block { ngx.exit(ngx.OK) } } --- upstream +server_name test-upstream.lvh.me; location /test { echo 'yay, api backend: $http_host, uri: $uri, is_args: $is_args, args: $args'; # echo 'yay, api backend: $http_host, uri:'; } --- request GET /test?user_key=value - --- response_body_like eval qw/yay, api backend: test:ooo\d+, uri: \/test, is_args: \?, args: user_key=value/ --- error_code: 200 --- error_log env -proxy request: GET http://127.0.0.1:$TEST_NGINX_SERVER_PORT/test?user_key=value HTTP/1.1 +proxy request: GET http://test-upstream.lvh.me:$TEST_NGINX_SERVER_PORT/test?user_key=value HTTP/1.1 --- no_error_log [error] @@ -427,14 +448,17 @@ proxy request: GET http://127.0.0.1:$TEST_NGINX_SERVER_PORT/test?user_key=value === TEST 9: upstream API connection uses proxy for https --- env eval -("https_proxy" => $ENV{TEST_NGINX_HTTPS_PROXY}) +( + "https_proxy" => $ENV{TEST_NGINX_HTTPS_PROXY}, + 'BACKEND_ENDPOINT_OVERRIDE' => "http://test_backend.lvh.me:$ENV{TEST_NGINX_SERVER_PORT}" +) --- configuration random_port env { "services": [ { "backend_version": 1, "proxy": { - "api_backend": "https://test:$TEST_NGINX_RANDOM_PORT", + "api_backend": "https://test-upstream.lvh.me:$TEST_NGINX_RANDOM_PORT", "proxy_rules": [ { "pattern": "/test", "http_method": "GET", "metric_system_name": "hits", "delta": 2 } ] @@ -443,27 +467,28 @@ proxy request: GET http://127.0.0.1:$TEST_NGINX_SERVER_PORT/test?user_key=value ] } --- backend +server_name test_backend.lvh.me; + location /transactions/authrep.xml { content_by_lua_block { ngx.exit(ngx.OK) } } --- upstream env -listen $TEST_NGINX_RANDOM_PORT ssl; +server_name test-upstream.lvh.me; +listen $TEST_NGINX_RANDOM_PORT ssl; ssl_certificate $TEST_NGINX_SERVER_ROOT/html/server.crt; ssl_certificate_key $TEST_NGINX_SERVER_ROOT/html/server.key; - location /test { echo_foreach_split '\r\n' $echo_client_request_headers; echo $echo_it; echo_end; - access_by_lua_block { assert = require('luassert') assert.equal('https', ngx.var.scheme) assert.equal('$TEST_NGINX_RANDOM_PORT', ngx.var.server_port) - assert.equal('test', ngx.var.ssl_server_name) + assert.equal('test-upstream.lvh.me', ngx.var.ssl_server_name) } } --- request @@ -477,25 +502,28 @@ ETag: foobar qr{ETag\: foobar}, qr{Connection\: close}, qr{User\-Agent\: Test\:\:APIcast\:\:Blackbox}, - qr{Host\: test\:\d+} + qr{Host\: test-upstream.lvh.me\:\d+} ]] --- error_code: 200 --- error_log env -proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT HTTP/1.1 +proxy request: CONNECT test-upstream.lvh.me:$TEST_NGINX_RANDOM_PORT HTTP/1.1 --- no_error_log [error] --- user_files fixture=tls.pl eval === TEST 10: Upstream API with HTTPS POST request, HTTPS_PROXY and HTTPS api_backend ---- env eval -("https_proxy" => $ENV{TEST_NGINX_HTTPS_PROXY}) +--- env random_port eval +( + 'https_proxy' => $ENV{TEST_NGINX_HTTPS_PROXY}, + 'BACKEND_ENDPOINT_OVERRIDE' => "https://test-backend.lvh.me:$ENV{TEST_NGINX_RANDOM_PORT}" +) --- configuration random_port env { "services": [ { "backend_version": 1, "proxy": { - "api_backend": "https://test:$TEST_NGINX_RANDOM_PORT", + "api_backend": "https://test-upstream.lvh.me:$TEST_NGINX_RANDOM_PORT", "proxy_rules": [ { "pattern": "/test", "http_method": "POST", "metric_system_name": "hits", "delta": 2 } ] @@ -503,13 +531,19 @@ proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT HTTP/1.1 } ] } ---- backend +--- backend env + server_name test-backend.lvh.me; + listen $TEST_NGINX_RANDOM_PORT ssl; + ssl_certificate $TEST_NGINX_SERVER_ROOT/html/server.crt; + ssl_certificate_key $TEST_NGINX_SERVER_ROOT/html/server.key; + location /transactions/authrep.xml { content_by_lua_block { ngx.exit(ngx.OK) } } --- upstream env +server_name test-upstream.lvh.me; listen $TEST_NGINX_RANDOM_PORT ssl; ssl_certificate $TEST_NGINX_SERVER_ROOT/html/server.crt; @@ -526,13 +560,13 @@ POST https://localhost/test?user_key=test3 { "some_param": "some_value" } --- error_code: 200 --- error_log env -proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT HTTP/1.1 +proxy request: CONNECT test-upstream.lvh.me:$TEST_NGINX_RANDOM_PORT HTTP/1.1 --- no_error_log [error] --- user_files fixture=tls.pl eval === TEST 11: Upstream Policy connection uses proxy ---- env eval +--- env random_port eval ("http_proxy" => $ENV{TEST_NGINX_HTTP_PROXY}) --- configuration { @@ -543,7 +577,7 @@ proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT HTTP/1.1 { "name": "apicast.policy.upstream", "configuration": { - "rules": [ { "regex": "/test", "url": "http://test" } ] + "rules": [ { "regex": "/test", "url": "http://test-upstream.lvh.me:$TEST_NGINX_SERVER_PORT" } ] } } ] @@ -552,6 +586,7 @@ proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT HTTP/1.1 ] } --- upstream +server_name test-upstream.lvh.me; location /test { echo_foreach_split '\r\n' $echo_client_request_headers; echo $echo_it; @@ -562,20 +597,18 @@ GET /test?user_key=test3 --- more_headers User-Agent: Test::APIcast::Blackbox ETag: foobar ---- response_body +--- response_body env GET /test?user_key=test3 HTTP/1.1 X-Real-IP: 127.0.0.1 -Host: test +Host: test-upstream.lvh.me:$TEST_NGINX_SERVER_PORT User-Agent: Test::APIcast::Blackbox ETag: foobar --- error_code: 200 --- error_log env -proxy request: GET http://127.0.0.1:$TEST_NGINX_SERVER_PORT/test?user_key=test3 HTTP/1.1 +proxy request: GET http://test-upstream.lvh.me:$TEST_NGINX_SERVER_PORT/test?user_key=test3 HTTP/1.1 --- no_error_log [error] - - === TEST 12: Upstream Policy connection uses proxy for https --- env eval ("https_proxy" => $ENV{TEST_NGINX_HTTPS_PROXY}) @@ -588,7 +621,7 @@ proxy request: GET http://127.0.0.1:$TEST_NGINX_SERVER_PORT/test?user_key=test3 { "name": "apicast.policy.upstream", "configuration": { - "rules": [ { "regex": "/test", "url": "https://test:$TEST_NGINX_RANDOM_PORT" } ] + "rules": [ { "regex": "/test", "url": "https://test-upstream.lvh.me:$TEST_NGINX_RANDOM_PORT" } ] } } ] @@ -597,6 +630,7 @@ proxy request: GET http://127.0.0.1:$TEST_NGINX_SERVER_PORT/test?user_key=test3 ] } --- upstream env +server_name test-upstream.lvh.me; listen $TEST_NGINX_RANDOM_PORT ssl; ssl_certificate $TEST_NGINX_SERVER_ROOT/html/server.crt; @@ -611,7 +645,7 @@ location /test { assert = require('luassert') assert.equal('https', ngx.var.scheme) assert.equal('$TEST_NGINX_RANDOM_PORT', ngx.var.server_port) - assert.equal('test', ngx.var.ssl_server_name) + assert.equal('test-upstream.lvh.me', ngx.var.ssl_server_name) } } --- request @@ -625,16 +659,15 @@ ETag: foobar qr{ETag\: foobar}, qr{Connection\: close}, qr{User\-Agent\: Test\:\:APIcast\:\:Blackbox}, - qr{Host\: test\:\d+} + qr{Host\: test-upstream\.lvh\.me\:\d+} ]] --- error_code: 200 --- error_log env -proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT HTTP/1.1 +proxy request: CONNECT test-upstream.lvh.me:$TEST_NGINX_RANDOM_PORT HTTP/1.1 --- no_error_log [error] --- user_files fixture=tls.pl eval - === TEST 13: Upstream Policy connection uses proxy --- env eval ("http_proxy" => $ENV{TEST_NGINX_HTTP_PROXY}) @@ -647,7 +680,7 @@ proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT HTTP/1.1 { "name": "apicast.policy.upstream", "configuration": { - "rules": [ { "regex": "/test", "url": "http://test" } ] + "rules": [ { "regex": "/test", "url": "http://test-upstream.lvh.me:$TEST_NGINX_SERVER_PORT" } ] } } ] @@ -656,6 +689,7 @@ proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT HTTP/1.1 ] } --- upstream +server_name test-upstream.lvh.me; location /test { echo_foreach_split '\r\n' $echo_client_request_headers; echo $echo_it; @@ -669,21 +703,20 @@ POST /test?user_key=test3 this-is-some-request-body --- more_headers User-Agent: Test::APIcast::Blackbox ---- response_body +--- response_body env POST /test?user_key=test3 HTTP/1.1 X-Real-IP: 127.0.0.1 -Host: test +Host: test-upstream.lvh.me:$TEST_NGINX_SERVER_PORT Content-Length: 25 User-Agent: Test::APIcast::Blackbox this-is-some-request-body --- error_code: 200 --- error_log env -proxy request: POST http://127.0.0.1:$TEST_NGINX_SERVER_PORT/test?user_key=test3 HTTP/1.1 +proxy request: POST http://test-upstream.lvh.me:$TEST_NGINX_SERVER_PORT/test?user_key=test3 HTTP/1.1 --- no_error_log [error] - === TEST 14: Upstream Policy connection uses proxy for https and forwards request body --- env eval ("https_proxy" => $ENV{TEST_NGINX_HTTPS_PROXY}) @@ -696,7 +729,7 @@ proxy request: POST http://127.0.0.1:$TEST_NGINX_SERVER_PORT/test?user_key=test3 { "name": "apicast.policy.upstream", "configuration": { - "rules": [ { "regex": "/test", "url": "https://test:$TEST_NGINX_RANDOM_PORT" } ] + "rules": [ { "regex": "/test", "url": "https://test-upstream.lvh.me:$TEST_NGINX_RANDOM_PORT" } ] } } ] @@ -705,6 +738,7 @@ proxy request: POST http://127.0.0.1:$TEST_NGINX_SERVER_PORT/test?user_key=test3 ] } --- upstream env +server_name test-upstream.lvh.me; listen $TEST_NGINX_RANDOM_PORT ssl; ssl_certificate $TEST_NGINX_SERVER_ROOT/html/server.crt; @@ -722,7 +756,7 @@ location /test { assert = require('luassert') assert.equal('https', ngx.var.scheme) assert.equal('$TEST_NGINX_RANDOM_PORT', ngx.var.server_port) - assert.equal('test', ngx.var.ssl_server_name) + assert.equal('test-upstream.lvh.me', ngx.var.ssl_server_name) } } --- request @@ -735,28 +769,30 @@ User-Agent: Test::APIcast::Blackbox qr{POST \/test\?user_key=test3 HTTP\/1\.1}, qr{Connection\: close}, qr{User\-Agent\: Test\:\:APIcast\:\:Blackbox}, - qr{Host\: test\:\d+}, + qr{Host\: test-upstream\.lvh\.me\:\d+}, qr{Content\-Length\: 25}, qr{this\-is\-some\-request\-body}, ]] --- error_code: 200 --- error_log env -proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT HTTP/1.1 +proxy request: CONNECT test-upstream.lvh.me:$TEST_NGINX_RANDOM_PORT HTTP/1.1 --- no_error_log [error] --- user_files fixture=tls.pl eval - === TEST 15: upstream API connection uses proxy and correctly routes to a path. --- env eval -("http_proxy" => $ENV{TEST_NGINX_HTTP_PROXY}) +( + "http_proxy" => $ENV{TEST_NGINX_HTTP_PROXY}, + 'BACKEND_ENDPOINT_OVERRIDE' => "http://test-backend.lvh.me:$ENV{TEST_NGINX_SERVER_PORT}" +) --- configuration { "services": [ { "backend_version": 1, "proxy": { - "api_backend": "http://test:$TEST_NGINX_SERVER_PORT/foo", + "api_backend": "http://test-upstream.lvh.me:$TEST_NGINX_SERVER_PORT/foo", "proxy_rules": [ { "pattern": "/test", "http_method": "GET", "metric_system_name": "hits", "delta": 2 } ] @@ -764,13 +800,16 @@ proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT HTTP/1.1 } ] } ---- backend +--- backend env + server_name test-backend.lvh.me; + location /transactions/authrep.xml { content_by_lua_block { ngx.exit(ngx.OK) } } --- upstream +server_name test-upstream.lvh.me; location / { echo $request; } @@ -780,11 +819,10 @@ GET /test?user_key=value GET /foo/test?user_key=value HTTP/1.1 --- error_code: 200 --- error_log env -proxy request: GET http://127.0.0.1:$TEST_NGINX_SERVER_PORT/foo/test?user_key=value HTTP/1.1 +proxy request: GET http://test-upstream.lvh.me:$TEST_NGINX_SERVER_PORT/foo/test?user_key=value HTTP/1.1 --- no_error_log [error] - === TEST 16: Upstream Policy connection uses proxy and correctly routes to a path. --- env eval ("http_proxy" => $ENV{TEST_NGINX_HTTP_PROXY}) @@ -797,7 +835,7 @@ proxy request: GET http://127.0.0.1:$TEST_NGINX_SERVER_PORT/foo/test?user_key=va { "name": "apicast.policy.upstream", "configuration": { - "rules": [ { "regex": "/test", "url": "http://test/foo" } ] + "rules": [ { "regex": "/test", "url": "http://test-upstream.lvh.me:$TEST_NGINX_SERVER_PORT/foo" } ] } } ] @@ -806,6 +844,7 @@ proxy request: GET http://127.0.0.1:$TEST_NGINX_SERVER_PORT/foo/test?user_key=va ] } --- upstream +server_name test-upstream.lvh.me; location / { echo $request; } @@ -815,20 +854,23 @@ GET /test?user_key=value GET /foo/test?user_key=value HTTP/1.1 --- error_code: 200 --- error_log env -proxy request: GET http://127.0.0.1:$TEST_NGINX_SERVER_PORT/foo/test?user_key=value HTTP/1.1 +proxy request: GET http://test-upstream.lvh.me:$TEST_NGINX_SERVER_PORT/foo/test?user_key=value HTTP/1.1 --- no_error_log [error] === TEST 17: Upstream policy with HTTPS POST request, HTTPS_PROXY and HTTPS backend ---- env eval -("https_proxy" => $ENV{TEST_NGINX_HTTPS_PROXY}) +--- env random_port eval +( + 'https_proxy' => $ENV{TEST_NGINX_HTTPS_PROXY}, + 'BACKEND_ENDPOINT_OVERRIDE' => "https://test-backend.lvh.me:$ENV{TEST_NGINX_RANDOM_PORT}" +) --- configuration random_port env { "services": [ { "backend_version": 1, "proxy": { - "api_backend": "https://test:$TEST_NGINX_RANDOM_PORT", + "api_backend": "https://test-upstream.lvh.me:$TEST_NGINX_RANDOM_PORT", "proxy_rules": [ { "pattern": "/test", "http_method": "POST", "metric_system_name": "hits", "delta": 2 } ], @@ -836,7 +878,7 @@ proxy request: GET http://127.0.0.1:$TEST_NGINX_SERVER_PORT/foo/test?user_key=va { "name": "apicast.policy.upstream", "configuration": { - "rules": [ { "regex": "/test", "url": "https://test:$TEST_NGINX_RANDOM_PORT" } ] + "rules": [ { "regex": "/test", "url": "https://test-upstream.lvh.me:$TEST_NGINX_RANDOM_PORT" } ] } }, { @@ -847,13 +889,19 @@ proxy request: GET http://127.0.0.1:$TEST_NGINX_SERVER_PORT/foo/test?user_key=va } ] } ---- backend +--- backend env + server_name test-backend.lvh.me; + listen $TEST_NGINX_RANDOM_PORT ssl; + ssl_certificate $TEST_NGINX_SERVER_ROOT/html/server.crt; + ssl_certificate_key $TEST_NGINX_SERVER_ROOT/html/server.key; + location /transactions/authrep.xml { content_by_lua_block { ngx.exit(ngx.OK) } } --- upstream env +server_name test-upstream.lvh.me; listen $TEST_NGINX_RANDOM_PORT ssl; ssl_certificate $TEST_NGINX_SERVER_ROOT/html/server.crt; @@ -870,7 +918,7 @@ POST https://localhost/test?user_key=test3 { "some_param": "some_value" } --- error_code: 200 --- error_log env -proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT HTTP/1.1 +proxy request: CONNECT test-upstream.lvh.me:$TEST_NGINX_RANDOM_PORT HTTP/1.1 --- no_error_log [error] --- user_files fixture=tls.pl eval @@ -887,7 +935,7 @@ proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT HTTP/1.1 { "name": "apicast.policy.upstream", "configuration": { - "rules": [ { "regex": "/test", "url": "http://test" } ] + "rules": [ { "regex": "/test", "url": "http://test-upstream.lvh.me:$TEST_NGINX_SERVER_PORT" } ] } } ] @@ -896,6 +944,7 @@ proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT HTTP/1.1 ] } --- upstream +server_name test-upstream.lvh.me; location /test { echo_foreach_split '\r\n' $echo_client_request_headers; echo $echo_it; @@ -903,18 +952,18 @@ proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT HTTP/1.1 } --- request GET /test ---- response_body +--- response_body env GET /test HTTP/1.1 X-Real-IP: 127.0.0.1 -Host: test +Host: test-upstream.lvh.me:$TEST_NGINX_SERVER_PORT --- error_code: 200 --- error_log env -proxy request: GET http://127.0.0.1:$TEST_NGINX_SERVER_PORT/test HTTP/1.1 +proxy request: GET http://test-upstream.lvh.me:$TEST_NGINX_SERVER_PORT/test HTTP/1.1 --- no_error_log [error] === TEST 19: The path is set correctly when there are no args and proxied to an https upstream -Regression test: the string 'nil' was appended to the path +Regression test.lvh.me: the string 'nil' was appended to the path --- env eval ("https_proxy" => $ENV{TEST_NGINX_HTTPS_PROXY}) --- configuration random_port env @@ -926,7 +975,7 @@ Regression test: the string 'nil' was appended to the path { "name": "apicast.policy.upstream", "configuration": { - "rules": [ { "regex": "/test", "url": "https://test:$TEST_NGINX_RANDOM_PORT" } ] + "rules": [ { "regex": "/test", "url": "https://test-upstream.lvh.me:$TEST_NGINX_RANDOM_PORT" } ] } } ] @@ -935,6 +984,7 @@ Regression test: the string 'nil' was appended to the path ] } --- upstream env +server_name test-upstream.lvh.me; listen $TEST_NGINX_RANDOM_PORT ssl; ssl_certificate $TEST_NGINX_SERVER_ROOT/html/server.crt; @@ -949,7 +999,7 @@ location /test { assert = require('luassert') assert.equal('https', ngx.var.scheme) assert.equal('$TEST_NGINX_RANDOM_PORT', ngx.var.server_port) - assert.equal('test', ngx.var.ssl_server_name) + assert.equal('test-upstream.lvh.me', ngx.var.ssl_server_name) } } --- request @@ -963,16 +1013,15 @@ ETag: foobar qr{ETag\: foobar}, qr{Connection\: close}, qr{User\-Agent\: Test\:\:APIcast\:\:Blackbox}, - qr{Host\: test\:\d+} + qr{Host\: test-upstream\.lvh\.me\:\d+} ]] --- error_code: 200 --- error_log env -proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT HTTP/1.1 +proxy request: CONNECT test-upstream.lvh.me:$TEST_NGINX_RANDOM_PORT HTTP/1.1 --- no_error_log [error] --- user_files fixture=tls.pl eval - === TEST 20: Body is larger than client_body_buffer --- env eval ("https_proxy" => $ENV{TEST_NGINX_HTTPS_PROXY}) @@ -985,7 +1034,7 @@ proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT HTTP/1.1 { "name": "apicast.policy.upstream", "configuration": { - "rules": [ { "regex": "/test", "url": "https://test:$TEST_NGINX_RANDOM_PORT" } ] + "rules": [ { "regex": "/test", "url": "https://test-upstream.lvh.me:$TEST_NGINX_RANDOM_PORT" } ] } } ] @@ -994,6 +1043,7 @@ proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT HTTP/1.1 ] } --- upstream env +server_name test-upstream.lvh.me; listen $TEST_NGINX_RANDOM_PORT ssl; ssl_certificate $TEST_NGINX_SERVER_ROOT/html/server.crt; @@ -1008,7 +1058,7 @@ location /test { assert = require('luassert') assert.equal('https', ngx.var.scheme) assert.equal('$TEST_NGINX_RANDOM_PORT', ngx.var.server_port) - assert.equal('test', ngx.var.ssl_server_name) + assert.equal('test-upstream.lvh.me', ngx.var.ssl_server_name) ngx.req.read_body() local handle = io.open(ngx.req.get_body_file(), "r") @@ -1020,7 +1070,7 @@ location /test { "POST /test \n" . $ENV{LARGE_BODY} --- error_code: 200 --- error_log env -proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT HTTP/1.1 +proxy request: CONNECT test-upstream.lvh.me:$TEST_NGINX_RANDOM_PORT HTTP/1.1 Request body is bigger than client_body_buffer_size --- no_error_log [error] From d9b6f308696f1428dfde41e41c7e4e2f6d14ea17 Mon Sep 17 00:00:00 2001 From: kevprice83 Date: Wed, 20 Apr 2022 13:00:28 +0200 Subject: [PATCH 3/5] updates CHANGELOG entry, reverts some unwanted changes introduced in commit 97c0c46 & adds back repeat_each(3) in integration test --- CHANGELOG.md | 1 + gateway/src/apicast/http_proxy.lua | 14 +++++++++++--- gateway/src/resty/http/proxy.lua | 6 +++++- t/http-proxy.t | 2 +- 4 files changed, 18 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f8edb9f58..e235bb554 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,6 +37,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/). - Fixed dirty context [PR #1328](https://github.com/3scale/APIcast/pull/1328) [THREESCALE-8000](https://issues.redhat.com/browse/THREESCALE-8000) [THREESCALE-8007](https://issues.redhat.com/browse/THREESCALE-8007) - Fixed jwk alg confusion [PR #1329](https://github.com/3scale/APIcast/pull/1329) [THREESCALE-8249](https://issues.redhat.com/browse/THREESCALE-8249) - Fixed issue with resolving target server hostnames to IP when using CONNECT method [PR #1323](https://github.com/3scale/APIcast/pull/1323) [THREESCALE-7967](https://issues.redhat.com/browse/THREESCALE-7967) +- Fixed issue with resolving target server hostnames to IPs when forwarding requests through http/s proxy [PR #1323](https://github.com/3scale/APIcast/pull/1323) [THREESCALE-7967](https://issues.redhat.com/browse/THREESCALE-7967) ### Added diff --git a/gateway/src/apicast/http_proxy.lua b/gateway/src/apicast/http_proxy.lua index 6439f9827..5a62e1bd1 100644 --- a/gateway/src/apicast/http_proxy.lua +++ b/gateway/src/apicast/http_proxy.lua @@ -71,7 +71,8 @@ local function resolve(uri) end local function absolute_url(uri) - +-- target server requires hostname not IP and DNS resolution is left to the proxy itself as specified in the RFC #7231 +-- https://httpwg.org/specs/rfc7231.html#CONNECT return format('%s://%s:%s%s', uri.scheme, uri.host, @@ -159,9 +160,16 @@ end function _M.request(upstream, proxy_uri) local uri = upstream.uri - if uri.scheme == 'http' then + if uri.scheme == 'http' then -- rewrite the request to use http_proxy + local err + local host = upstream:set_host_header() + upstream:use_host_header(host) + upstream.servers, err = resolve_servers(proxy_uri) + if err then + ngx.log(ngx.WARN, "HTTP proxy is set, but no servers have been resolved, err: ", err) + end + upstream.uri.path = absolute_url(uri) upstream:rewrite_request() - forward_https_request(proxy_uri, uri, upstream.skip_https_connect) return elseif uri.scheme == 'https' then upstream:rewrite_request() diff --git a/gateway/src/resty/http/proxy.lua b/gateway/src/resty/http/proxy.lua index a399d6c81..72a299d30 100644 --- a/gateway/src/resty/http/proxy.lua +++ b/gateway/src/resty/http/proxy.lua @@ -72,6 +72,8 @@ local function _connect_proxy_https(httpc, request, host, port) end local function connect_proxy(httpc, request, skip_https_connect) + -- target server requires hostname not IP and DNS resolution is left to the proxy itself as specified in the RFC #7231 + -- https://httpwg.org/specs/rfc7231.html#CONNECT local uri = request.uri local proxy_uri = request.proxy @@ -93,9 +95,11 @@ local function connect_proxy(httpc, request, skip_https_connect) ngx.log(ngx.DEBUG, 'connection to ', proxy_uri.host, ':', proxy_uri.port, ' established', ', pool: ', options.pool, ' reused times: ', httpc:get_reused_times()) + ngx.log(ngx.DEBUG, 'targeting server ', uri.host, ':', uri.port) + if uri.scheme == 'http' then -- http proxy needs absolute URL as the request path - request.path = format('%s://%s:%s%s', uri.scheme, uri.host, uri.port, uri.path or request.path or '/') + request.path = format('%s://%s:%s%s', uri.scheme, uri.host, uri.port, uri.path or '/') return httpc elseif uri.scheme == 'https' and skip_https_connect then request.path = format('%s://%s:%s%s', uri.scheme, uri.host, uri.port, request.path or '/') diff --git a/t/http-proxy.t b/t/http-proxy.t index 657ed3ba0..06fc744fb 100644 --- a/t/http-proxy.t +++ b/t/http-proxy.t @@ -15,7 +15,7 @@ $ENV{'LARGE_BODY'} = large_body(); require("policies.pl"); -repeat_each(1); +repeat_each(3); run_tests(); From dc752e93c60085751c03ec6d22010805074c9928 Mon Sep 17 00:00:00 2001 From: kevprice83 Date: Wed, 20 Apr 2022 14:59:25 +0200 Subject: [PATCH 4/5] fixes broken http proxy policy integration test --- t/apicast-policy-http-proxy.t | 27 +++++++++++++++------------ 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/t/apicast-policy-http-proxy.t b/t/apicast-policy-http-proxy.t index 5dfd15c23..d5865896e 100644 --- a/t/apicast-policy-http-proxy.t +++ b/t/apicast-policy-http-proxy.t @@ -19,7 +19,7 @@ __DATA__ "backend_authentication_type": "service_token", "backend_authentication_value": "token-value", "proxy": { - "api_backend": "http://test:$TEST_NGINX_SERVER_PORT/", + "api_backend": "http://test-upstream.lvh.me:$TEST_NGINX_SERVER_PORT/", "proxy_rules": [ { "pattern": "/", "http_method": "GET", "metric_system_name": "hits", "delta": 2 } ], @@ -46,12 +46,13 @@ __DATA__ } } --- upstream +server_name test-upstream.lvh.me; location / { access_by_lua_block { local host = ngx.req.get_headers()["Host"] - local result = string.match(host, "^test:") + local result = string.match(host, "^test%-upstream%.lvh%.me:") local assert = require('luassert') - assert.equals(result, "test:") + assert.equals(result, "test-upstream.lvh.me:") ngx.say("yay, api backend") } } @@ -73,7 +74,7 @@ using proxy: $TEST_NGINX_HTTP_PROXY "backend_authentication_type": "service_token", "backend_authentication_value": "token-value", "proxy": { - "api_backend": "http://test:$TEST_NGINX_SERVER_PORT/", + "api_backend": "http://test-upstream.lvh.me:$TEST_NGINX_SERVER_PORT/", "proxy_rules": [ { "pattern": "/", "http_method": "GET", "metric_system_name": "hits", "delta": 2 } ], @@ -100,12 +101,13 @@ using proxy: $TEST_NGINX_HTTP_PROXY } } --- upstream +server_name test-upstream.lvh.me; location / { access_by_lua_block { local host = ngx.req.get_headers()["Host"] - local result = string.match(host, "^test:") + local result = string.match(host, "^test%-upstream%.lvh%.me:") local assert = require('luassert') - assert.equals(result, "test:") + assert.equals(result, "test-upstream.lvh.me:") ngx.say("yay, api backend") } } @@ -125,7 +127,7 @@ using proxy: $TEST_NGINX_HTTP_PROXY { "backend_version": 1, "proxy": { - "api_backend": "https://test:$TEST_NGINX_RANDOM_PORT", + "api_backend": "https://test-upstream.lvh.me:$TEST_NGINX_RANDOM_PORT", "proxy_rules": [ { "pattern": "/test", "http_method": "GET", "metric_system_name": "hits", "delta": 2 } ], @@ -151,6 +153,7 @@ using proxy: $TEST_NGINX_HTTP_PROXY } } --- upstream env +server_name test-upstream.lvh.me; listen $TEST_NGINX_RANDOM_PORT ssl; ssl_certificate $TEST_NGINX_SERVER_ROOT/html/server.crt; @@ -165,11 +168,11 @@ location /test { assert = require('luassert') assert.equal('https', ngx.var.scheme) assert.equal('$TEST_NGINX_RANDOM_PORT', ngx.var.server_port) - assert.equal('test', ngx.var.ssl_server_name) + assert.equal('test-upstream.lvh.me', ngx.var.ssl_server_name) local host = ngx.req.get_headers()["Host"] - local result = string.match(host, "^test:") - assert.equals(result, "test:") + local result = string.match(host, "^test%-upstream%.lvh%.me:") + assert.equals(result, "test-upstream.lvh.me:") } } --- request @@ -183,11 +186,11 @@ ETag: foobar qr{ETag\: foobar}, qr{Connection\: close}, qr{User\-Agent\: Test\:\:APIcast\:\:Blackbox}, - qr{Host\: test\:\d+} + qr{Host\: test-upstream.lvh.me\:\d+} ]] --- error_code: 200 --- error_log env -proxy request: CONNECT 127.0.0.1:$TEST_NGINX_RANDOM_PORT HTTP/1.1 +proxy request: CONNECT test-upstream.lvh.me:$TEST_NGINX_RANDOM_PORT HTTP/1.1 --- user_files fixture=tls.pl eval --- error_log env using proxy: $TEST_NGINX_HTTPS_PROXY From 4b47c81966e83840c36b30c17ffd62ef8f35d094 Mon Sep 17 00:00:00 2001 From: Samuele Illuminati Date: Wed, 20 Apr 2022 15:07:53 +0200 Subject: [PATCH 5/5] fixed camel policy tests --- t/apicast-policy-camel.t | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/t/apicast-policy-camel.t b/t/apicast-policy-camel.t index ef666eeb7..f95dcc081 100644 --- a/t/apicast-policy-camel.t +++ b/t/apicast-policy-camel.t @@ -9,7 +9,7 @@ run_tests(); __DATA__ -=== TEST 1: API backend connection uses http proxy +=== TEST 1: API backend connection uses http proxy --- configuration { "services": [ @@ -19,7 +19,7 @@ __DATA__ "backend_authentication_type": "service_token", "backend_authentication_value": "token-value", "proxy": { - "api_backend": "http://test:$TEST_NGINX_SERVER_PORT/", + "api_backend": "http://test-upstream.lvh.me:$TEST_NGINX_SERVER_PORT/", "proxy_rules": [ { "pattern": "/", "http_method": "GET", "metric_system_name": "hits", "delta": 2 } ], @@ -46,12 +46,13 @@ __DATA__ } } --- upstream + server_name test-upstream.lvh.me; location / { access_by_lua_block { local host = ngx.req.get_headers()["Host"] - local result = string.match(host, "^test:") + local result = string.match(host, "^test%-upstream%.lvh%.me:") local assert = require('luassert') - assert.equals(result, "test:") + assert.equals(result, "test-upstream.lvh.me:") ngx.say("yay, api backend") } } @@ -73,7 +74,7 @@ using proxy: $TEST_NGINX_HTTP_PROXY "backend_authentication_type": "service_token", "backend_authentication_value": "token-value", "proxy": { - "api_backend": "http://test:$TEST_NGINX_SERVER_PORT/", + "api_backend": "http://test-upstream.lvh.me:$TEST_NGINX_SERVER_PORT/", "proxy_rules": [ { "pattern": "/", "http_method": "GET", "metric_system_name": "hits", "delta": 2 } ], @@ -100,12 +101,13 @@ using proxy: $TEST_NGINX_HTTP_PROXY } } --- upstream + server_name test-upstream.lvh.me; location / { access_by_lua_block { local host = ngx.req.get_headers()["Host"] - local result = string.match(host, "^test:") + local result = string.match(host, "^test%-upstream%.lvh%.me:") local assert = require('luassert') - assert.equals(result, "test:") + assert.equals(result, "test-upstream.lvh.me:") ngx.say("yay, api backend") } } @@ -117,8 +119,7 @@ yay, api backend --- error_log env using proxy: $TEST_NGINX_HTTP_PROXY - -=== TEST 3: using HTTPS proxy for backend +=== TEST 3: using HTTPS proxy for backend. --- init eval $Test::Nginx::Util::PROXY_SSL_PORT = Test::APIcast::get_random_port(); $Test::Nginx::Util::ENDPOINT_SSL_PORT = Test::APIcast::get_random_port(); @@ -207,8 +208,6 @@ ETag: foobar using proxy: http://127.0.0.1:$Test::Nginx::Util::PROXY_SSL_PORT, EOF - - === TEST 4: using HTTPS proxy without api_backend upstream --- init eval $Test::Nginx::Util::PROXY_SSL_PORT = Test::APIcast::get_random_port();