Replace existing proxy config with global-agent
#572
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Cruikshanks
approved these changes
Sep 30, 2021
Cruikshanks
added a commit
to DEFRA/water-abstraction-system
that referenced
this pull request
Nov 10, 2022
DEFRA/water-abstraction-team#54 Our status page depends on sending requests to the other running apps and 3rd party services. All is fine in our Vagrant box but when we deployed to our AWS environments the requests just seemed to hang. After some investigation inspiration hit! Our AWS environments require all requests to go through a proxy server and without support for this, our requests are getting stuck. We tackled this same problem in [Replace existing proxy config with global-agent](DEFRA/sroc-charging-module-api#572) in the sroc-charging-module-api. In the Charging Module, we used [global-agent](https://github.com/gajus/global-agent). This intercepts all http/https requests and adds the proxy details them. Perfect! The problem is it was last updated more than a year ago, the build is marked as failing, test coverage is low, and issues and PR's seem to be going unanswered. Our other issue is it does not recognise the lowercase versions of standard proxy env vars (`http_proxy`, `https_proxy` and `no_proxy`). It's a shame because it works seemlessly in the background. The folks behind the request package we do use, [Got](https://github.com/sindresorhus/got), recommend a couple. We liked [hpagent](https://github.com/delvedor/hpagent/). The example was simple, it's being regularly updated, supports [keep-alive connections](https://en.wikipedia.org/wiki/HTTP_persistent_connection), and even uses [standardjs](https://standardjs.com/)! So, this change brings in support for working behind a proxy using **hpagent**. As part of this we'll be refactoring the work of making HTTP requests into its own service. We'll also solve one of the other things we noted; the requests don't timeout if there is a problem making a connection.
Cruikshanks
added a commit
to DEFRA/water-abstraction-system
that referenced
this pull request
Nov 10, 2022
DEFRA/water-abstraction-team#54 Our status page depends on sending requests to the other running apps and 3rd party services. All is fine in our Vagrant box but when we deployed to our AWS environments the requests just seemed to hang. After some investigation inspiration hit! Our AWS environments require all requests to go through a proxy server and without support for this, our requests are getting stuck. We tackled this same problem in [Replace existing proxy config with global-agent](DEFRA/sroc-charging-module-api#572) in the sroc-charging-module-api. In the Charging Module, we used [global-agent](https://github.com/gajus/global-agent). This intercepts all http/https requests and adds the proxy details them. Perfect! The problem is it was last updated more than a year ago, the build is marked as failing, test coverage is low, and issues and PR's seem to be going unanswered. Our other issue is it does not recognise the lowercase versions of standard proxy env vars (`http_proxy`, `https_proxy` and `no_proxy`). It's a shame because it works seemlessly in the background. The folks behind the request package we do use, [Got](https://github.com/sindresorhus/got), recommend a couple. We liked [hpagent](https://github.com/delvedor/hpagent/). The example was simple, it's being regularly updated, supports [keep-alive connections](https://en.wikipedia.org/wiki/HTTP_persistent_connection), and even uses [standardjs](https://standardjs.com/)! So, this change brings in support for working behind a proxy using **hpagent**. As part of this we'll be refactoring the work of making HTTP requests into its own service. We'll also solve one of the other things we noted; the requests don't timeout if there is a problem making a connection.
Cruikshanks
added a commit
to DEFRA/water-abstraction-system
that referenced
this pull request
Dec 12, 2022
DEFRA/water-abstraction-team#54 Our status page depends on sending requests to the other running apps and 3rd party services. All is fine in our Vagrant box but when we deployed to our AWS environments the requests just seemed to hang. After some investigation inspiration hit! Our AWS environments require all requests to go through a proxy server and without support for this, our requests are getting stuck. We tackled this same problem in [Replace existing proxy config with global-agent](DEFRA/sroc-charging-module-api#572) in the sroc-charging-module-api. In the Charging Module, we used [global-agent](https://github.com/gajus/global-agent). This intercepts all http/https requests and adds the proxy details them. Perfect! The problem is it was last updated more than a year ago, the build is marked as failing, test coverage is low, and issues and PR's seem to be going unanswered. Our other issue is it does not recognise the lowercase versions of standard proxy env vars (`http_proxy`, `https_proxy` and `no_proxy`). It's a shame because it works seemlessly in the background. The folks behind the request package we do use, [Got](https://github.com/sindresorhus/got), recommend a couple. We liked [hpagent](https://github.com/delvedor/hpagent/). The example was simple, it's being regularly updated, supports [keep-alive connections](https://en.wikipedia.org/wiki/HTTP_persistent_connection), and even uses [standardjs](https://standardjs.com/)! So, this change brings in support for working behind a proxy using **hpagent**. As part of this we'll be refactoring the work of making HTTP requests into its own service. We'll also solve one of the other things we noted; the requests don't timeout if there is a problem making a connection.
Cruikshanks
added a commit
to DEFRA/water-abstraction-system
that referenced
this pull request
Dec 15, 2022
Our AWS environments require all requests to go through a proxy server; without support, our requests would get stuck. We tackled this same problem in [Replace existing proxy config with global-agent](DEFRA/sroc-charging-module-api#572) in the sroc-charging-module-api. In the Charging Module, we used [global-agent](https://github.com/gajus/global-agent). This intercepts all http/https requests and adds the proxy details to them. Perfect! The problem is it was last updated more than a year ago, the build is marked as failing, test coverage is low, and issues and PR's seem to be going unanswered. Our other issue is it does not recognise the lowercase versions of standard proxy env vars (`http_proxy`, `https_proxy` and `no_proxy`). It's a shame because it works seamlessly in the background. The folks behind the request package we do use, [Got](https://github.com/sindresorhus/got), recommend a couple. We liked [hpagent](https://github.com/delvedor/hpagent/). The example was simple, it's being regularly updated, supports [keep-alive connections](https://en.wikipedia.org/wiki/HTTP_persistent_connection), and even uses [standardjs](https://standardjs.com/)! So, this change supports working behind a proxy using **hpagent**.
Cruikshanks
added a commit
to DEFRA/water-abstraction-system
that referenced
this pull request
Dec 15, 2022
Our AWS environments require all requests to go through a proxy server; without support, our requests would get stuck. We tackled this same problem in [Replace existing proxy config with global-agent](DEFRA/sroc-charging-module-api#572) in the sroc-charging-module-api. In the Charging Module, we used [global-agent](https://github.com/gajus/global-agent). This intercepts all http/https requests and adds the proxy details to them. Perfect! The problem is it was last updated more than a year ago, the build is marked as failing, test coverage is low, and issues and PR's seem to be going unanswered. Our other issue is it does not recognise the lowercase versions of standard proxy env vars (`http_proxy`, `https_proxy` and `no_proxy`). It's a shame because it works seamlessly in the background. The folks behind the request package we do use, [Got](https://github.com/sindresorhus/got), recommend a couple. We liked [hpagent](https://github.com/delvedor/hpagent/). The example was simple, it's being regularly updated, supports [keep-alive connections](https://en.wikipedia.org/wiki/HTTP_persistent_connection), and even uses [standardjs](https://standardjs.com/)! So, this change supports working behind a proxy using **hpagent**.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://eaflood.atlassian.net/jira/software/projects/CMEA/boards/907?selectedIssue=CMEA-167
We previously fixed proxy support for requests going to the Rules Service, however this is also needed for S3 requests -- WRLS are currently having issues with sending bill runs in their local setup and this may be due to S3 requests failing for proxy reasons.
While investigating how to configure the AWS SDK to use a proxy, we discovered that the
tunnelpackage we are using for Rules Service proxying does not appear to be currently maintained. Researching a suitable alternative led us toglobal-agent, which would allow us to configure the proxy in.envand have it automatically used for all outgoing requests.We therefore implement this, and remove the existing proxy support entirely as it is no longer required. Note that the package uses the environment variable
GLOBAL_AGENT_HTTP_PROXY-- the docs explain that some packages will automatically change their behaviour ifHTTP_PROXYis set, so a different env var is used to prevent this from happening.One issue we ran into was that Nock does not correctly intercept requests during unit testing after
GlobalAgent.bootstrap()has been called (regardless of whether or not a proxy is defined) -- we therefore check the environment is nottestbefore we call it.For manual testing, we recommend enabling the nginix proxy in
docker-compose.yml. Addingdns: 0.0.0.0to theappdefinition is a quick way of requiring all traffic to go via the proxy and therefore verify thatglobal-agentis working with both Rules Service and S3 requests.