[ResponseOps] POC: switch to use hpagent for proxy#131971
[ResponseOps] POC: switch to use hpagent for proxy#131971pmuellr wants to merge 2 commits intoelastic:mainfrom
Conversation
|
Note that there appears to be a ready-to-run docker image for Squid, to use for testing a presumably http proxy: https://hub.docker.com/r/minimum2scp/squid Quick test seems to show it works: $ docker run -i --rm -p 3128:3128 minimum2scp/squid
WARNING: The requested image's platform (linux/amd64) does not match the detected
host platform (linux/arm64/v8) and no specific platform was requestedNote that it runs fine on my Mac M1 box despite the warning, though does not respond to ctrl-c to exit. So running with -i is a little pointless. Instead, find the container with $ http_proxy=http://127.0.0.1:3128 curl -v http://example.com/
* Uses proxy env variable http_proxy == 'http://127.0.0.1:3128'
* Trying 127.0.0.1:3128...
* Connected to 127.0.0.1 (127.0.0.1) port 3128 (#0)
> GET http://example.com/ HTTP/1.1
> Host: example.com
> User-Agent: curl/7.79.1
> Accept: */*
> Proxy-Connection: Keep-Alive
> ... |
pmuellr
force-pushed
the
actions/proxy-tests-102935
branch
from
780aae7 to
5e4cfd2
Compare
💔 Build FailedFailed CI StepsTest Failures
Metrics [docs]Unknown metric groupsESLint disabled in files
ESLint disabled line counts
Total ESLint disabled count
History
To update your PR or re-run it, just comment with: |
|
As of commit 5e4cfd2, there are a number of new proxy tests added; testing http vs https for the target and proxy, and using auth or not. The proxy code this is using does not really support https proxy's, unless the CA is somehow automatically trusted by node, which is unlikely to be the case. See delvedor/hpagent#69 for more details. I think we'd want a way to use rejectUnauthorized: false, or equivalent, in the agent, before we adopt this. The tests are kind of fudged as well. It appears we may not be setting all the options correctly for private certs, so some of the tests are setting options to do that, that are known to work. Hoping this can be resolved with some additional tests in the Now, the telling moment. I took this new test, and ran it with the old agents. Results, with all of the newly added tests in a new section test results
most of them failed! :-( Some of the failures could be because of the proxy implementation added in these tests. The tests did all pass with hpagent, but perhaps there was something slightly off. Some known failure scenarios in the field (https target with http proxy) failed here also, so there's a least a little correlation. The table removed tests that were expected to be errors, since that's non-interesting, and it turns out the two agents handle errors differently. hpagent throws errors, where the old agent code returns http status responses when possible. It doesn't matter too much, an error will end up getting thrown by the connector run, regardless. And those are just kinda of noisy with all the other combinations. Tests using an https proxy were not run, as they are known to be unsupported at the moment in hpagent (for what we need - support for self-signed certs), but they also all failed with the older agents. raw test results using existing agent |
|
Brandon suggested moving the tests to a jest integration folder, since they are more integrate-y than unit-y. It appears what we'll need to do is add a new |
|
I re-found this SO post regarding the "old" http-proxy package we're currently using for integration tests. It doesn't support HTTP CONNECT, just simple http forwarding, and we likely should replace it with the https://stackoverflow.com/questions/8165570/https-proxy-server-in-node-js |
pmuellr
changed the title
3 participants
resolves: #102935
resolves: #125837
Todo
axiosdependency (0.21.1→0.27.2). #111655 to be merged (axios upgrade)Summary
Currently just trying to build some tests for testing different proxy combinations. To start with, combinations of the following:
We'll probably want more, for example, testing https targets with custom certs - it's not clear if proxy's having custom certs is a thing that's even possible (due to the way the proxy CONNECT protocol works). So that makes 16 tests.
At this point, the tests I have basically hang, so hit jest timeouts. Debugging these is awful, especially in jest, so I've created a new stand-alone proxy in
x-pack/plugins/actions/server/manual_tests/forward_proxy_ng.js- this one can create multiple proxies in one invocation, with different options for each - port, protocol, auth.Here are some pre-configured webhook connectors that will make it easy to "test" the proxies from Kibana, outside of jest; the
webhook-https-esis a bit of a hack, I think I'll change this to a_searchinstead - we want something we canPOSTwith that we can send a simple body to for the test. More will be required, for auth and such, if we want to reuse these for other manual tests. But hoping we can automate it all.pre-configured webhook connectors
At this point, I'd like to wait for axios update PR to merge, and likely we will switch from using
http-proxy-agentandhttps-proxy-agent, to usinghpagentinstead, which I've had better luck with than the packages I'm replacing it with.Checklist
Delete any items that are not applicable to this PR.
Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.
When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:
For maintainers