proxy support: add support for proxy creds and no_proxy config

[MatMaul]

Currently the credentials to use the proxy are injected through a local patch in the deploy machine.
Would be nice to parse that from the https_proxy string instead.

Also we would need some no_proxy support, since we want to be able to both reach internal Tchap Sygnal (no proxy) and matrix.org Sygnal (proxy needed).

[anoadragon453]

Currently the credentials to use the proxy are injected through a local patch in the deploy machine.
Would be nice to parse that from the https_proxy string instead.

@MatMaul could you post the contents of the patch (without the credentials of course) so we could see what exactly you're changing?

Otherwise in terms of additional parsing, what we currently parse is done here:

synapse-dinsic/synapse/http/proxyagent.py

Lines 172 to 207 in 63e8ab5

	def _http_proxy_endpoint(proxy, reactor, **kwargs):
	"""Parses an http proxy setting and returns an endpoint for the proxy
	Args:
	proxy (bytes|None): the proxy setting
	reactor: reactor to be used to connect to the proxy
	kwargs: other args to be passed to HostnameEndpoint
	Returns:
	interfaces.IStreamClientEndpoint|None: endpoint to use to connect to the proxy,
	or None
	"""
	if proxy is None:
	return None
	# currently we only support hostname:port. Some apps also support
	# protocol://<host>[:port], which allows a way of requiring a TLS connection to the
	# proxy.
	host, port = parse_host_port(proxy, default_port=1080)
	return HostnameEndpoint(reactor, host, port, **kwargs)
	def parse_host_port(hostport, default_port=None):
	# could have sworn we had one of these somewhere else...
	if b":" in hostport:
	host, port = hostport.rsplit(b":", 1)
	try:
	port = int(port)
	return host, port
	except ValueError:
	# the thing after the : wasn't a valid port; presumably this is an
	# IPv6 address.
	pass
	return hostport, default_port

Also we would need some no_proxy support, since we want to be able to both reach internal Tchap Sygnal (no proxy) and matrix.org Sygnal (proxy needed).

NO_PROXY is an environment variables consisting of a comma-separated list of domains that the proxy should not be used for.

This would involve adding a conditional to using the proxy for (for right now) requests to the Identity Service. There's a limited number of places that Synapse attempts to contact an identity server. It usually makes use of SimpleHttpClient to do so. However, I'm not currently sure how difficult it will be to turn the proxy on or off for different domains.

[richvdh]

related: matrix-org/synapse#9000

[anoadragon453]

The NO_PROXY side of this is being handled by this community PR: matrix-org/synapse#9372

[anoadragon453]

The above PR has been merged 🎉

@MatMaul Still waiting on that patch for the proxy creds so we can upstream it (with some changes possibly)? 🙂

[MatMaul]

diff -u -r synapse-dinsic-dinsic12_2020-04-28/synapse/http/connectproxyclient.py synapse-dinsic-dinsic12_2020-04-28_hotfix1/synapse/http/connectproxyclient.py
--- synapse-dinsic-dinsic12_2020-04-28/synapse/http/connectproxyclient.py       2020-04-28 12:53:21.000000000 +0200
+++ synapse-dinsic-dinsic12_2020-04-28_hotfix1/synapse/http/connectproxyclient.py       2020-04-28 16:16:06.056143846 +0200
@@ -180,6 +180,7 @@
     def connectionMade(self):
         logger.debug("Connected to proxy, sending CONNECT")
         self.sendCommand(b"CONNECT", b"%s:%d" % (self.host, self.port))
+        self.sendHeader(b"Proxy-Authorization", b"basic secret==")
         self.endHeaders()

     def handleStatus(self, version, status, message):

[anoadragon453]

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization#directives describes the method of transforming username/password information into a base64 encoded string.

It's just base64(username:password).

[anoadragon453]

NO_PROXY support has been ported from mainline to dinsic in #93.

[anoadragon453]

Opened a PR on mainline to address proxy credentials: matrix-org/synapse#9657