HTTPS proxy connection not initialized if first request is chunked

[bpitman]

Using code below, will produce 403 error without https_proxy:

$ ./test.py
<Response [403]>

But ssl error with https_proxy (because CONNECT request is never sent):

$ https_proxy=http://10.32.2.100:1080/ ./test.py
Traceback (most recent call last):
  File "./test.py", line 11, in <module>
    res = session.put("https://github.com/kennethreitz", data=StringIO.StringIO(""))
  File "/opt/pepperdata/native/lib/python2.7/site-packages/requests/sessions.py", line 533, in put
    return self.request('PUT', url, data=data, **kwargs)
  File "/opt/pepperdata/native/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/pepperdata/native/lib/python2.7/site-packages/requests/sessions.py", line 596, in send
    r = adapter.send(request, **kwargs)
  File "/opt/pepperdata/native/lib/python2.7/site-packages/requests/adapters.py", line 492, in send
    raise ConnectionError(err, request=request)
ConnectionError: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:590)

For now, I'm patching my local requests/adaptor.py to init proxy when first conn use is with chunked request:

>                 if hasattr(conn, 'proxy'):
>                     if conn.proxy is not None and not getattr(low_conn, 'sock', None):
>                         conn._prepare_proxy(low_conn)

## test.py ########################################################
import requests
import traceback
import StringIO
import sys

try:
   session = requests.session()
   res = session.put("https://github.com/kennethreitz", data=StringIO.StringIO(""))
   sys.stdout.write(str(res) + "\n")
except Exception:
    traceback.print_exc()

session.close()

[Lukasa]

Can you provide some more information please? What requests version are you using. What proxy? What Python version?

[ssuchter]

I work with Brent. I've seen this issue on both Linux and OSX. On Linux I've seen it with python 2.7.11 and requests 2.11.1. On OSX (my laptop) I've seen it with python 2.7.11+requests 2.11.1, python 2.7.11+requests 2.13.0 and python 3.5.1+requests 2.13.0. I believe Brent used Apache as a proxy, but I'm not positive.

I think, though, that given the patch, (btw, it's around like 432 of requests/adapters.py), it's probably not relevant what the proxy software was.

[bpitman]

I'm using apache2. Config below (don't do this unless server is secure). Like Sean said, though, I don't think it's relevant. Packet captures clearly show that CONNECT request isn't sent.

add to ports.conf

Listen 1080

add to a file in sites-enabled

<VirtualHost *:1080>
ProxyRequests On
ProxyVia On

[Lukasa]

Yeah, I suspect this is a real problem. I think this can be resolved by writing a patch to use the urlopen method for chunked uploads with the parameter chunked=True in HTTPAdapter.send. I'd welcome a patch from anyone who wants to make it.

[EricLin89]

Found this thread when searching for the "unknonw protocol" problem.
In fact, there's another bug in the "chunked" branch, the timeout parameter is not used in this branch, turns out the request will hang forever if it's a "chunked" request.
Another similar bug was reported (#2336) in 2014 but seems only fixed in the branch that uses urlopen().
It's not a good idea to call the private method of another library(urllib3), maybe it's better to try use urlopen() in both branches.

[ynouri]

(as commented on #4179)

I was experiencing the same issue (chunked encoding through an HTTPS proxy tunnel resulting in an SSL unknown protocol error) with requests 2.22.0.

I upgraded to a9ee0ee and the issue disappeared 👍 . After spending some time analyzing the issue I'm confident that the fix came from #5128.

Since as of Jan 10, 2020 the fix is not released I used this command to upgrade requests to head of master branch: pip install git+https://github.com/psf/requests.git --upgrade