HTTPS proxy is not supported

[mgaido91]

What kind of issue is this?

• Question. This issue tracker is not the place for questions. If you want to ask how to do
  something, or to understand why something isn't working the way you expect it to, use Stack
  Overflow. https://stackoverflow.com/questions/tagged/okhttp

• Bug report. If you’ve found a bug, spend the time to write a failing test. Bugs with tests
  get fixed. Here’s an example: https://gist.github.com/swankjesse/981fcae102f513eb13ed

• Feature Request. Start by telling us what problem you’re trying to solve. Often a solution
  already exists! Don’t send pull requests to implement new features without first getting our
  support. Sometimes we leave features out on purpose to keep the project small.

Hello, I tried to use okhttp to connect through an https proxy, but I wasn't able to. Here is a reproduction:

curl --proxy-insecure -x https://myproxy:3129 https://httpbin.org/get

works like a charm, instead:

OkHttpClient.Builder okHttpClientBuilder = new OkHttpClient().newBuilder();

final Integer proxyPort = 3129;
final Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxyHost, proxyPort));
okHttpClientBuilder.proxy(proxy);
        
okHttpClientBuilder.hostnameVerifier(new HostnameVerifier() {
            @Override
            public boolean verify(String s, SSLSession sslSession) {
                if (proxyHost.equalsIgnoreCase(s)) {
                    return true;
                }
                return OkHostnameVerifier.INSTANCE.verify(s, sslSession);
            }
});
        
okHttpClientBuilder.build().newCall(new Request.Builder().url("https://httpbin.org/get").build()).execute();

fails with the following stacktrace:

Exception in thread "main" java.io.IOException: unexpected end of stream on null
	at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:205)
	at okhttp3.internal.connection.RealConnection.createTunnel(RealConnection.java:323)
	at okhttp3.internal.connection.RealConnection.connectTunnel(RealConnection.java:197)
	at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:145)
	at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:192)
	at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:121)
	at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:100)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
	at okhttp3.RealCall.execute(RealCall.java:69)
	at Test.main(Test.java:34)
Caused by: java.io.EOFException: \n not found: limit=0 content=…
	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:226)
	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:210)
	at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
	... 20 more

I digged a bit into the code and I think that the issue is that here, if the proxy is HTTP, it is used socketFactory, while there should be an additional check whether the proxy is running HTTPS or not (and if so, sslSocketFactory should be used instead).

[swankjesse]

HTTPS proxies are not supported. I don't think this is a thing. Got a spec for this?

[mgaido91]

Yes, there are companies using HTTPS proxies. In these environments, this is quite a vital requirement.

[swankjesse]

Wow, I've never heard of that. Got a spec? Typically we just follow RFC 2817.
https://tools.ietf.org/html/rfc2817

[mgaido91]

Honestly I don't know if there is a spec and I am not sure where to find it. Here you can find how to set up squid for doing this: http://www.squid-cache.org/Doc/config/https_port/ (of course there are many other proxies supporting this, it is just an example).

[swankjesse]

@mgaido91 in that example Squid is not acting as a proxy from the client’s perspective. Try doing the same thing but without a proxy configured.

[zi6xuan]

Here what's happening:
HTTP: the client send directly the full request to the proxy, with the proxy-auth headers. The proxy is in charge to forward to server.
HTTPS: the client want to send a request to a server, encrypted with the server public key, passing through an http proxy. So
before making the request itself, the client have to get the server public key (i.e. make SSL handshake, i.e. etablish a SSL/TLS tunnel to the server)
the HTTPS request will be encrypted so the proxy won't have access to the request headers (in case of a normal proxy)
What happens:
The client send a HTTP CONNECT to the proxy, indicating the hostname of the server to connect with and also probably the proxy-auth headers
The proxy connect the client with the server (here with an internal https server instead of remote one, to be able to decrypt and modify)
Client and server (internal one here) makes SSL handshake
Client send the HTTPS request to the server, without proxy-auth headers because the proxy is not supposed to have access to it

   private Request createTunnelRequest() {
     return new Request.Builder()
         .url(route.address().url())
         .header("Host", Util.hostHeader(route.address().url(), true))
         .header("Proxy-Connection", "Keep-Alive") // For HTTP/1.0 proxies like Squid.
         .header("User-Agent", Version.userAgent())
         .build();
   }

OKHttp3.9.1,I watch the source code and debug it,I found the createTunnelRequest not add the Proxy-Authorization,so The https request is can not be connected
@swankjesse

    private Request createTunnelRequest(Call call) {
         Request.Builder builder = new Request.Builder()
                 .url(route.address().url())
                .header("Host", Util.hostHeader(route.address().url(), true))
                 .header("Proxy-Connection", "Keep-Alive") // For HTTP/1.0 proxies like Squid.
                 .header("User-Agent", Version.userAgent());
         String value = call.request().header("Proxy-Authorization");
         if (value != null) {
             builder.header("Proxy-Authorization", value);
         }
         return builder.build();
     }

it's worked
@mgaido91 i think it's a bug

[mgaido91]

@swankjesse what do you mean that it is not acting like a proxy? It is and this is confirmed by the curl command working fine.

[swankjesse]

Can you find the spec that curl is following? I don't doubt that curl can do something here but we don't use curl as a spec and this behavior is new to me.

What do Chrome and Firefox do?

[swankjesse]

https://curl.haxx.se/bug/?i=1127

[mgaido91]

@swankjesse both Chrome and Firefox support it. Here I found an article about Chromium support (https://www.chromium.org/developers/design-documents/secure-web-proxy). Actually it is often referred as SSL proxy (Firefox terminology) or Secure Web Proxy (OSX terminology).

I have not been able to find any specification and I don't think that there can be, since there is nothing to specify: it is simply an implementation of a HTTP proxy using SSL.

You already found the curl PR for it, so I am not sending you the commit which implements it in curl. As you can see with a quick search with the keywords "SSL Proxy" the support is widely spread.

[mgaido91]

@swankjesse anyway here there is a description of a proxy authentication using a SSL proxy: https://tools.ietf.org/html/draft-loreto-httpbis-trusted-proxy20-01#section-3. This shows that IETF considers this use case as valid.

[swankjesse]

Got it. Seems like a thing we can do. How do we know whether the caller wants TLS with their proxy connection?

At least for now the easiest way to get this working for yourself is to provide your own socket factory. That will have to do the TLS handshake when connecting the proxy.

[mgaido91]

I think that the easiest way is to slightly change the .proxy method accepting an enum telling the kind of proxy (HTTP, HTTPS, SOCKS or NONE) and the InetSocketAddress of the proxy. In this way when you create the socket you know how to create it. What do you think?

[swankjesse]

We’ve got something similar from Java but it also doesn't know about TLS to the proxy.
https://docs.oracle.com/javase/8/docs/api/java/net/Proxy.Type.html

In the interim I recommend doing it yourself with a socket factory.

[mgaido91]

Thanks for the suggestion. Do you mean passing a socket factory to the OkHttpClient.Builder? May I kindly ask you to show me a fast example? Thank you very much for your answers and your time.

[swankjesse]

This might lead you in the right direction:

    OkHttpClient client = new OkHttpClient.Builder()
        .socketFactory(SSLSocketFactory.getDefault())
        .build();

In practice there’s a bunch more work due to features like ALPN and SNI, and having to verify that the proxy server you connect to is the one who’s certificate chain is trusted.

[yeyb]

I find the same problem now ,okhttp Proxy.Type.HTTP don't support https request ,http request is fine. is any solution for this problem?

[yeyb]

anybody can help me ?okhttp https httpProxy

[mgaido91]

@yeyb you can follow @swankjesse 's suggestion. You can find my implementation of his suggestion in nifi here: apache/nifi@37271e8

[yeyb]

okhttp Proxy.Type.HTTP don't support https request ? follow is my code,https request don't work

sslSocketFactory = HttpsUtils.setCertificates(new Buffer().writeUtf8(HttpsCert.NEW_CER_IBU).inputStream());
OkHttpClient okHttpClient = new OkHttpClient.Builder()
.connectTimeout(10000L, TimeUnit.MILLISECONDS)
.readTimeout(10000L, TimeUnit.MILLISECONDS)
.addInterceptor(new LoggerInterceptor("TAG", BuildConfig.BUILD_TYPE.equals("debug")))
///////////////////////////////////////////////////////////////////////////////////////////////
//添加代理
.addNetworkInterceptor(new MyNetworkInterceptor())
//add proxy
.proxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress("", )))
// .proxySelector(new MyProxySelector())
///////////////////////////////////////////////////////////////////////////////////////////////
//https证书
.hostnameVerifier(new HostnameVerifier() {
@OverRide
public boolean verify(String hostname, SSLSession session) {
return true;
}
})
.sslSocketFactory(sslSocketFactory)
///////////////////////////////////////////////////////////////////////////////////////////////
.build();
OkHttpUtils.initClient(okHttpClient);

[yeyb]

@mgaido91 how to call the method in your sample?

[mgaido91]

which method?

[yeyb]

when I set cer ,call which method?

[mgaido91]

@yeyb I don't understand your question anyway the answer to your problem is: if you want to support HTTPS proxy, use ProxyType.HTTP and set the socketFactory to a SSLSocketFactory

[yeyb]

@mgaido91 yes this is my problem

[yeyb]

I am android develper,and I don't konw how to set socketFactory to a SSLSocketFactory in your suggestion

[yeyb]

@mgaido91 I am a chinese ,thank you for your help!!!

[mgaido91]

@yeyb I am not sure which is the problem you are having, but the example by @swankjesse seems very clear to me, I don't know what else to say...

[yeyb]

@mgaido91 ,my problem is what you say.but as you say,if I want to support HTTPS proxy, use ProxyType.HTTP and set the socketFactory to a SSLSocketFactory , how I make a socketFactory ? my code is set the SSLSocketFactoryto a SSLSocketFactory

[yeyb]

@mgaido91 do I need to call okhttpClient.builder..proxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress("host", port)))?

[yeyb]

@mgaido91 follow code find a problem:
javax.net.ssl.SSLHandshakeException: SSL handshake aborted: ssl=0xd33088c0: I/O error during system call, Connection reset by peer

sslSocketFactory = HttpsUtils.setCertificates(new Buffer().writeUtf8(HttpsCert.NEW_CER_IBU).inputStream());
OkHttpClient okHttpClient = new OkHttpClient.Builder()
.connectTimeout(10000L, TimeUnit.MILLISECONDS)
.readTimeout(10000L, TimeUnit.MILLISECONDS)
.addNetworkInterceptor(new MyNetworkInterceptor())
.proxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress("172.16.1.207", 8876)))
///////////////////////////////////////////////////////////////////////////////////////////////
//https cer
.hostnameVerifier(new HostnameVerifier() {
@OverRide
public boolean verify(String hostname, SSLSession session) {
return true;
}
})
.sslSocketFactory(sslSocketFactory)
.socketFactory(sslSocketFactory)
///////////////////////////////////////////////////////////////////////////////////////////////
.build();

[yeyb]

public static SSLSocketFactory setCertificates(InputStream... certificates) {
try {
CertificateFactory certificateFactory = CertificateFactory.getInstance("X.509");
KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType());
keyStore.load(null);
int index = 0;
for (InputStream certificate : certificates) {
String certificateAlias = Integer.toString(index++);
keyStore.setCertificateEntry(certificateAlias, certificateFactory.generateCertificate(certificate));
try {
if (certificate != null) certificate.close();
} catch (IOException e) {

            }
        }
        TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
        trustManagerFactory.init(keyStore);

        SSLContext sslContext = SSLContext.getInstance("TLS");
        sslContext.init(null, trustManagerFactory.getTrustManagers(), new SecureRandom());
        return sslContext.getSocketFactory();
    } catch (Exception e) {
        e.printStackTrace();
    }
    return null;
}

[yeyb]

@swankjesse @mgaido91 see #3782
okhttp3.internal.connection#createTunnel
Response response = tunnelConnection.readResponseHeaders(false)
.request(tunnelRequest)
.build();
it return null.This problem can only be solved by modifying the source code？

[lordyavin]

Is this issue still valid? With version 4.9 I could connect to a HTTPS resource via proxy.

[Mahoney]

@lordyavin you can connect to an HTTPS origin resource via an HTTP proxy - which is the normal way to do it, the HTTP proxy becomes a dumb TCP/IP tunnel and the TLS handshake happens between the client and the origin server.

This issue is about connecting to both the origin server and the proxy server over HTTPS. So the data stream ends up double encrypted - you have an encrypted stream between the client and the proxy server, and that is tunnelling an encrypted stream between the client and the origin server.

It's a pretty unusual thing to want to do - all it really additionally protects is the CONNECT request from the client to the proxy, so it stops a man in the middle knowing which origin server it is you want to proxy to. The CONNECT request could include a Proxy-Authorization header which would be worth protecting.

[lordyavin]

@Mahoney thanks for the clarification

[yschimke]

Examples like #7748 show this is possible, but leaving open in case we add support cleanly.

[yschimke]

Is it valid to consider the spec Secure Web Proxy?

If so, does anyone have a known working test server? I'm happy to add custom certificates, but want a known working base.

It looks like Google cloud can enable this, but I'd like to know we agree what this means.

https://cloud.google.com/secure-web-proxy/docs/quickstart