client side proxy when using ManagedChannelBuilder.forTarget()

[anusha-hv]

Please answer these questions before submitting your issue.

What version of gRPC are you using?

I have been using 1.16.1 version. Now upgraded to 1.20.0 version.
The machine where am running the client side ie, ManagedChannelBuilder.forTarget(serveraddress) call is used is proxy enabled.
I get an error saying "Channel closed while trying to CONNECT through proxy"

I can read the proxy address from environment variable http_proxy.
For example : http_proxy="10.9.3.2.9090" and my target Address = "10.45.6.1:7070" (ie grpc server is running on another machine)
How do i make the grpc allow this proxy and send request to the target Address.

What did you expect to see?

I need a sample example. I see a proxydetector that is available but not sure about the usage of it since the method takes a single address.

I am new to grpc. Need help.

[ejona86]

Your setup previously worked with 1.16 and broke when upgraded to 1.20? grpc-java does not observe the http_proxy environment variable. It observes Java's proxy settings, which generally implies you should specify -Dhttps.proxyHost=example.com and -Dhttps.proxyPort=8080 on the command line to java (before any -jar or class name to run).

If you want to override Java's default behavior for proxy detection, call ProxySelector.setDefault(). If you need to override the proxy for a single channel, like within a library, that's when to use ProxyDetector.

[anusha-hv]

@ejona86 No issues with the upgrade of grpc as such. Let me explain the problem in detail.
My product is a war which includes a GRPC client and is deployed on Tomcat . Since it is deployed on tomcat, i have made the proxy settings http.proxyHost=10.194.10.21 http.proxyPort=3128 in catalina.properties of tomcat.

My GRPC server is a jar that runs on the machine with proxy enabled.
Whenever a call goes from the grpc client (part of war deployed on tomcat) to grpc server , I get an error on the client side saying ""Channel closed while trying to CONNECT through proxy".
Unable to resolve this. Need help.

[ejona86]

Ah, okay. Then I think the only problem is you need to set https.proxyHost/proxyPort instead of/in addition to http. gRPC only supports the HTTP Connect proxy protocol, which is what is used with https, so independent of whether you are using TLS or plaintext in gRPC you need to set the https configuration.

[anusha-hv]

GrpcUsage.docx

@ejona86 I already have the https configuration too along with http configuration in catalina.properties of tomcat. I have both http.proxyHost=10.194.10.21 http.proxyPort=3128 and https.proxyHost=10.194.10.21 https.proxyPort=3128 in catalina.properties of tomcat (grpc client) . I am using plaintext in grpc.

I get the same error "Channel closed while trying to CONNECT through proxy". I have also attached the pictorial representation of the problem in the document.

[ejona86]

@anusha-hv, sorry. It should have been clear to me that the configuration was working, since "Channel closed while trying to CONNECT through proxy" implies that gRPC tried to use the proxy.

The proxy closed the connection without responding with an error. So it is unclear what is wrong. It may be that your proxy doesn't support CONNECT or that it is missing your server from some whitelist of servers to forward to. For the first, you could check by using HTTPS through the proxy (not grpc, just some "plain" https client).

I'd suggest looking over the proxy server's configuration. If you can check the proxy's logs or enable more debugging logs, you might be able to discover what's wrong.

[anusha-hv]

@ejona86 one thing i noticed. When i use localhost:port_number on client side, even with client's side proxy enabled, i don't see this issue.I have both grpc client(war) and grpc server(jar) running on the same machine. Everything works fine when i use localhost.
Instead of localhost, if i use the hostname or the machine IP , I get the same error "Channel closed while trying to CONNECT through proxy". I have both my grpc client and server on the same machine.
Why does it behave differently when we use hostname or ipaddress ?

[ejona86]

Proxies commonly have whitelists for allowed hosts. There may be another reason. But again, you will have to debug your proxy; it's not something we can really help with.

If you provide a tcpdump (or the plain-text output, since it is HTTP/1) of the communication from grpc to the proxy, I could look at it to confirm that gRPC is behaving as expected and see if the proxy provides any hint as to what is wrong. But this would be more of a double-check.

[anusha-hv]

Hi @ejona86, I tried a sample example with grpc-java client and server on the proxy enabled machines. Both are jars. I did not provide the proxy flags as well while running the jars. It all works fine. My guess is proxy settings control http 1.1 traffic and is not controlling the http 2 traffic (gRPC uses http 2). So gRPC client is able to talk to server on proxy enabled machines.

when the gRPC client is part of an existing war deployed on tomcat( tomcat version 8.5 is used), it complains telling "Channel closed while trying to CONNECT through proxy" even after having the the proxy flags defined in tomcat. Ideally it should not be bothered with gRPC client server communication.
Right now I have included gRPC client code as part of my main code. WIll it help if i use gRPC client as JAR in the main code?
will provide the dump in sometime.

[ejona86]

Ideally it should not be bothered with gRPC client server communication.

Oh, so you want to disable the proxy just for gRPC? If you can configure your code specifically for this environment, you can disable the proxy in gRPC with ManagedChannelBuilder.proxyDetector():

managedChannelBuilder.proxyDetector(new ProxyDetector() {
  @Override ProxiedSocketAddress proxyFor(SocketAddress addr) {
    return null;
  }
});

[anusha-hv]

Hi @ejona86
Like you suggested, I am attaching the dump captured via wireshark . Let me know if there is anything else i can try.
dumptobeshared2.zip

[ejona86]

@anusha-hv, it was as I suspected: the proxy is just hanging up. That means I just have to guess what could be going wrong. It is strange that the proxy doesn't respond with anything to the CONNECT request. And then 10 seconds after the CONNECT request the client sends... something more. I don't recognize what it sends. But at that point is when the proxy disconnects.

There is one successful CONNECT handshake in the dump, from something non-gRPC. It does show a 200 OK response from the proxy with just the HTTP CONNECT request. It takes .6 seconds for the reply.

So the riddle is why is there a 10 second gap. Could it be that the proxy can't connect to the destination machine?

[anusha-hv]

@ejona86 , I have both my grpc client(deployed on tomcat as war) and my grpc server (as jar) on the same machine. So am not sure when you say proxy can't connect to the destination machine.

I missed out sending the logs. Whenever the proxy is enabled for the grpc client, i can see this in the logs.

io.grpc.StatusRuntimeException: UNAVAILABLE: Channel closed while trying to CONNECT through proxy
        at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233)
        at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214)
        at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139)
        at com.dcc.protobuf.grpc.ServiceGrpc$ServiceBlockingStub.getdata(ServiceGrpc.java:446)
 ....
....
.....
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:189)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:800)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1038)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:897)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:84)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:158)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:74)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215)
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:200)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:836)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1747)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Unknown Source)

Let me know if there is anything else i can try.

[ejona86]

All signs point to "debug the proxy." The client knows nothing. There's no more help we can provide, unless you want to avoid using the proxy with gRPC.

[ejona86]

Closing since there's not much more we can help with. If there's some more information provided that makes it seem like it's a gRPC bug instead of a proxy configuration issue, comment and it can be reopened.