RestartableHttpClient shuts down connectionManager making reconnects impossible

[scubasau]

Version: latest 2.0.3-SNAPSHOT (also tested against 2.0.2).
Client is configured as:

return new ClientBuilder()
          .name("CHosebird")
          .hosts("https://stream.twitter.com")
          .authentication(auth) // OAuth1
          .gzipEnabled(gzipEnabled) // true
          .endpoint(new StatusesSampleEndpoint())
          .processor(new StringDelimitedProcessor(messageQueue))
          .eventMessageQueue(eventQueue)
          .build();

Example log showing disconnect and reconnect failure:

2014-05-13 17:20:35,543 o-thread-0 INFO  c.t.h.h.ClientBase        - CHosebird Disconnected during processing - will reconnect
2014-05-13 17:20:35,543 o-thread-0 INFO  c.t.h.h.ClientBase        - CHosebird Done processing, preparing to close connection
2014-05-13 17:20:35,543 event      WARN  t.s.TwitterHosebirdClient - DISCONNECTED : Unexpected end of ZLIB input stream : java.io.EOFException: Unexpected end of ZLIB input stream
2014-05-13 17:20:35,547 o-thread-0 INFO  c.t.h.h.ClientBase        - CHosebird Establishing a connection
2014-05-13 17:20:35,547 event      INFO  t.s.TwitterHosebirdClient - CONNECTION_ATTEMPT : GET https://stream.twitter.com/1.1/statuses/sample.json?delimited=length&stall_warnings=true HTTP/1.1
2014-05-13 17:20:35,555 o-thread-0 ERROR c.t.h.h.ClientBase        - CHosebird Unknown exception while establishing connection to https://stream.twitter.com/1.1/statuses/sample.json?delimited=len
gth&stall_warnings=true
java.lang.IllegalStateException: Connection pool shut down
        at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:167) ~[chug-jar-with-dependencies.jar:na]
        at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:199) ~[chug-jar-with-dependencies.jar:na]
        at org.apache.http.impl.conn.PoolingClientConnectionManager.requestConnection(PoolingClientConnectionManager.java:188) ~[chug-jar-with-dependencies.jar:na]
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:447) ~[chug-jar-with-dependencies.jar:na]
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) ~[chug-jar-with-dependencies.jar:na]
        at org.apache.http.impl.client.DecompressingHttpClient.execute(DecompressingHttpClient.java:137) ~[chug-jar-with-dependencies.jar:na]
        at org.apache.http.impl.client.DecompressingHttpClient.execute(DecompressingHttpClient.java:108) ~[chug-jar-with-dependencies.jar:na]
        at com.twitter.hbc.httpclient.RestartableHttpClient.execute(RestartableHttpClient.java:86) ~[chug-jar-with-dependencies.jar:na]
        at com.twitter.hbc.httpclient.Connection.connect(Connection.java:44) ~[chug-jar-with-dependencies.jar:na]
        at com.twitter.hbc.httpclient.ClientBase.establishConnection(ClientBase.java:179) [chug-jar-with-dependencies.jar:na]
        at com.twitter.hbc.httpclient.ClientBase.run(ClientBase.java:142) [chug-jar-with-dependencies.jar:na]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_17]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_17]
        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_17]
2014-05-13 17:20:35,555 o-thread-0 INFO  c.t.h.h.ClientBase        - CHosebird exit event - Connection pool shut down
2014-05-13 17:20:35,555 event      WARN  t.s.TwitterHosebirdClient - STOPPED_BY_ERROR : Connection pool shut down : java.lang.IllegalStateException: Connection pool shut down
2014-05-13 17:20:35,555 o-thread-0 WARN  c.t.h.h.ClientBase        - CHosebird failed to establish connection properly
2014-05-13 17:20:35,556 event      INFO  t.s.TwitterHosebirdClient - CONNECTION_ERROR : Failed to establish connection properly
2014-05-13 17:20:35,556 o-thread-0 INFO  c.t.h.h.ClientBase        - CHosebird Done processing, preparing to close connection
2014-05-13 17:20:35,556 o-thread-0 INFO  c.t.h.h.ClientBase        - CHosebird Shutting down httpclient connection manager

To duplicate:

1. Start a stream with a gzip-enabled, default ConnectionManager Client
2. Once connected and processing, kill the underlying TCP connection (e.g., sudo tcpkill -i eth0 host 199.16.156.xxx)
3. Observe reconnection failure

The cause is RestartableHttpClient#restart which invokes shutdown() on the 'old' connectionManager. Since there is only a single connectionManager, shutting this down prevents any future connection retrieval.

I have a dirty hack in place but am not comfortable with a pull request for this since I am unclear as to the purpose RestartableHttpClient serves. Additionally the hack relies on all ClientConnectionManagers sharing a constructor signature which is not valid for custom implementations.

[kevinoliver]

Good catch, thanks. We're looking into it the fix. Hope to have it out soon.

[kevinoliver]

@scubasau can you see if pull #119 fixed this for you?

[scubasau]

@kevinoliver confirmed #119 fixes this issue. Thanks! (and thanks to @paradigmsort)