Thrift Connection - Open connection issue

[lukapor]

When I switch to client version 1.4.x (before I was using 1.3.0) have some serius issue with thrift connection. Here I paste stack trace. With 1.3.0 I didn't have no connection issues. Both clients was tested on ES 1.4.4 with thrift plugin version 2.4.1

2015-02-20T07:34:19.86Z Failed after retrying 2 times: 'POST _bulk?consist
ShortMessage: Failed after retrying 2 times: 'POST _bulk?consistency=all'.
InnerException: TTransportException, InnerMessage: Failed to connect, InnerStackTrace: at Elasticsearch.Net.Connection.Thrift.Transport.TSocket.Open()
at Elasticsearch.Net.Connection.Thrift.Transport.TBufferedTransport.Open()
at Elasticsearch.Net.Connection.Thrift.ThriftConnection.Execute(RestRequest restRequest, Object requestConfiguration)
InnerException: TTransportException, InnerMessage: Failed to connect, InnerStackTrace: at Elasticsearch.Net.Connection.Thrift.Transport.TSocket.Open()
at Elasticsearch.Net.Connection.Thrift.Transport.TBufferedTransport.Open()
at Elasticsearch.Net.Connection.Thrift.ThriftConnection.Execute(RestRequest restRequest, Object requestConfiguration)
InnerException: TTransportException, InnerMessage: Failed to connect, InnerStackTrace: at Elasticsearch.Net.Connection.Thrift.Transport.TSocket.Open()
at Elasticsearch.Net.Connection.Thrift.Transport.TBufferedTransport.Open()
at Elasticsearch.Net.Connection.Thrift.ThriftConnection.Execute(RestRequest restRequest, Object requestConfiguration)
Elasticsearch.Net.Exceptions.MaxRetryException: Failed after retrying 2 times: 'POST _bulk?consistency=all'.
InnerException: TTransportException, InnerMessage: Failed to connect, InnerStackTrace: at Elasticsearch.Net.Connection.Thrift.Transport.TSocket.Open()
at Elasticsearch.Net.Connection.Thrift.Transport.TBufferedTransport.Open()
at Elasticsearch.Net.Connection.Thrift.ThriftConnection.Execute(RestRequest restRequest, Object requestConfiguration)
InnerException: TTransportException, InnerMessage: Failed to connect, InnerStackTrace: at Elasticsearch.Net.Connection.Thrift.Transport.TSocket.Open()
at Elasticsearch.Net.Connection.Thrift.Transport.TBufferedTransport.Open()
at Elasticsearch.Net.Connection.Thrift.ThriftConnection.Execute(RestRequest restRequest, Object requestConfiguration)
InnerException: TTransportException, InnerMessage: Failed to connect, InnerStackTrace: at Elasticsearch.Net.Connection.Thrift.Transport.TSocket.Open()
at Elasticsearch.Net.Connection.Thrift.Transport.TBufferedTransport.Open()
at Elasticsearch.Net.Connection.Thrift.ThriftConnection.Execute(RestRequest restRequest, Object requestConfiguration) ---> System.AggregateException: One or more errors occurred. ---> Elasticsearch.Net.Connection.Thrift.Transport.TTransportException: Failed to connect
at Elasticsearch.Net.Connection.Thrift.Transport.TSocket.Open()
at Elasticsearch.Net.Connection.Thrift.Transport.TBufferedTransport.Open()
at Elasticsearch.Net.Connection.Thrift.ThriftConnection.Execute(RestRequest restRequest, Object requestConfiguration)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at Elasticsearch.Net.Connection.RequestHandlers.RequestHandlerBase.ThrowMaxRetryExceptionWhenNeeded[T](TransportRequestState1 requestState, Int32 maxRetries) at Elasticsearch.Net.Connection.RequestHandlers.RequestHandler.RetryRequest[T](TransportRequestState1 requestState)
at Elasticsearch.Net.Connection.RequestHandlers.RequestHandler.DoRequest[T](TransportRequestState1 requestState) at Elasticsearch.Net.Connection.RequestHandlers.RequestHandler.RetryRequest[T](TransportRequestState1 requestState)
at Elasticsearch.Net.Connection.RequestHandlers.RequestHandler.DoRequest[T](TransportRequestState1 requestState) at Elasticsearch.Net.Connection.RequestHandlers.RequestHandler.RetryRequest[T](TransportRequestState1 requestState)
at Elasticsearch.Net.Connection.RequestHandlers.RequestHandler.DoRequest[T](TransportRequestState1 requestState) at Elasticsearch.Net.Connection.RequestHandlers.RequestHandler.Request[T](TransportRequestState1 requestState, Object data)
at Elasticsearch.Net.Connection.Transport.DoRequest[T](String method, String path, Object data, IRequestParameters requestParameters)
at Elasticsearch.Net.ElasticsearchClient.DoRequest[T](String method, String path, Object data, IRequestParameters requestParameters)
at Elasticsearch.Net.ElasticsearchClient.Bulk[T](Object body, Func2 requestParameters) at Nest.RawDispatch.BulkDispatch[T](ElasticsearchPathInfo1 pathInfo, Object body)
at Nest.ElasticClient.b__220(ElasticsearchPathInfo1 p, BulkDescriptor d) at Nest.ElasticClient.Dispatch[D,Q,R](D descriptor, Func3 dispatch)
at Nest.ElasticClient.Dispatch[D,Q,R](Func2 selector, Func3 dispatch)
at Nest.ElasticClient.Bulk(Func2 bulkSelector) at Marg.BusinessConnect.Search.ElasticSearchClient.IndexCompositeData(IEnumerable1 compositeSearchData)
at Marg.BusinessConnect.Search.BuildIndex.BuildIndex.PopulateIndexData()
at Marg.BusinessConnect.Search.BuildIndex.BuildIndexBase.ASyncIndexingThreadExec()

[Mpdreamz]

Hi @lukapor we did introduce some new changes to make the thrift connection play nicely with the SniffingConnectionPool and pretty much spend an entire day continuously bulk indexing and deleting using the updated ThriftConnection to see it would blow up. Very bummed to hear you still ran into a newly introduced problem, we tried hard to avoid that.

Can you give us some more information to your usage? Do you use connectionpooling, maxretry setting, does it happen straight away, often, sometimes?

[lukapor]

Hi martin,

Currently I am so busy with our release, but some strange exceptions raised in our development environment in also in test environment when we upgraded the version. Maybe is that because we also update elasticsearch version.

Btw we use SingleNodeConnectionPool, on cluster environment we have virtual ip (load balancer, test environment haproxy, production amazon lb) and maxretris set to 2.

So when rush passed (never :)), I will write same test and look the changes.

[lukapor]

ConnectTimeout is maybe answer.

In previous version Ping Timeout was Connection timeout (ping timeout is 1 or 2 s), now connect timeout default value is 200 ms. For that reason maybe connection falls. I will set up connect timeout correctly.

[lukapor]

That's also doesn't help.
I saw that in TFrameTransport.cs we have TSocketV2 socket propertie where open and close are implemented I think ok (both BeginConnection and EndConnection is called). In TStreamTransport we call only BeginConnect, I think that this is problem. Why we don't use same (object TsockectV2) code for opening connection. I use TStreamTransport (binary protokol).

I will try this and next week I can tell you if we don't have exception any more.