sun/security/ec/SunEC.java:-2:in `initialize': java.lang.InternalError

[laoshancun]

sun/security/ec/SunEC.java:-2:in initialize': java.lang.InternalError from sun/security/ec/SunEC.java:49:inaccess$000'
from sun/security/ec/SunEC.java:61:in run' from sun/security/ec/SunEC.java:58:inrun'
from java/security/AccessController.java:-2:in doPrivileged' from sun/security/ec/SunEC.java:58:in'
from java/lang/reflect/Constructor.java:526:in newInstance' from java/lang/Class.java:383:innewInstance'
from sun/security/jca/ProviderConfig.java:221:in run' from sun/security/jca/ProviderConfig.java:206:inrun'
from java/security/AccessController.java:-2:in doPrivileged' from sun/security/jca/ProviderConfig.java:206:indoLoadProvider'
from sun/security/jca/ProviderConfig.java:187:in getProvider' from sun/security/jca/ProviderList.java:233:ingetProvider'
from sun/security/jca/ProviderList.java:331:in getService' from sun/security/jca/GetInstance.java:157:ingetInstance'
from javax/net/ssl/TrustManagerFactory.java:138:in getInstance' from org/apache/http/conn/ssl/SSLContextBuilder.java:104:inloadTrustMaterial'
from java/lang/reflect/Method.java:606:in invoke' from /opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/client.rb:622:insetup_trust_store'
from /opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/client.rb:602:in ssl_socket_factory_from_options' from /opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/client.rb:394:inpool_builder'
from /opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/client.rb:402:in pool' from /opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/client.rb:208:ininitialize'
from /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/http/manticore.rb:58:in build_client' from /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/http/manticore.rb:49:ininitialize'
from /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/client.rb:118:in initialize' from /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport.rb:26:innew'
from /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:129:in build_client' from /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:20:ininitialize'
from /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:44:in build' from /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch.rb:134:inbuild_client'
from /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:14:in register' from /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/output_delegator.rb:75:inregister'
from /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:181:in start_workers' from org/jruby/RubyArray.java:1613:ineach'
from /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:181:in start_workers' from /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:136:inrun'
from /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/agent.rb:473:in start_pipeline' from java/lang/Thread.java:745:inrun'

[suyograo]

@laoshancun can you give us JVM, OS and LS versions in addition to LS config?

[laoshancun]

@suyograo

uname -a
Linux localhost 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 00:31:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

java version
java version "1.7.0_111"
OpenJDK Runtime Environment (rhel-2.6.7.2.el6_8-x86_64 u111-b01)
OpenJDK 64-Bit Server VM (build 24.111-b01, mixed mode)

logstash-all-plugins-2.3.4-1.noarch.rpm

lsconfig
input {
kafka {
topic_id => 'log'
}
}
filter {

}
output {
elasticsearch {
hosts => ["192.168.0.150"] #ElasticSearch host, can be array.
ssl => false
}
}

[stvdilln]

I'm building in a docker container that has worked fine many times before. My stack trace starts off the same but then differs later on. If this helps: (On a container I built a month ago, java version was 1.78.0_101, logstash versions the same). I'm going to try to back off my java version and see what happens.

sh-4.1# java -version
java version "1.7.0_111"
OpenJDK Runtime Environment (rhel-2.6.7.2.el6_8-x86_64 u111-b01)
OpenJDK 64-Bit Server VM (build 24.111-b01, mixed mode)
sh-4.1# uname -a
Linux logstash 3.10.0-327.10.1.el7.x86_64 #1 SMP Tue Feb 16 17:03:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
sh-4.1# yum list | grep logstash
logstash.noarch 1.4.5-1_a2bacae @logstash-1.4
logstash-contrib.noarch 1.4.5-1_2bad350 @logstash-1.4

SunEC.java:-2:in initialize': java.lang.InternalError from SunEC.java:49:inaccess$000'
from SunEC.java:61:in run' from SunEC.java:58:inrun'
from AccessController.java:-2:in doPrivileged' from SunEC.java:58:in'
from NativeConstructorAccessorImpl.java:-2:in newInstance0' from NativeConstructorAccessorImpl.java:57:innewInstance'
from DelegatingConstructorAccessorImpl.java:45:in newInstance' from Constructor.java:526:innewInstance'
from Class.java:383:in newInstance' from ProviderConfig.java:221:inrun'
from ProviderConfig.java:206:in run' from AccessController.java:-2:indoPrivileged'
from ProviderConfig.java:206:in doLoadProvider' from ProviderConfig.java:187:ingetProvider'
from ProviderList.java:233:in getProvider' from ProviderList.java:331:ingetService'
from GetInstance.java:157:in getInstance' from SSLContext.java:156:ingetInstance'
from SecurityHelper.java:519:in getSSLContext' from SSLContext.java:431:increateDummySSLEngine'
from SSLContext.java:352:in matchedCiphers' from SSLContext.java:390:inset_ciphers'
from SSLContext$INVOKER$i$1$0$set_ciphers.gen:-1:in call' from JavaMethod.java:769:incall'
from RubyClass.java:568:in finvoke' from RubyBasicObject.java:1507:insend19'
from RubyBasicObject$INVOKER$i$send19.gen:-1:in call' from JavaMethod.java:367:incall'
from CachingCallSite.java:202:in call' from CallTwoArgNode.java:59:ininterpret'
from NewlineNode.java:105:in interpret' from ASTInterpreter.java:112:inINTERPRET_BLOCK'
from Interpreted19Block.java:206:in evalBlockBody' from Interpreted19Block.java:157:inyield'
from Block.java:142:in yield' from RubyHash.java:1354:invisit'
from RubyHash.java:648:in visitLimited' from RubyHash.java:634:invisitAll'
from RubyHash.java:1305:in iteratorVisitAll' from RubyHash.java:1350:ineach_pairCommon'
from RubyHash.java:1341:in each19' from RubyHash$INVOKER$i$0$0$each19.gen:-1:incall'
from CachingCallSite.java:316:in cacheAndCall' from CachingCallSite.java:145:incallBlock'
from CachingCallSite.java:154:in callIter' from CallNoArgBlockNode.java:64:ininterpret'
from NewlineNode.java:105:in interpret' from BlockNode.java:71:ininterpret'
from ASTInterpreter.java:74:in INTERPRET_METHOD' from InterpretedMethod.java:139:incall'
from DefaultMethod.java:187:in call' from CachingCallSite.java:306:incacheAndCall'
from CachingCallSite.java:136:in call' from CallNoArgNode.java:60:ininterpret'
from NewlineNode.java:105:in interpret' from BlockNode.java:71:ininterpret'
from ASTInterpreter.java:74:in INTERPRET_METHOD' from InterpretedMethod.java:139:incall'
from DefaultMethod.java:187:in `call'
"logstash.err" 156L, 6834C

sh-4.1# cat /etc/logstash/conf.d/*
input {
lumberjack {
port => 5010
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/dy-logstash-5010.service.consul.PublicCert.pem"
ssl_key => "/etc/pki/tls/private/dy-logstash-5010.service.consul.PrivateKey.pem"
}
syslog {
port => 5011
type => "syslog"
}

}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
elasticsearch { host => "dy-elasticsearch-9200.service.consul:9200"
protocol => "http"
}
#stdout { codec => rubydebug }
}

[stvdilln]

I backleveled java from 1.7.0_111 to 1.7.0_101 and things are working again. In my Dockerfile I just added a version to my yum install and other than that the two containers are exactly the same:

FROM centos:6.6
MAINTAINER Steve Dillon
-# Version 1.7.0.111 has an issue with logstash
RUN yum install -y java-1.7.0-openjdk-1.7.0.101-2.6.6.4.el6_8
COPY sync /

RUN /tmp/install-logstash.sh

-# 5010 Lumberjack (logstash-forwarder)
-# 5011 syslog
EXPOSE 5010 5011
CMD [ "/usr/local/bin/start.sh" ]

[laoshancun]

@stvdilln thanks.
I just use elasticsearch_java instead of elasticsearch,it works again.

# For detail structure of this file
# Set: https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html
input {
  kafka {
    topic_id => 'log' 
  }
}
filter {
#  Only matched data are send to output.
    grok {
        match => ["message", "%{IP:client}\|%{HTTPDATE:timestamp}\|%{HOSTNAME:sitename}\|(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\|%{NUMBER:response:int}\|%{NUMBER:total_time:float}\|%{NUMBER:bytes:int}\|(?:%{HOSTPORT:upstream_addr}|-)\|(?:%{NUMBER:upstream_status:int}|-)\|(?:%{NUMBER:upstream_time:float}|-)\|(?:%{QS:referrer}|-)\|(?:%{HOSTPORT:forwarded_for}|-)\|(?:%{QS:agent}|-)"]
        remove_field => [ "message" ]
    }    
    geoip {
        source => "client"
        #database => "/root/GeoIPASNum.dat"
    }
}
output {
  # For detail config for elasticsearch as output, 
  # See: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html
  elasticsearch_java {
    #action => "index"          #The operation on ES
    hosts  => ["192.168.0.150:9300"]   #ElasticSearch host, can be array.
    cluster  => "logcenter"
    #index  => "logstash-%{+YYYYMMdd}"         #The index to write data to.
    network_host => "192.168.0.21"
    node_name  => "logstash"
    protocol   => "transport"
    sniffing  => true
    #workers => 2
  }
 file {
   path => "/data/logs/logstash/test.log"
 }
}

[jordansissel]

This seems like something wrong in the JVM/JRE -- the crash is pretty deep with the java crypto library.

[laoshancun]

thanks, i have upgraded logstash to 2.4.0.