elasticsearch fails to start tribe node

[kt97679]

Hi folks,

I'm trying to start tribe node using following config:

transport.tcp.port: 9301
http.port: 9201
network.host: 0.0.0.0
path.data: /var/lib/elasticsearch/
path.logs: /var/log/elasticsearch/

tribe:
    kibana:
        cluster.name: logstash-kibana
        discovery.zen.ping.multicast.enabled: false
        discovery.zen.ping.unicast.hosts: ["127.0.0.1"]
    els:
        cluster.name: logstash-data
        discovery.zen.ping.multicast.enabled: false
        discovery.zen.ping.unicast.hosts: ["10.128.69.48", "10.128.75.237"]

This config resides in the file /etc/tribe-elasticseach/elasticsearch.yml. I'm starting it using following command:

sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch -Ddefault.path.conf=/etc/tribe-elasticsearch/

Elasticsearch fails with following output:

[2015-11-05 17:07:42,433][INFO ][node                     ] [Bucky] version[2.0.0], pid[25943], build[de54438/2015-10-22T08:09:48Z]
[2015-11-05 17:07:42,434][INFO ][node                     ] [Bucky] initializing ...
[2015-11-05 17:07:42,596][INFO ][plugins                  ] [Bucky] loaded [], sites []
Exception in thread "main" java.security.AccessControlException: access denied ("java.io.FilePermission" "/usr/share/elasticsearch/config/elasticsearch.yml" "read")
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
        at java.security.AccessController.checkPermission(AccessController.java:884)
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
        at java.lang.SecurityManager.checkRead(SecurityManager.java:888)
        at sun.nio.fs.UnixPath.checkRead(UnixPath.java:795)
        at sun.nio.fs.UnixFileSystemProvider.checkAccess(UnixFileSystemProvider.java:290)
        at java.nio.file.Files.exists(Files.java:2385)
        at org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:87)
        at org.elasticsearch.node.Node.(Node.java:128)
        at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:145)
        at org.elasticsearch.tribe.TribeService.(TribeService.java:136)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at <<>>
        at org.elasticsearch.node.Node.(Node.java:198)
        at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:145)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:170)
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)

I'm not sure why it tries to access /usr/share/elasticsearch/config/elasticsearch.yml. There is no such file in the elasticsearch deb package. I created this file, but command above still fails with same output. Please advise how this can be resolved.

I'm running elasticsearch 2.0.0 installed from the debian package downloaded from the official site. I'm using ubuntu 14.

Thanks,
Kirill.

[clintongormley]

You're specifying the custom config file location incorrectly.

See https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_20_setting_changes.html#_custom_config_file

[kt97679]

Hi @clintongormley , thanks for quick response. As you can see from the description I've provided I was using option -Ddefault.path.conf. I tried again same command with option --path.conf. There was no exception because of config access issue, but I had to specify also --path.data and --path.logs because for some reason those settings were ignored in the config I've provided. In my config I also specify nonstandard ports to use and those settings are also not used. Any advise what can be wrong?

Thanks,
Kirill.

[kt97679]

Looks like config is ignored completely. If I specify all options via command line I still get exception like above:

# sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch --path.conf=/etc/tribe-elasticsearch/ --path.logs=/var/log/elasticsearch --path.data=/var/lib/elasticsearch/ --transport.tcp.port=9301 --http.port=9201 --network.host=0.0.0.0 --tribels.cluster.name=logstash-data --tribe.els.discovery.zen.ping.multicast.enabled=false --tribe.els.discovery.zen.ping.unicast.hosts=["10.128.69.48","10.128.75.237"]                                                                                                              
log4j:WARN No appenders could be found for logger (bootstrap).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.security.AccessControlException: access denied ("java.io.FilePermission" "/usr/share/elasticsearch/config/elasticsearch.yml" "read")
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
        at java.security.AccessController.checkPermission(AccessController.java:884)
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
        at java.lang.SecurityManager.checkRead(SecurityManager.java:888)
        at sun.nio.fs.UnixPath.checkRead(UnixPath.java:795)
        at sun.nio.fs.UnixFileSystemProvider.checkAccess(UnixFileSystemProvider.java:290)
        at java.nio.file.Files.exists(Files.java:2385)
        at org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:87)
        at org.elasticsearch.node.Node.(Node.java:128)
        at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:145)
        at org.elasticsearch.tribe.TribeService.(TribeService.java:136)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at <<>>
        at org.elasticsearch.node.Node.(Node.java:198)
        at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:145)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:170)
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)

[clintongormley]

Thanks for persisting. I've managed to replicate this and it is indeed a bug.

When the tribe node attempts to instantiate a node for the tribe service, it checks for access to the config directory, but that setting is no longer available to it and so it defaults to checking for path.home.

This can be replicated with a simple config file, saved as foo/elasticsearch.yml:

node.name: foo

tribe:
    foo:
        cluster.name: bar

Start elasticsearch as:

./elasticsearch-2.0.0/bin/elasticsearch --path.conf foo/

And it fails with:

[2015-11-17 13:54:47,763][INFO ][node                     ] [foo] version[2.0.0], pid[5940], build[de54438/2015-10-22T08:09:48Z]
[2015-11-17 13:54:47,763][INFO ][node                     ] [foo] initializing ...
[2015-11-17 13:54:47,836][INFO ][plugins                  ] [foo] loaded [], sites []
Exception in thread "main" java.security.AccessControlException: access denied ("java.io.FilePermission" "/Users/clinton/workspace/servers/elasticsearch-2.0.0/config/elasticsearch.yml" "read")
  at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
  at java.security.AccessController.checkPermission(AccessController.java:884)
  at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
  at java.lang.SecurityManager.checkRead(SecurityManager.java:888)
  at sun.nio.fs.UnixPath.checkRead(UnixPath.java:795)
  at sun.nio.fs.UnixFileSystemProvider.checkAccess(UnixFileSystemProvider.java:290)
  at java.nio.file.Files.exists(Files.java:2385)
  at org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:87)
  at org.elasticsearch.node.Node.<init>(Node.java:128)
  at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:145)
  at org.elasticsearch.tribe.TribeService.<init>(TribeService.java:136)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
  at <<<guice>>>
  at org.elasticsearch.node.Node.<init>(Node.java:198)
  at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:145)
  at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:170)
  at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)
  at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)

[clintongormley]

@javanna could you take a look at this please?

[javanna]

I had a look at this. Only selected settings are forwarded to the inner tribe clients from the tribe node. path.home is one of them but path.conf is not. That said, if I remember correctly the tribe clients shouldn't read from configuration file (and sysprops) but only inherit a few settings from the parent node (like it happens in TribeService), something that we had enforced with #9721. I think something got lost with #13383 where loadConfigSettings was removed, which was our way to prevent loading anything from the config file. With that set to false I believe we wouldn't even check for the existence of the file, thus we wouldn't need any permission for that. At this point it seems to me that we would have to forward path.conf to the tribe clients just because we are going to check for its existence at some point although we have nothing to load from it (otherwise we check for path.home that we have no permissions for)? I think I'd need @rjernst to verify if what I explained makes any sense, it might be that I overlooked something.

[rjernst]

If I understand the tribe node correctly, it is no different than any other client node (well, creating multiple client nodes internally). So to me, it should be passing along any settings it needs to configure the node (including path.conf). However, I'm not sure what this has to do with the transport client? The transport client by definition now does not use the config file settings (and the stack trace shown above indicates the exception was from building a node, not a transport client).