Init scripts reports "OK" when Elasticsearch fails to start from bad YAML

[martinb3]

Hello! The Chef cookbook for Elasticsearch has received a bug report regarding the packaging of ES, specifically the init scripts and status reporting. Here's the actual report: sous-chefs/elasticsearch#483

In short, the user is seeing the init script for Ubuntu 14.04 report "OK" when starting Elasticsearch, even though parsing of the YAML configuration file has failed. Perhaps Elasticsearch needs a 'configtest' akin to the way Apache and Nginx have implemented it, or perhaps the init script just needs improvement to catch when Elasticsearch has exited or failed to start.

Elasticsearch version: 2.3.3
JVM version: 1.8.0_91
OS version: ubuntu 14.04

Steps to reproduce:

1. Add http.cors.allow-origin: * to elasticsearch.yml
2. Attempt to start the service

Provide logs (if relevant):
The user reporting the issue found this in his logs:

Exception in thread "main" SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: ScannerException[while scanning an alias
 in 'reader', line 41, column 25:
    http.cors.allow-origin: *
                            ^
expected alphabetic or numeric character, but found but found

 in 'reader', line 41, column 26:
    http.cors.allow-origin: *
                             ^
];
Likely root cause: while scanning an alias
 in 'reader', line 41, column 25:
    http.cors.allow-origin: *
                            ^

[clintongormley]

I think this is a duplicate of #12716 and Elasticsearch is failing to return the proper exit code because of the double daemonization

[clintongormley]

@martinb3 any chance you could try what is suggested #12716 (comment) and let us know if it fixes your issue?

[martinb3]

Howdy @clintongormley -- I searched the tracker but didn't find that one -- sorry about that! I'll ask the original bug reporter to try the workaround.

[clintongormley]

@martinb3 no worries - i want to get this one fixed, so would be good to have corroboration

[jasontedor]

@martinb3 Can you share some more details here? I think on System V init compatible systems we should be okay here but maybe not on systemd systems? I think that Ubuntu 14.04 is a System V init compatible system by default?

The Elasticsearch JVM does successfully start, it just dies during node start because of the malformed configuration file. We do have sleeps in our scripts attempting to detect when this happens and I just tested it on a clean Ubuntu 14.04 install and it did the right thing:

# cat /etc/elasticsearch/elasticsearch.yml
node.name: test
# purposefully malformed
 cluster.name: test
# service elasticsearch start
 * Starting Elasticsearch Server [fail]
#

[clintongormley]

@jasontedor but does the double daemonization mentioned in #12716 actually make sense? If we remove that, surely we wouldn't need the sleep?

[jasontedor]

@clintongormley To be clear, the double daemonization needs to be removed and we should just rely on daemonization from the underlying service implementation but we still need a sleep somewhere otherwise in cases like this Elasticsearch will successfully daemonize and later die. The successful daemonization will appear like successful startup, so a sleep is needed to give it time to die if it's going to.