logstash.yml file not found if run the logstash command directly

[purbon]

If installing logstash as a package, but then using the logstash command directly, for example to check a config or to check for a version, it can not find the config file.

see

vagrant@packer-virtualbox-iso-1452039329:~$ sudo /usr/share/logstash/bin/logstash --version
--- jar coordinate com.fasterxml.jackson.core:jackson-annotations already loaded with version 2.7.1 - omit version 2.7.0
--- jar coordinate com.fasterxml.jackson.core:jackson-databind already loaded with version 2.7.1 - omit version 2.7.1-1
Errno::ENOENT: No such file or directory - /usr/share/logstash/config/logstash.yml
       read at org/jruby/RubyIO.java:3804
       read at org/jruby/RubyIO.java:3987
  read_yaml at /usr/share/logstash/vendor/local_gems/47abefa3/logstash-core-5.0.0.dev-java/lib/logstash/settings.rb:100
  from_yaml at /usr/share/logstash/vendor/local_gems/47abefa3/logstash-core-5.0.0.dev-java/lib/logstash/settings.rb:94
        run at /usr/share/logstash/vendor/local_gems/47abefa3/logstash-core-5.0.0.dev-java/lib/logstash/runner.rb:141
        run at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132
     (root) at /usr/share/logstash/lib/bootstrap/environment.rb:66
vagrant@packer-virtualbox-iso-1452039329:~$ 

logstash expect to have the file in the LS_HOME location, /usr/share/logstash however it also has one in /etc/logstash and not sure if users will also think that when running as a command it will pick up a config file in the $HOME directory.

I guess is resonable to think about having only 1 reference to the config file and this could be the one inside /etc/logstash.

This has been tested with the deb package, but will also have the same behaviour with the rpm I guess.

//cc @jsvd this is the bug I told you. @untergeek might be also of your interest.

[jsvd]

@untergeek how do you think going around this? IMO, blocking users from running logstash on the command line when installed from a package seems harsh..
What if we moved the sourcing of startup.options to bin/logstash.lib.sh so path.settings is read even when running from cli?

[jsvd]

we can follow 2 strategies to address this:

1. be less strict on the need to find a logstash.yml file. If not found, just print a warning and use the internal defaults
2. enrich the path.settings setting to have a list of default locations, circle through them by default, and remove the explicit setting from startup.options

thoughts?

[purbon]

I think I like 1 a lot, it makes a lot of sense to me 'cause why do we require to have a config file if we've defaults for everything?

[ph]

I agree with strategy #1, I think its better to have a working logstash than a cryptic error.

[suyograo]

how do you think going around this? IMO, blocking users from running logstash on the command line when installed from a package seems harsh..

Why? We are not blocking users, its just about providing the right feedback. You can still run Logstash manually from /usr/share/logstash when installed from packages, but you have to pass path.settings via command line. We discussed this briefly in Slack with @untergeek on Friday. FWIW, ES also does the same thing when installed via packages and run manually.

As I mentioned in #5393, IMO, fail fast is better than implicitly assuming defaults. This can lead to confusions down the line to the user, and someone debugging it. We should keep it simple: either pass the settings via command line, or via settings file whether you install through packages or tarball. Designing another option to override these two will only complicate things in code and user behavior in the future.

I agree with strategy #1, I think its better to have a working logstash than a cryptic error.

We don't have to settle for a cryptic message, we can catch the message as done in #5393 and provide a more meaningful feedback to the user, but still bail.

Happy to discuss this more..

[jsvd]

I don't see an issue with the warning + skipping the yml file. More often than not it will be completely commented out, so it's ok to ignore it as long as we make it clear.
for reference, in elasticsearch we also get an explosion in the face (as Suyog mentioned):

$ sudo dpkg -i elasticsearch-5.0.0-alpha3.deb 
Selecting previously unselected package elasticsearch.
(Reading database ... 64155 files and directories currently installed.)
Preparing to unpack elasticsearch-5.0.0-alpha3.deb ...
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Unpacking elasticsearch (5.0.0-alpha3) ...
Setting up elasticsearch (5.0.0-alpha3) ...
Processing triggers for ureadahead (0.100.0-16) ...
$ /usr/share/elasticsearch/bin/elasticsearch
Exception in thread "main" ElasticsearchException[Failed to load logging configuration]; nested: NoSuchFileException[/usr/share/elasticsearch/config];
Likely root cause: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
.....

[jsvd]

after some more discussion we decided to treat this as a fatal exception to avoid surprises that the yml file isn't being loaded.
so the action will be to print an error message and terminate

[jsvd]

closed by #5393