Die with dignity by jasontedor · Pull Request #19272 · elastic/elasticsearch

jasontedor · 2016-07-05T17:00:26Z

Today when a thread encounters a fatal unrecoverable error that
threatens the stability of the JVM, Elasticsearch marches on. This
includes out of memory errors, stack overflow errors and other errors
that leave the JVM in a questionable state. Instead, the Elasticsearch
JVM should die when these errors are encountered. This commit causes
this to be the case.

Relates #19231

rmuir · 2016-07-05T18:25:59Z

+1

Today when a thread encounters a fatal unrecoverable error that threatens the stability of the JVM, Elasticsearch marches on. This includes out of memory errors, stack overflow errors and other errors that leave the JVM in a questionable state. Instead, the Elasticsearch JVM should die when these errors are encountered. This commit causes this to be the case.

tlrx · 2016-07-05T20:17:31Z

LGTM, I like it

tlrx · 2016-07-05T20:18:20Z

We might want to document the exit codes somewhere too

jasontedor · 2016-07-05T21:22:27Z

We might want to document the exit codes somewhere too

@tlrx I think that's a good idea, but I'm unsure of a good place to document it. Do you or @clintongormley have any ideas?

dadoonet · 2016-07-05T22:08:11Z

FYI we documented something similar for plugins: https://www.elastic.co/guide/en/elasticsearch/plugins/current/_other_command_line_parameters.html

clintongormley · 2016-07-06T12:16:59Z

@jasontedor not perfect, but perhaps a section under Setup?

This commit adds docs for the Elasticsearch shutdown process, distinguishing between a clean shutdown and an unclean halt due to fatal error.

tlrx · 2016-07-07T15:59:19Z

docs/reference/setup/stopping.asciidoc

+If you're running Elasticsearch directly, you can stop Elasticsearch by sending control-C if you're
+running Elasticsearch in the console, or by sending `SIGTERM` to the Elasticsearch process on a
+POSIX system. You can obtain the PID to send the signal to via various tools (e.g., `ps` or `jps`)
+or by specifying a location to write a PID file to on startup (`-p <path>`).


When ran directly the pid is shown in the logs/stdout too.

Thanks @tlrx, I pushed 94ef28b.

Sorry, I should be more clear. I think we can rephrase like:

Elasticsearch indicates the PID in logs or standard ouput during startup:
[INFO ][node ] [Araña] version[2.3.3], pid[6674], build[218bdf1/2016-05-17T15:40:04Z]
You can also obtain...

or something like this

I pushed 4550140. Is that more like what you had in mind?

4550140 is good, thanks!

This commit adds a note to the Elasticsearch stopping docs that indicates that the Elasticsearch PID is also available from the Elasticsearch startup logs.

jasontedor · 2016-07-07T16:03:38Z

@tlrx @clintongormley I pushed docs in 74c1708; could you take a look?

tlrx · 2016-07-07T16:05:33Z

LGTM

clintongormley · 2016-07-08T15:41:39Z

Sorry I'm late to this. You can also get the PID from the node info API:

curl -XGET "http://192.168.1.10:9200/_nodes/process"

{
  "_nodes": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "cluster_name": "elasticsearch",
  "nodes": {
    "uOPHTfzUQUyGRuxdSUBniQ": {
      "name": "Karl Lykos",
      "transport_address": "127.0.0.1:9300",
      "host": "127.0.0.1",
      "ip": "127.0.0.1",
      "version": "5.0.0-alpha4",
      "build_hash": "3f5b994",
      "http_address": "127.0.0.1:9200",
      "roles": [
        "master",
        "data",
        "ingest"
      ],
      "process": {
        "refresh_interval_in_millis": 1000,
        "id": 26283,  ##### HERE #####
        "mlockall": false
      }
    }
  }
}

jasontedor · 2016-07-08T19:01:50Z

@clintongormley Yes, and the cat nodes API too. The concern I have there is that that will list all the nodes in the cluster when only the local node is needed and that leads to more effort on the end-user? I'm happy to add these though if you think otherwise. 😄

jasontedor added >enhancement >breaking review resiliency v5.0.0-alpha5 labels Jul 5, 2016

clintongormley added :Exceptions and removed >enhancement labels Jul 6, 2016

Add docs for stopping Elasticsearch

74c1708

This commit adds docs for the Elasticsearch shutdown process, distinguishing between a clean shutdown and an unclean halt due to fatal error.

tlrx reviewed Jul 7, 2016
View reviewed changes

Add note regarding Elasticsearch PID in logs

94ef28b

This commit adds a note to the Elasticsearch stopping docs that indicates that the Elasticsearch PID is also available from the Elasticsearch startup logs.

Add detail for getting PID to shutdown docs

4550140

jasontedor merged commit e86aa29 into elastic:master Jul 7, 2016

jasontedor deleted the die-with-dignity branch July 7, 2016 18:44

This was referenced Jul 11, 2016

Do not catch throwable #19231

Merged

Should the Elasticsearch packages configure respawn? #19438

Closed

jasontedor mentioned this pull request Jul 20, 2016

Acquire Java version simply netty/netty#5552

Closed

bleskes mentioned this pull request Jul 27, 2016

Split regular histograms from date histograms. #19551

Merged

jasontedor mentioned this pull request Jul 29, 2016

Kibana bug locks up Elasticsearch #19690

Closed

jasontedor mentioned this pull request Aug 10, 2016

Mark halting the virtual machine as privileged #19923

Merged

jasontedor mentioned this pull request Oct 17, 2016

Detect when benchmark candidate fails with OutOfMemoryError elastic/rally#169

Closed

danielmitterdorfer mentioned this pull request Oct 18, 2016

Check exit code of the benchmark candidate elastic/rally#170

Closed

This was referenced Nov 21, 2016

Die with dignity on the network layer #21720

Merged

Die with dignity on the Lucene layer #21721

Merged

jasontedor mentioned this pull request Dec 2, 2016

Netty should stop swallowing netty/netty#6096

Open

This was referenced Sep 14, 2017

Revert 'ExitOnOutOfMemoryError' setting because it requires minimum Java 8u92 elastic/logstash#8254

Closed

Add fatal JVM error handling elastic/logstash#8279

Open

jasontedor mentioned this pull request Sep 16, 2017

[o.e.b.ElasticsearchUncaughtExceptionHandler] [node01-es-dev] fatal error in thread [elasticsearch[node01-es-dev][search][T#6]], exiting java.lang.OutOfMemoryError: Java heap space #26525

Closed

jasontedor mentioned this pull request Nov 3, 2017

Die with dignity while merging #27265

Merged

lcawl added :Core/Infra/Core Core issues without another label and removed :Exceptions labels Feb 13, 2018

jasontedor mentioned this pull request Mar 12, 2018

Add test for dying with dignity #28987

Merged

jaymode mentioned this pull request Aug 27, 2018

ldapsdk should stop suppressing Error pingidentity/ldapsdk#49

Closed

Conversation

jasontedor commented Jul 5, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rmuir commented Jul 5, 2016

Uh oh!

tlrx commented Jul 5, 2016

Uh oh!

tlrx commented Jul 5, 2016

Uh oh!

jasontedor commented Jul 5, 2016

Uh oh!

dadoonet commented Jul 5, 2016

Uh oh!

clintongormley commented Jul 6, 2016

Uh oh!

tlrx Jul 7, 2016

Choose a reason for hiding this comment

Uh oh!

jasontedor Jul 7, 2016

Choose a reason for hiding this comment

Uh oh!

tlrx Jul 7, 2016

Choose a reason for hiding this comment

Uh oh!

jasontedor Jul 7, 2016

Choose a reason for hiding this comment

Uh oh!

tlrx Jul 7, 2016

Choose a reason for hiding this comment

Uh oh!

jasontedor commented Jul 7, 2016

Uh oh!

tlrx commented Jul 7, 2016

Uh oh!

clintongormley commented Jul 8, 2016

Uh oh!

jasontedor commented Jul 8, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Comments

jasontedor commented Jul 5, 2016 •

edited

Loading