Prevent unexpected native controller output hanging the process #56491
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In normal operation native controllers are not expected to write anything to stdout or stderr. However, if due to an error or something unexpected with the environment a native controller does write something to stdout or stderr then it will block if nothing is reading that output. This change makes the stdout and stderr of native controllers reuse the same stdout and stderr as the Elasticsearch JVM (which are by default redirected to es.stdout.log and es.stderr.log) so that if something unexpected is written to native controller output then: 1. The native controller process does not block, waiting for something to read the output 2. We can see what the output was, making it easier to debug obscure environmental problems Relates elastic#56366
droberts195
added
>bug
:Core/Infra/Core
Collaborator
|
Pinging @elastic/es-core-infra (:Core/Infra/Core) |
pugnascotia
approved these changes
May 12, 2020
droberts195
pushed a commit
to droberts195/elasticsearch
that referenced
this pull request
May 13, 2020
In normal operation native controllers are not expected to write anything to stdout or stderr. However, if due to an error or something unexpected with the environment a native controller does write something to stdout or stderr then it will block if nothing is reading that output. This change makes the stdout and stderr of native controllers reuse the same stdout and stderr as the Elasticsearch JVM (which are by default redirected to es.stdout.log and es.stderr.log) so that if something unexpected is written to native controller output then: 1. The native controller process does not block, waiting for something to read the output 2. We can see what the output was, making it easier to debug obscure environmental problems Backport of elastic#56491
droberts195
pushed a commit
to droberts195/elasticsearch
that referenced
this pull request
May 13, 2020
In normal operation native controllers are not expected to write anything to stdout or stderr. However, if due to an error or something unexpected with the environment a native controller does write something to stdout or stderr then it will block if nothing is reading that output. This change makes the stdout and stderr of native controllers reuse the same stdout and stderr as the Elasticsearch JVM (which are by default redirected to es.stdout.log and es.stderr.log) so that if something unexpected is written to native controller output then: 1. The native controller process does not block, waiting for something to read the output 2. We can see what the output was, making it easier to debug obscure environmental problems Backport of elastic#56491
droberts195
pushed a commit
to droberts195/elasticsearch
that referenced
this pull request
May 13, 2020
In normal operation native controllers are not expected to write anything to stdout or stderr. However, if due to an error or something unexpected with the environment a native controller does write something to stdout or stderr then it will block if nothing is reading that output. This change makes the stdout and stderr of native controllers reuse the same stdout and stderr as the Elasticsearch JVM (which are by default redirected to es.stdout.log and es.stderr.log) so that if something unexpected is written to native controller output then: 1. The native controller process does not block, waiting for something to read the output 2. We can see what the output was, making it easier to debug obscure environmental problems Backport of elastic#56491
This was referenced May 13, 2020
droberts195
pushed a commit
that referenced
this pull request
May 13, 2020
In normal operation native controllers are not expected to write anything to stdout or stderr. However, if due to an error or something unexpected with the environment a native controller does write something to stdout or stderr then it will block if nothing is reading that output. This change makes the stdout and stderr of native controllers reuse the same stdout and stderr as the Elasticsearch JVM (which are by default redirected to es.stdout.log and es.stderr.log) so that if something unexpected is written to native controller output then: 1. The native controller process does not block, waiting for something to read the output 2. We can see what the output was, making it easier to debug obscure environmental problems Backport of #56491 Co-authored-by: Elastic Machine <[email protected]>
droberts195
pushed a commit
that referenced
this pull request
May 13, 2020
In normal operation native controllers are not expected to write anything to stdout or stderr. However, if due to an error or something unexpected with the environment a native controller does write something to stdout or stderr then it will block if nothing is reading that output. This change makes the stdout and stderr of native controllers reuse the same stdout and stderr as the Elasticsearch JVM (which are by default redirected to es.stdout.log and es.stderr.log) so that if something unexpected is written to native controller output then: 1. The native controller process does not block, waiting for something to read the output 2. We can see what the output was, making it easier to debug obscure environmental problems Backport of #56491 Co-authored-by: Elastic Machine <[email protected]>
droberts195
pushed a commit
that referenced
this pull request
May 14, 2020
In normal operation native controllers are not expected to write anything to stdout or stderr. However, if due to an error or something unexpected with the environment a native controller does write something to stdout or stderr then it will block if nothing is reading that output. This change makes the stdout and stderr of native controllers reuse the same stdout and stderr as the Elasticsearch JVM (which are by default redirected to es.stdout.log and es.stderr.log) so that if something unexpected is written to native controller output then: 1. The native controller process does not block, waiting for something to read the output 2. We can see what the output was, making it easier to debug obscure environmental problems Backport of #56491
droberts195
pushed a commit
that referenced
this pull request
May 14, 2020
In normal operation native controllers are not expected to write anything to stdout or stderr. However, if due to an error or something unexpected with the environment a native controller does write something to stdout or stderr then it will block if nothing is reading that output. This change makes the stdout and stderr of native controllers reuse the same stdout and stderr as the Elasticsearch JVM (which are by default redirected to es.stdout.log and es.stderr.log) so that if something unexpected is written to native controller output then: 1. The native controller process does not block, waiting for something to read the output 2. We can see what the output was, making it easier to debug obscure environmental problems Backport of #56491
droberts195
pushed a commit
to droberts195/ml-cpp
that referenced
this pull request
Jun 11, 2020
This change fixes two related issues with named pipe connection when the controller process first starts up: 1. If the ES JVM dies before the controller connects its logging named pipe then since elastic/elasticsearch#56491 the resulting errors from the controller process will be seen in the ES stderr file. There is a change to the logging to make it clearer that the controller didn't fail, but exited due to the ES JVM disappearing. 2. Interrupted system calls while connecting the named pipes could cause the pipes to unnecessarily fail to connect. There is a change to retry the calls on getting an EINTR error unless the interrupt was caused by the functionality of controller that kills off the connection attempts if the ES JVM dies (i.e. the scenario described in point 1). Relates elastic#564 Relates elastic/elasticsearch#57969
droberts195
pushed a commit
to elastic/ml-cpp
that referenced
this pull request
Jun 17, 2020
This change fixes two related issues with named pipe connection when the controller process first starts up: 1. If the ES JVM dies before the controller connects its logging named pipe then since elastic/elasticsearch#56491 the resulting errors from the controller process will be seen in the ES stderr file. There is a change to the logging to make it clearer that the controller didn't fail, but exited due to the ES JVM disappearing. 2. Interrupted system calls while connecting the named pipes could cause the pipes to unnecessarily fail to connect. There is a change to retry the calls on getting an EINTR error unless the interrupt was caused by the functionality of controller that kills off the connection attempts if the ES JVM dies (i.e. the scenario described in point 1). Relates #564 Relates elastic/elasticsearch#57969
droberts195
pushed a commit
to droberts195/ml-cpp
that referenced
this pull request
Jun 17, 2020
This change fixes two related issues with named pipe connection when the controller process first starts up: 1. If the ES JVM dies before the controller connects its logging named pipe then since elastic/elasticsearch#56491 the resulting errors from the controller process will be seen in the ES stderr file. There is a change to the logging to make it clearer that the controller didn't fail, but exited due to the ES JVM disappearing. 2. Interrupted system calls while connecting the named pipes could cause the pipes to unnecessarily fail to connect. There is a change to retry the calls on getting an EINTR error unless the interrupt was caused by the functionality of controller that kills off the connection attempts if the ES JVM dies (i.e. the scenario described in point 1). Backport of elastic#1311
droberts195
pushed a commit
to droberts195/ml-cpp
that referenced
this pull request
Jun 17, 2020
This change fixes two related issues with named pipe connection when the controller process first starts up: 1. If the ES JVM dies before the controller connects its logging named pipe then since elastic/elasticsearch#56491 the resulting errors from the controller process will be seen in the ES stderr file. There is a change to the logging to make it clearer that the controller didn't fail, but exited due to the ES JVM disappearing. 2. Interrupted system calls while connecting the named pipes could cause the pipes to unnecessarily fail to connect. There is a change to retry the calls on getting an EINTR error unless the interrupt was caused by the functionality of controller that kills off the connection attempts if the ES JVM dies (i.e. the scenario described in point 1). Backport of elastic#1311
droberts195
pushed a commit
to droberts195/ml-cpp
that referenced
this pull request
Jun 17, 2020
This change fixes two related issues with named pipe connection when the controller process first starts up: 1. If the ES JVM dies before the controller connects its logging named pipe then since elastic/elasticsearch#56491 the resulting errors from the controller process will be seen in the ES stderr file. There is a change to the logging to make it clearer that the controller didn't fail, but exited due to the ES JVM disappearing. 2. Interrupted system calls while connecting the named pipes could cause the pipes to unnecessarily fail to connect. There is a change to retry the calls on getting an EINTR error unless the interrupt was caused by the functionality of controller that kills off the connection attempts if the ES JVM dies (i.e. the scenario described in point 1). Backport of elastic#1311
This was referenced Jun 17, 2020
droberts195
pushed a commit
to elastic/ml-cpp
that referenced
this pull request
Jun 17, 2020
This change fixes two related issues with named pipe connection when the controller process first starts up: 1. If the ES JVM dies before the controller connects its logging named pipe then since elastic/elasticsearch#56491 the resulting errors from the controller process will be seen in the ES stderr file. There is a change to the logging to make it clearer that the controller didn't fail, but exited due to the ES JVM disappearing. 2. Interrupted system calls while connecting the named pipes could cause the pipes to unnecessarily fail to connect. There is a change to retry the calls on getting an EINTR error unless the interrupt was caused by the functionality of controller that kills off the connection attempts if the ES JVM dies (i.e. the scenario described in point 1). Backport of #1311
droberts195
pushed a commit
to elastic/ml-cpp
that referenced
this pull request
Jun 18, 2020
This change fixes two related issues with named pipe connection when the controller process first starts up: 1. If the ES JVM dies before the controller connects its logging named pipe then since elastic/elasticsearch#56491 the resulting errors from the controller process will be seen in the ES stderr file. There is a change to the logging to make it clearer that the controller didn't fail, but exited due to the ES JVM disappearing. 2. Interrupted system calls while connecting the named pipes could cause the pipes to unnecessarily fail to connect. There is a change to retry the calls on getting an EINTR error unless the interrupt was caused by the functionality of controller that kills off the connection attempts if the ES JVM dies (i.e. the scenario described in point 1). Backport of #1311
droberts195
pushed a commit
to elastic/ml-cpp
that referenced
this pull request
Jun 18, 2020
This change fixes two related issues with named pipe connection when the controller process first starts up: 1. If the ES JVM dies before the controller connects its logging named pipe then since elastic/elasticsearch#56491 the resulting errors from the controller process will be seen in the ES stderr file. There is a change to the logging to make it clearer that the controller didn't fail, but exited due to the ES JVM disappearing. 2. Interrupted system calls while connecting the named pipes could cause the pipes to unnecessarily fail to connect. There is a change to retry the calls on getting an EINTR error unless the interrupt was caused by the functionality of controller that kills off the connection attempts if the ES JVM dies (i.e. the scenario described in point 1). Backport of #1311
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In normal operation native controllers are not expected to write
anything to stdout or stderr. However, if due to an error or
something unexpected with the environment a native controller
does write something to stdout or stderr then it will block if
nothing is reading that output. (On Linux both glibc and ld.so.1
can write errors to stderr in certain situations.)
This change makes the stdout and stderr of native controllers
reuse the same stdout and stderr as the Elasticsearch JVM (which
are by default redirected to es.stdout.log and es.stderr.log) so
that if something unexpected is written to native controller
output then:
something to read the output
obscure environmental problems
Relates #56366