Fix leaked HTTP response sent after close#105293
Merged
elasticsearchmachine merged 4 commits intoelastic:mainfrom Feb 8, 2024
Merged
Fix leaked HTTP response sent after close#105293elasticsearchmachine merged 4 commits intoelastic:mainfrom
elasticsearchmachine merged 4 commits intoelastic:mainfrom
Conversation
Today a `HttpResponse` is always released via a `ChannelPromise` which means the release happens on a network thread. However, it's possible we try and send a `HttpResponse` after the node has got far enough through shutdown that it doesn't have any running network threads left, which means the response just leaks. This is no big deal in production, it becomes irrelevant when the process exits, but in tests we start and stop many nodes within the same process so mustn't leak anything. At this point in shutdown, all HTTP channels are now closed, so it's sufficient to check whether the channel is open first, and to fail the listener on the calling thread if not. That's what this commit does. Closes elastic#104651
DaveCTurner
added
>bug
:Distributed/Network
Collaborator
|
Hi @DaveCTurner, I've created a changelog YAML for you. |
Collaborator
|
Pinging @elastic/es-distributed (Team:Distributed) |
pxsalehi
approved these changes
Feb 8, 2024
| try (var client = RestClient.builder(new HttpHost(address.getAddress(), address.getPort())).build()) { | ||
| client.performRequestAsync(new Request("GET", url), ActionTestUtils.wrapAsRestResponseListener(ActionListener.noop())); | ||
| safeAwait(handlingRequestLatch); | ||
|
|
Member
There was a problem hiding this comment.
Tiny nit: it's probably on purpose, but that empty line seems unnecessary!
DaveCTurner
added
auto-merge-without-approval
DaveCTurner
added a commit
to DaveCTurner/elasticsearch
that referenced
this pull request
Feb 8, 2024
Today a `HttpResponse` is always released via a `ChannelPromise` which means the release happens on a network thread. However, it's possible we try and send a `HttpResponse` after the node has got far enough through shutdown that it doesn't have any running network threads left, which means the response just leaks. This is no big deal in production, it becomes irrelevant when the process exits, but in tests we start and stop many nodes within the same process so mustn't leak anything. At this point in shutdown, all HTTP channels are now closed, so it's sufficient to check whether the channel is open first, and to fail the listener on the calling thread if not. That's what this commit does. Closes elastic#104651
Collaborator
💚 Backport successful
|
DaveCTurner
added a commit
to DaveCTurner/elasticsearch
that referenced
this pull request
Feb 8, 2024
Similar to elastic#97301, the fix in elastic#105293 was still not quite correct: we could in principle shut down the transport after checking `isOpen()` but before sending the message. Applying the same fix as for the transport layer here.
Member
Author
|
Ah damn even this isn't quite enough, see #105306. |
elasticsearchmachine
pushed a commit
that referenced
this pull request
Feb 8, 2024
* Fix leaked HTTP response sent after close (#105293) Today a `HttpResponse` is always released via a `ChannelPromise` which means the release happens on a network thread. However, it's possible we try and send a `HttpResponse` after the node has got far enough through shutdown that it doesn't have any running network threads left, which means the response just leaks. This is no big deal in production, it becomes irrelevant when the process exits, but in tests we start and stop many nodes within the same process so mustn't leak anything. At this point in shutdown, all HTTP channels are now closed, so it's sufficient to check whether the channel is open first, and to fail the listener on the calling thread if not. That's what this commit does. Closes #104651 * Fix backport
Member
|
@DaveCTurner how did you find out this fix wasn't enough? Was there a test failure? |
Member
Author
|
No, I just took a break and my subconscious told me to take another look. It would have caused test failures tho, just a little rarer than before. |
DaveCTurner
added a commit
that referenced
this pull request
Feb 9, 2024
DaveCTurner
added a commit
to DaveCTurner/elasticsearch
that referenced
this pull request
Feb 9, 2024
Similar to elastic#97301, the fix in elastic#105293 was still not quite correct: we could in principle shut down the transport after checking `isOpen()` but before sending the message. Applying the same fix as for the transport layer here.
elasticsearchmachine
pushed a commit
that referenced
this pull request
Feb 9, 2024
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Today a
HttpResponseis always released via aChannelPromisewhichmeans the release happens on a network thread. However, it's possible we
try and send a
HttpResponseafter the node has got far enough throughshutdown that it doesn't have any running network threads left, which
means the response just leaks.
This is no big deal in production, it becomes irrelevant when the
process exits, but in tests we start and stop many nodes within the same
process so mustn't leak anything.
At this point in shutdown, all HTTP channels are now closed, so it's
sufficient to check whether the channel is open first, and to fail the
listener on the calling thread if not. That's what this commit does.
Closes #104651