Fix a concurrency problem with web socket cleanup #8121
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We were seeing crashes in Deflater.deflate with a root cause of closing the deflater while it was in use. The underlying problem is that web socket streams were closed when the web socket was canceled, but not by the thread that owned those streams. This moves stream cleanup to always run on the owning thread: - writer cleanup happens on the task queue - reader cleanup happens after loopReader This also introduces a new function, Streams.cancel() so a failed writer can break an active reader, and vise-versa. Previously we were using close() to do this job. There's a lot of test changes but it's mostly in adding the TaskFaker facet to RealWebSocketTest. The test now runs with a fake clock, with the deterministic benefits that brings. See also: #6719
swankjesse
commented
Dec 2, 2023
| if (closed) { | ||
| throw new AssertionError("Already closed"); | ||
| } | ||
| try { | ||
| getSource().close(); | ||
| } catch (IOException ignored) { |
There was a problem hiding this comment.
TODO: convert to Kotlin and get closeQuietly() !
swankjesse
commented
Dec 2, 2023
|
|
||
| // Even if messages are no longer being read we need to wait for the connection close signal. | ||
| connectionClose.await() | ||
| } catch (e: IOException) { | ||
| webSocket.failWebSocket(e, null) |
There was a problem hiding this comment.
Moved into loopReader()
swankjesse
commented
Dec 2, 2023
| listener.onOpen(this@RealWebSocket, response) | ||
| loopReader() | ||
| } catch (e: Exception) { | ||
| failWebSocket(e, null) |
There was a problem hiding this comment.
moved into loopReader()
swankjesse
commented
Dec 2, 2023
| } | ||
|
|
||
| if (!failed && streamsToClose != null && receivedCloseCode != -1) { | ||
| listener.onClosed(this, code, reason!!) |
There was a problem hiding this comment.
This is now published either when the reader exits, or when the writer writes Close, whichever happens last
| } | ||
|
|
||
| try { |
There was a problem hiding this comment.
all this stuff moved to finishReader()
| @@ -542,9 +554,8 @@ class RealWebSocket( | |||
|
|
|||
| return true | |||
| } finally { | |||
| streamsToClose?.closeQuietly() | |||
| readerToClose?.closeQuietly() | |||
There was a problem hiding this comment.
Bugfix 1: the writer is closing the reader here. Racy and bad!
| taskQueue.shutdown() | ||
| } | ||
|
|
||
| try { | ||
| listener.onFailure(this, e, response) | ||
| } finally { | ||
| streamsToClose?.closeQuietly() | ||
| readerToClose?.closeQuietly() | ||
| writerToClose?.closeQuietly() |
There was a problem hiding this comment.
Bugfix 2: the reader is closing the writer here. Racy and bad!
yschimke
reviewed
Dec 2, 2023
yschimke
approved these changes
Dec 2, 2023
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We were seeing crashes in Deflater.deflate with a root cause of closing the deflater while it was in use.
The underlying problem is that web socket streams were closed when the web socket was canceled, but not by the thread that owned those streams.
This moves stream cleanup to always run on the owning thread:
This also introduces a new function, Streams.cancel() so a failed writer can break an active reader, and vise-versa. Previously we were using close() to do this job.
There's a lot of test changes but it's mostly in adding the TaskFaker facet to RealWebSocketTest. The test now runs with a fake clock, with the deterministic benefits that brings.
See also: #6719