Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HTTP/2 max local stream count exceeded when request fails #2796

Closed
sbordet opened this issue Aug 13, 2018 · 1 comment
Closed

HTTP/2 max local stream count exceeded when request fails #2796

sbordet opened this issue Aug 13, 2018 · 1 comment

Comments

@sbordet
Copy link
Contributor

sbordet commented Aug 13, 2018 

java.lang.IllegalStateException: Max local stream count 2 exceeded
        at org.eclipse.jetty.http2.HTTP2Session.createLocalStream(HTTP2Session.java:696)
        at org.eclipse.jetty.http2.HTTP2Session.newStream(HTTP2Session.java:491)
        at org.eclipse.jetty.http2.client.http.HttpSenderOverHTTP2.sendHeaders(HttpSenderOverHTTP2.java:93)
        at org.eclipse.jetty.client.HttpSender.send(HttpSender.java:212)
        at org.eclipse.jetty.http2.client.http.HttpChannelOverHTTP2.send(HttpChannelOverHTTP2.java:95)
        at org.eclipse.jetty.client.HttpChannel.send(HttpChannel.java:128)
        at org.eclipse.jetty.client.HttpConnection.send(HttpConnection.java:201)
        at org.eclipse.jetty.http2.client.http.HttpConnectionOverHTTP2.send(HttpConnectionOverHTTP2.java:72)
        at org.eclipse.jetty.http2.client.http.HttpDestinationOverHTTP2.send(HttpDestinationOverHTTP2.java:38)
        at org.eclipse.jetty.client.HttpDestination.process(HttpDestination.java:347)
        at org.eclipse.jetty.client.HttpDestination.process(HttpDestination.java:305)
        at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:295)
        at org.eclipse.jetty.client.HttpDestination.release(HttpDestination.java:400)
        at org.eclipse.jetty.http2.client.http.HttpConnectionOverHTTP2.release(HttpConnectionOverHTTP2.java:99)
        at org.eclipse.jetty.http2.client.http.HttpChannelOverHTTP2.release(HttpChannelOverHTTP2.java:101)
        at org.eclipse.jetty.http2.client.http.HttpChannelOverHTTP2.exchangeTerminated(HttpChannelOverHTTP2.java:121)
        at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:465)
        at org.eclipse.jetty.client.HttpReceiver.abort(HttpReceiver.java:552)
        at org.eclipse.jetty.client.HttpChannel.abortResponse(HttpChannel.java:156)
        at org.eclipse.jetty.client.HttpChannel.abort(HttpChannel.java:149)
        at org.eclipse.jetty.http2.client.http.HttpChannelOverHTTP2.abort(HttpChannelOverHTTP2.java:108)
        at org.eclipse.jetty.client.HttpExchange.abort(HttpExchange.java:257)
        at org.eclipse.jetty.client.HttpConversation.abort(HttpConversation.java:141)
        at org.eclipse.jetty.client.HttpRequest.abort(HttpRequest.java:767)
        at org.eclipse.jetty.client.TimeoutCompleteListener.onTimeoutExpired(TimeoutCompleteListener.java:52)
        at org.eclipse.jetty.io.CyclicTimeout$Wakeup.run(CyclicTimeout.java:271)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:299)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
        at java.base/java.lang.Thread.run(Thread.java:844)
@sbordet
Copy link
Contributor Author

sbordet commented Aug 13, 2018

The issue is that HttpChannelOverHTTP2.abort() first calls super and then resets the stream.
Unfortunately, calling super returns the connection to the pool, which will then be available for another stream before the aborted stream is closed, which would decrease the local stream count.

sbordet added a commit that referenced this issue Aug 14, 2018
@sbordet
Merge pull request #2797 from eclipse/jetty-9.4.x-2796-max_concurrent…
03af18c 
…_streams_local_timeout

Fixes #2796 - Max local stream count exceeded when request fails.
sbordet added a commit that referenced this issue Oct 9, 2018
@sbordet
Issue #2796 - Max local stream count exceeded when request fails.
cec84cf 
Reviewed other possible places where max local stream count may
overflow.

Fixed handling of HTTP/2 stream idle timeouts.

Signed-off-by: Simone Bordet <[email protected]>
@sbordet sbordet mentioned this issue Oct 9, 2018
Merged
sbordet added a commit that referenced this issue Oct 11, 2018
@sbordet
Issue #2796 - Max local stream count exceeded when request fails.
ec2b5b1 
Bound release of the channel to stream close event.

Signed-off-by: Simone Bordet <[email protected]>
sbordet added a commit that referenced this issue Oct 18, 2018
@sbordet
Issue #2796 - Max local stream count exceeded when request fails.
907e7af 
Updates after review.

Signed-off-by: Simone Bordet <[email protected]>
@sbordet sbordet mentioned this issue Oct 29, 2018
Closed
sbordet added a commit that referenced this issue Oct 30, 2018
@sbordet
Issue #2796 - Max local stream count exceeded when request fails.
31cab3d 
Restored smaller maxContentLength to avoid that the test takes
too much time and fails.

Signed-off-by: Simone Bordet <[email protected]>
sbordet added a commit that referenced this issue Nov 1, 2018
@sbordet
Merge pull request #2965 from eclipse/jetty-9.4.x-2796-http2_max_conc…
11abe53 
…urrent_streams

Issue #2796 - Max local stream count exceeded when request fails.
@joakime joakime changed the title Max local stream count exceeded when request fails HTTP/2 max local stream count exceeded when request fails Nov 6, 2018
@nehamunjal nehamunjal mentioned this issue Mar 26, 2019
Closed
sbordet added a commit that referenced this issue Aug 22, 2020
@sbordet
Issue #5147 - HTTP2 RoundRobinConnectionPool with maxUsage
0af5f67 
Reworked HTTP/2 release after an exchange is terminated.

Previously, the release was bound to 2 events: onStreamClosed(),
introduced for #2796, and exchangeTerminated().
Unfortunately, if the former happens before the latter and
closes the connection, the latter will see the exchange as
aborted, while in fact it was successful, causing what
reported in #5147, an AsynchronousCloseException.

Now, the release is always performed by the exchangeTerminated()
event. With respect to #2796, the stream is always already
closed by the time the exchangeTerminated() event fires (it
was not before).

Reworked the implementation of RoundRobinConnectionPool using
a lock and aggressively trying to open new connections.

A second fix is related to HttpDestination.release(Connection).
If the connection is closed for e.g. overuse, we need to trigger
the processing of queued requests via send(create: true).

Signed-off-by: Simone Bordet <[email protected]>
@sbordet sbordet mentioned this issue Aug 22, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@sbordet