IllegalStateException when last write fails #11370
Assignees
Labels
Bug
For general bugs on Jetty side
Comments
|
Do you happen to have a reproduction test case? |
|
I can confirm now that this also happens only when I use the GZip Filter like in the original issue. I can also reproduce easily in my application. The application is rendering a JSP which references various static resources. When I just keep the reloading the page by keeping CTRL+F5 pressed in Firefox the error occurs all time. Now all the time doesn't mean on each request but within a few seconds I get it multiple times. In the firefox console i can see the NS_BINDING_ABORTED errors but not each of them causes this error. In this test case it seems happens only for the theme.css and not for the JS files. But the first time I saw the error in my logs it was for the JS files. So this doesn't mean much. And even for the theme.css there are more aborted requests in the browser than errors in the log. So a NS_BINDING_ABORTED doesn't automatically trigger the error
It also actually quite sensitive to timing. With a longer network console log that or if the console fills almost the whole screen it seems to be much be difficult to reproduce (running Windows on i7 11800) |
|
The See: https://web.archive.org/web/20091005053108/http://markmail.org/message/m6z77uoixf3qu7u6 Constantly (and aggressively) refreshing via the browser is sure to trigger a variety of failed network read / writes from the previous requests that were closed by the browser. Out of curiosity, what component in your webapp is serving those resources? |
|
Actually there was a mix of static and dynamicl content in the previous example. The theme.css I mention is actually a user configurable CSS and served from a servlet filter. But other CSS, JS etc is really static content and served by the Jetty DefaultServlet. The first time I saw this error in the logs it was for such static content but when I tried to reproduce it was reported only for the dynamic content. Don't ask me why. But just to make sure i switched back to the default theme which is a static resource and run then brute force test again. And it is as easy to reproduce as before with the static resources. What i noticed is that when it happens during the brute force test it is reported only for a single file So i changed the order of the references but that didn't change anything. But when I made one of the other files noticeable bigger so that it was even bigger than the jquery.min it started to fail for both files So I started playing with the file size to see whether that really means something. And guess what: up to 32.767 bytes it seems to be fine. But once it reaches 32.768 or more the error starts to appear. I also made a test by referencing two versions of the file in the JSP, one 32.767 bytes and one 32.768 and only the 32.768 version fails There are is no Expires headers. But in the end that would wouldn't change much about the underlying problem when as the brute force test is using CTRL-F5 to force a complete reload. It doesn't happen with a standard reload even though FF still logs NS_BINDING_ABORTED even on a simple reload with just F5 |
|
Small addition: the short version of the file ends with sdfsdfasdkjfhgskj11 and the long one with sdfsdfasdkjfhgskj111 |
sbordet
changed the title
sbordet
added a commit
that referenced
this issue
Feb 22, 2024
Removed the call to `ServletChannel.abort()` from the write callback. As the write was issued from `ServletChannel.handle()` case COMPLETE, it was eventually calling `ServletChannelState.completed(Throwable)`, which is expecting the requestState to be COMPLETING. However, calling `abort()` would set the requestState to COMPLETED, causing the IllegalStateException. There should be no need to call `abort()` from the callback of failed writes, since failing the various callbacks should be enough, eventually failing the `HttpStream`, which would take care of tearing down the connection (HTTP/1) or the stream (HTTP/2+). Signed-off-by: Simone Bordet <[email protected]>
|
I was able to reproduce the issue. It does not depend on gzip, it happens for failed writes also without gzip. The fix is in #11439, along with analysis and fix. |
sbordet
added a commit
that referenced
this issue
Feb 23, 2024
Now aborting the response from ServletChannelState.completed(Throwable). Fixed SizeLimitHandler exception message. Signed-off-by: Simone Bordet <[email protected]>
sbordet
added a commit
that referenced
this issue
Feb 23, 2024
Fixed SizeLimitHandlerTest. Signed-off-by: Simone Bordet <[email protected]>
|
I am also seeing this warning after upgrading 11.0.17 -> 12.0.6 and question is if this is log noise that can be muted until the fix is released OR if it has some real negative impact on user requests? |
|
It has no negative impact. You can tune the logger for that particular class. The fix will be in the upcoming 12.0.7 release. |
sbordet
added a commit
that referenced
this issue
Feb 26, 2024
Removed the call to `ServletChannel.abort()` from the write callback. As the write was issued from `ServletChannel.handle()` case COMPLETE, it was eventually calling `ServletChannelState.completed(Throwable)`, which is expecting the requestState to be COMPLETING. However, calling `abort()` would set the requestState to COMPLETED, causing the IllegalStateException. There should be no need to call `abort()` from the callback of failed writes, since failing the various callbacks should be enough, eventually failing the `HttpStream`, which would take care of tearing down the connection (HTTP/1) or the stream (HTTP/2+). Now aborting the response from ServletChannelState.completed(Throwable). Fixed SizeLimitHandler exception message. Signed-off-by: Simone Bordet <[email protected]>
github-project-automation
bot
moved this from 🏗 In progress
to ✅ Done
in Jetty 12.0.7 - FROZEN
|
I am also seeing same issue after upgrading jetty 11.0.17 -> 12.0.5. the same issue is in jetty 12.0.5 also java.lang.IllegalStateException: s=HANDLING rs=COMPLETED os=ABORTED is=IDLE awp=false se=false i=false al=0 How do i handle it? This exception is coming from jetty. |
|
@at01071988 please update to 12.0.7. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jetty version(s)
12.0.6
Description
This is almost the same as in #11095 but this time the call stack is different. But while it was this call stack there
java.lang.IllegalStateException: s=HANDLING rs=COMPLETED os=ABORTED is=IDLE awp=false se=false i=false al=0
at org.eclipse.jetty.ee10.servlet.ServletChannelState.completed(ServletChannelState.java:1052) ~[jetty-ee10-servlet-12.0.5.jar:12.0.5]
at org.eclipse.jetty.ee10.servlet.ServletChannel.lambda$handle$3(ServletChannel.java:553) ~[jetty-ee10-servlet-12.0.5.jar:12.0.5]
at org.eclipse.jetty.util.Callback$3.succeeded(Callback.java:162) ~[jetty-util-12.0.5.jar:12.0.5]
at org.eclipse.jetty.ee10.servlet.HttpOutput.complete(HttpOutput.java:424) ~[jetty-ee10-servlet-12.0.5.jar:12.0.5]
...
I got this now with a different call stack. Probably because the requests were cancelled by the browser
java.lang.IllegalStateException: s=HANDLING rs=COMPLETED os=ABORTED is=IDLE awp=false se=false i=false al=0
at org.eclipse.jetty.ee10.servlet.ServletChannelState.completed(ServletChannelState.java:1052)
at org.eclipse.jetty.util.Callback$3.failed(Callback.java:168)
at org.eclipse.jetty.ee10.servlet.HttpOutput.onWriteComplete(HttpOutput.java:252)
at org.eclipse.jetty.ee10.servlet.HttpOutput$WriteCompleteCB.failed(HttpOutput.java:1760)
at org.eclipse.jetty.ee10.servlet.HttpOutput$CompleteWriteCompleteCB.failed(HttpOutput.java:1777)
at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.accept(ContextHandler.java:1179)
at org.eclipse.jetty.server.handler.ContextResponse$1.failed(ContextResponse.java:47)
at org.eclipse.jetty.util.IteratingNestedCallback.onCompleteFailure(IteratingNestedCallback.java:58)
at org.eclipse.jetty.server.handler.gzip.GzipResponseAndCallback$GzipBufferCB.onCompleteFailure(GzipResponseAndCallback.java:332)
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:335)
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:231)
at org.eclipse.jetty.server.handler.gzip.GzipResponseAndCallback.gzip(GzipResponseAndCallback.java:157)
at org.eclipse.jetty.server.handler.gzip.GzipResponseAndCallback.write(GzipResponseAndCallback.java:135)
at org.eclipse.jetty.server.Response$Wrapper.write(Response.java:754)
at org.eclipse.jetty.server.handler.ContextResponse.write(ContextResponse.java:56)
at org.eclipse.jetty.ee10.servlet.ServletContextResponse.write(ServletContextResponse.java:288)
at org.eclipse.jetty.ee10.servlet.HttpOutput.channelWrite(HttpOutput.java:204)
at org.eclipse.jetty.ee10.servlet.HttpOutput.complete(HttpOutput.java:435)
at org.eclipse.jetty.ee10.servlet.ServletContextResponse.completeOutput(ServletContextResponse.java:212)
at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:562)
at org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:464)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:571)
at org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:703)
at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:763)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:597)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:181)
at org.eclipse.jetty.server.Server.handle(Server.java:179)
at org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:619)
at org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:410)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:410)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:971)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1201)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1156)
at java.base/java.lang.Thread.run(Thread.java:833)
I haven't tried to verify whether it was really caused by the GZip filter too in this case yet as that is not as easy to reproduce
The text was updated successfully, but these errors were encountered: