Regression: Entity stream closed in version 2.4 when Connection: close header used

[Sovietaced]

I'm working on a project that heavily uses Restlet. Periodically we update our maven dependencies and I think I have discovered a regression introduced in Restlet 2.4.

After updating only Restlet from 2.3.12 to 2.4 we discovered that all of our POST request handlers were blowing up when trying to parse json from the entity stream since it was closed.

Caused by: java.io.IOException: Stream closed
	at java.io.PushbackInputStream.ensureOpen(PushbackInputStream.java:74)
	at java.io.PushbackInputStream.read(PushbackInputStream.java:166)
	at com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.ensureLoaded(ByteSourceJsonBootstrapper.java:522)
	at com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.detectEncoding(ByteSourceJsonBootstrapper.java:129)
	at com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.constructParser(ByteSourceJsonBootstrapper.java:246)
	at com.fasterxml.jackson.core.JsonFactory._createParser(JsonFactory.java:1315)
	at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:820)
	at org.projectfloodlight.db.service.internal.DataNodeJsonHandler.parseData(DataNodeJsonHandler.java:662)
	... 64 common frames omitted

After tooling around with curl I discovered that this issue was triggered whenever the Connection: close header was used in POST requests.

Thanks for your time.

[thboileau]

Hello @Sovietaced

I feel sorry for the late answer.
I wonder which HTTP server connector are you using: is it Jetty?

[Sovietaced]

Thanks for the reply @thboileau

We're using simple framework. Here is a peek at our maven dependencies.

<!-- Restlet version -->
<restlet.version>2.3.12</restlet.version>

<dependency>
      <groupId>org.restlet.jse</groupId>
      <artifactId>org.restlet</artifactId>
      <version>${restlet.version}</version>
</dependency>
<dependency>
      <groupId>org.restlet.jse</groupId>
      <artifactId>org.restlet.ext.simple</artifactId>
      <version>${restlet.version}</version>
</dependency>
<dependency>
      <groupId>org.restlet.jse</groupId>
      <artifactId>org.restlet.ext.slf4j</artifactId>
      <version>${restlet.version}</version>
</dependency>
<dependency>
      <groupId>org.restlet.jse</groupId>
      <artifactId>org.restlet.ext.jackson</artifactId>
      <version>${restlet.version}</version>
</dependency>

[CharliePF]

We have come across the same issue. It doesn't matter which connector is used our app used the Jetty connector. We have recently switched our app to use Under Tow and expose Restlet via the Servlet Adapter. All the connectors we have used have the same behaviour.

From playing around with the debugger it looks like, whats actually happening is Restlet is passing a null input stream to Jackson. The stream backing the PushBackInputStream in the stack trace is null. If the Connection: keep-alive header is set an instance of UnclosableInputStream is passed to Jackson.

Incidentally our app was using v2.4. However in my testing I can't seem to get this to work in v.2.3.12 either.

Incidentally we only found this in testing in our staging environment, since the K8 load balancer was configured to disable connection keep-alive. I am guessing this is causing failures for a lot of other users behind proxies.

[CharliePF]

Just to clarify my previous post. After trying lots of prior versions of Restlet. It does seem to be 2.4 that introduced this issue. 2.3.12 does indeed work fine.

I think the easiest solution for anyone suffering this is to roll back to 2.3.12, until this is fixed.

[01es]

Had exactly the same problem with Restlet 2.4.0 and Jetty 9.4.x behind HAProxy. Changing HAProxy option from http-server-close to http-keep-alive solved it for us.

[CharliePF]

The problem with doing that is that it just enables keep alive in the proxy. Most clients will request keep-alive, and so will now work. Theres likely to be a small subset of clients who aren't going to request keep alive. Users behind corporate proxies etc. In which case I believe the proxy will go back to making connection: close requests. Certainly the HTTP 1.1 spec says any client is within its rights to make these requests.

Until this is fixed it means for us and probably others, we can't use restlet > 2.4. We simply can't risk having clients who can't connect.

[ebner]

In my case this problem is triggered when using Apache HTTPD as reverse proxy. When accessing the Restlet application directly there is no problem.

[ebner]

Rolling back to 2.3.12 due this bug is problematic since this also means that we can't use Java 10 as it breaks method annotations with Restlet prior to 2.4.

[okocaoglu1]

I am having the same issue through Angular 6 proxy. Proxy is generating a Connection: close header, which causes the body of my proxied requests not get parsed. Restlet version 2.4 and Jetty version 9.4.11

[qsiebers]

I think I've found the cause of the problem:

adcd73b#diff-7afa0f2a1dc04e5097eeadff9d314bc3R195

A PushbackInputStream is used to check for presence of an entity, and is passed as the entity input stream. The changes in adcd73b cause this PushbackInputStream to be closed as part of the try-with-resources block.

[Tembrel]

Ooo, good catch! I guess the lesson here is that any use of try-with-resources should replace both a creation of a resource and a close of that resource.

[ebner]

This issue is tagged with release 2.4 but it is not included in the most recent changelog up the recently released 2.4.3 at https://restlet.talend.com/documentation/2.4/changelog. Does this mean that this issue didn't make it into 2.4.3 or was it not included in the changelog by mistake?

EDIT: Found the 2.4.3 tag on 2b70e87#diff-7afa0f2a1dc04e5097eeadff9d314bc3 so I assume that the changelog is incomplete.

[flatlanderer]

Experiencing this error even client having Connection: keep-alive header. What's the solution for this bug?