Check inactive WebSocket/SockJS sessions before they're connected on the STOMP level [SPR-11884]

[spring-projects-issues]

Prashant Deva opened SPR-11884 and commented

This is referencing the issue I wsa discussing with Rossen over email during the begining of the week.

Its reproducible on both Tomcat 7.0.53 and 8.0.8.

The number of files created by Spring keeps on increasing if there are a high number of connections (100s or 1000s). Increasing limit using ulimit is of no use since the number of files created will keep growing till the limit is hit, no matter what limit is set.

The issue is not visible when there are only a small number of connections.

I dont know of a good way to simulate hundreds or thousands of users so I have given Rossen full access to our production servers (over email) to reproduce the issue, including the source code of our app.

This is a blocker since our app crashes within minutes of starting and there is nothing we can do to keep it up

---

Affects: 4.0.5

Referenced from: commits 2b1ff4c, 40c203c, d18fc53, 113fd11, 7dc2b29, 86de416, 618771d, 3af488a, a3fa9c9

[spring-projects-issues]

Rossen Stoyanchev commented

From initial investigation it appears the count of connections on the Tomcat side is about 10x the count of connections to the STOMP broker relay side (they should be about the same) as shown through lsof -p <pid> -i -n -P -a and then grepping for the STOMP broker TCP port (61613 by default) orand the Tomcat server id and port (e.g. 127.0.0.1:80).

Early indications point to an issue on the Tomcat side but it's not clear what the cause is yet. The access logs show just under 2 million websocket requests made with the vast majority using the WebSocket transport and much smaller numbers (50-100K) of xhr_streaming and xhr requests.

[spring-projects-issues]

Rossen Stoyanchev commented

A more detailed investigation showed WebSocket/SockJS sessions that manage to get established successfully at first but never go as far as connecting on the STOMP level. This is often an indication of proxy issues but can also be a slow network.

For client sessions that do connect on the STOMP level, when a heartbeat interval is negotiated on startup, the STOMP broker automatically closes connections after a period of inactivity (or slow activity). However for sessions that never connect on the STOMP level there is no such protection.

To deal with this, a change has been introduced to check for WebSocket/SockJS sessions that have never received any messages within 60 seconds of connecting and close them. This is effectively makes up for the lack of heartbeats before the session connects on the level of STOMP.

Thanks Prashant Deva for letting me work with you on this and test it out.