Jetty is not creating new connection after receiving graceful goaway

[mdsuhaibsa]

Jetty version(s)
11.0.11

Description

1. Our application has received graceful GOAWAY frame(last stream ID = 2147483647) from destination dest1.

2. After receiving this GOAWAY frame from destination dest1, jetty marks the connection status as REMOTELY_CLOSED to indicate that the remote server (dest1) will not process any further streams.

3. However, when we attempt to send a new message to the same destination (dest1), Jetty is not creating a new connection as expected.

4. Instead, Jetty is raising an exception with the message "session closed," indicating that the connection is considered closed, preventing the creation of a new connection to dest1.

Attaching the logs for reference:
Jetty Logs.txt

[sbordet]

This seems a duplicate of #8678, which has been fixed.

The logs you provide are vastly incomplete: we need the full DEBUG logs for logging category "org.eclipse.jetty".

Please review the discussion in #8678, and report back whether you have the same or a different use case.
Do you happen to hit the unavoidable race described in #8678?

If you have a different condition, then we need the full DEBUG logs.

[mdsuhaibsa]

Hi @sbordet ,
Thanks for your response.

I don't see any race condition here.

Jetty client received a graceful GOAWAY frame at 14:39:02.890+0000 from the server, prompting Jetty to label the connection as REMOTELY_CLOSED at the same timestamp. Subsequently, when attempting to send a message at 14:39:07.071+0000 i.e. after 5 seconds to the same server, Jetty selected the same connection which is in REMOTELY_CLOSED state, without initiating a new connection.

Full logs attached. Some important log snippets pasted below:
Jetty_debug_logs.log

{"instant":{"epochSecond":1704206347,"nanoOfSecond":73853816},"thread":"thread-2","level":"DEBUG","loggerName":"org.eclipse.jetty.client.AbstractConnectionPool","message":"Activated MultiEntry@65906d8a{ACTIVE,usage=2,multiplex=1,pooled=HttpConnectionOverHTTP2@31c001a1(closed=false)[HTTP2ClientSession@13bd7766{local:/10.191.0.502:55214,remote:dest1.org/10.22.11.19:80,sendWindow=268431768,recvWindow=16776911,state=[streams=0,REMOTELY_CLOSED,goAwayRecv=GoAwayFrame@7d8f9433{2147483647/no_error/},goAwaySent=null,failure=null]}]} [@66a09b34[inUse=1,size=1,max=4,closed=false]","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":318,"threadPriority":5,"messageId":"b23026ae-8e0f-4abc-a319-a60d90b90559","messageTimestamp":"2024-01-02T14:39:07.073+0000","namespace":"app-21","node":"signalling-instance1-numa0-84b8f4f8bc-64kvd","pod":"app1-bd9c6c99d-shcpw","subsystem":"router","instanceType":"prod","processId":"1"}]

After this IllegalStateException exception comes since the connection is in REMOTELY_CLOSED state.

[mdsuhaibsa]

Hi @sbordet ,
Could you please provide an update?
If you require additional information or assistance, please let me know.
Thanks,
Mohamed Suhaib.

[sbordet]

@mdsuhaibsa this issue has been fixed in 10.0.13/11.0.13 and following.

Please update your Jetty to the latest version (11.0.19) and report back if you still see the issue.