NoMoreElementsNeeded Exception on Upgrade to Akka HTTP 10.1.12 with Akka 2.5.30

[aisven]

In a stable service of our platform based on Scala 2.12.10, Akka, Akka HTTP and alpakka Cassandra, we have a WebSocket route that allows OAuth2-based clients to receive new values of 1 to n selected time series as these values come in via IoT and reach our main time series database.

Our project-local test of this route is based on Akka HTTP Testkit. It also tests the WebSocket route using the Akka HTTP Web Socket client-side just as in this example in the documentation: https://doc.akka.io/docs/akka-http/current/client-side/websocket-support.html#singlewebsocketrequest

The Problem: We just now encountered a problem during maintenance of this service, as described in the following. If we update Akka HTTP from 10.1.11 to 10.1.12 and Akka from 2.5.27 to 2.5.30 we get the following Exception error message during the execution of several of the test cases of the WebSocket route: "akka.stream.SubscriptionWithCancelException$NoMoreElementsNeeded$ was thrown"

I tried to find the root cause of this, and thereby saw it could have to do with something new from Akka 2.6 recently having been back-ported to Akka 2.5 possibly also to ease a later migration of a user-land projects from 2.5 to 2.6.

As @johanandren said on gitter, "The exception is part of the new support for distinguishing cancellation and failure coming upstream." Thanks for this information and for motivating me to open an Issue.

Due to time constraints I do not have more insights and also no shareable reproducer. I suggest to test the example from https://doc.akka.io/docs/akka-http/current/client-side/websocket-support.html#singlewebsocketrequest within an Akka HTTP Testkit scenario with this version stack and see if you get any problems with that. It would also be nice if you could then share this test if it works, so I could compare it to mine.

[johanandren]

Another mention of the same from gitter (@joroKr21):

I saw the exception in production only (thankfully recoverable scenario), no tests and no staging environment uncovered it. So basically impossible to reproduce for me. But it happened after update of akka-http from 10.1.11 to 10.1.12 (akka version fixed to 2.6.5).

[jrudolph]

Thanks for the reports, @IOSven and @joroKr21.

You can try increasing akka.http.client.stream-cancellation-delay and see if that helps. In general, the fixes were needed to prevent that streams complete without error when, in fact, there was an error. So it could be a bug in our code but it could also be a bug/unexpected condition in your code or tests that was masked before.

We have a test suite that passes, without any additional pointers, we would just be digging in the dark. So, if you have any of the following information, it would make it more likely that someone can look into it:

• code under test
• test code that fails
• debug logging when it fails (enable akka.http.client.log-unencrypted-network-bytes = 1000, same for the server side if affected)
• reproducer

[aisven]

@jrudolph I see, I am afraid I currently cannot provide any of these. However I have two questions for now.

1. A conceptual question, does akka.stream.SubscriptionWithCancelException$NoMoreElementsNeeded$ stand for a situation in which a subscriber (asynchronously) subscribes, or is still subscribed, while a cancellation has been issued or propagated already?

2. Would you really be searching in the dark... in other words, do you already have a test case that covers the code scenario documented in https://doc.akka.io/docs/akka-http/current/client-side/websocket-support.html#singlewebsocketrequest ? If not, I am convinced it would be worth if you would add such a test case to the suite, as I suggested at the end of the Issue description.

[jrudolph]

1. A conceptual question, does `akka.stream.SubscriptionWithCancelException$NoMoreElementsNeeded$` stand for a situation in which a subscriber (asynchronously) subscribes, or is still subscribed, while a cancellation has been issued or propagated already?

It's the default cause if downstream has cancelled. The cause will only be a different one, if a more explicit one has been given when calling cancel or when the cancellation was automatic because a stage was terminated because of an error.

[jrudolph]

2\. Would you really be searching in the dark... in other words, do you already have a test case that covers the code scenario documented in [doc.akka.io/docs/akka-http/current/client-side/websocket-support.html#singlewebsocketrequest](https://doc.akka.io/docs/akka-http/current/client-side/websocket-support.html#singlewebsocketrequest) ? If not, I am convinced it would be worth if you would add such a test case to the suite, as I suggested at the end of the Issue description.

[info] running example.SingleWebSocketRequest 
Success(Done)
hello world!
closed

[joroKr21]

Just hit this again with Alpakka S3 - downloading a file and processing it in a streaming fashion threw this exception after ~10 seconds. One of the issues is that there is not even a stack trace so I have no clue who called cancel to be able to debug and try to reproduce it.

[joroKr21]

I also found this thread: https://discuss.lightbend.com/t/about-nomoreelementsneeded-exception/8599 - sounds like the same problem

[joroKr21]

You can try increasing akka.http.client.stream-cancellation-delay and see if that helps. In general, the fixes were needed to prevent that streams complete without error when, in fact, there was an error. So it could be a bug in our code but it could also be a bug/unexpected condition in your code or tests that was masked before.

Thanks @jrudolph that helps - 1 second works in my case.

Although I must say - I don't understand why the producer is cancelling the stream at all - that should be at the consumer's discretion. I should be able to use the stream for hours if I want to.

[mdedetrich]

I am also getting this error and it only occurs in github actions which is likely due to it having a much faster network connection. Increasing it to 1 second help.

@jrudolph Does it make sense to increase the default akka.http.client.stream-cancellation-delay to 1 second since that appears to be a lot more reasonable?

[jrudolph]

Maybe it would make sense indeed. In the common case, the whole idea behind the cancellation delay is that cancellation is not needed at all because failure or completion will be reported through the forward channel of a bidirectional setup.

Only in the case where this forward signal is missing the cancellation timeout will eventually trigger. In that case, the timeout applies to every connection between stream stages so the eventual cleanup until the whole stream is brought down might take much longer.

[mdedetrich]

PR created at #4038

[btomala]

I got the same error with akka 2.6.18 and akka-http 10.2.9 and akka-stream-alpakka-google-cloud-storage 3.0.4

The error occurs under a few conditions when the stream is slow, stream-cancellation-delay is small and Sink is an upload to the Google bucket.

Why does it happen? When I increase stream-cancellation-delay it's getting better. However, when the data set become bigger I have to increase the delay respectively to process the job successfully.

Is it possible to set it to infinity? What are the consequences of setting it to a significant number?