[web3.js] Infinite recursion caused by _updateSubscriptions

[Disperito]

Problem

I'm using the version 1.44.1 of @solana/web3.js, to develop an application with high-frequency calling.
And the connection between network on my machine and Solana' Network seems to be not so stable: it's often to throw error about transaction confirmation timeout
In this situation, it suddenly to fastly and infinitely throw errors like follow:

Here is the key log:

signatureSubscribe error for argument [
  '5R9WXhnHtz5umHHh9LpXXz9CSXjucTbrUPCJV2iQAkUgBX4jQ4UcQYRyta3LbBFZwuMGcKMaqCPiPkyuPT2JPKSr',
  { commitment: 'processed' }
] WebSocket is not open: readyState 0 (CONNECTING)

As you can see, the connection is frantically trying to build up subscriptions on TWO transaction signatures.

Then I found out, the problem was caused in function _updateSubscriptions() in connection.ts:

Proposed Solution

Maybe you team should fix the logic of error handling. Here's my humble opinion: if there are some subsciptions failing for too many times(whatever the reason is), they should be directly removed from subscriptions list.

[steveluscher]

Thanks for this report!

Other than spamming the error console, does this interfere with the operation of your app?

[Disperito]

Thanks for this report!

Other than spamming the error console, does this interfere with the operation of your app?

Of course. It caused infinite recursion and made CPU usage soar to 100%, so that my app cannot run as normal. @steveluscher

[gallynaut]

I was also seeing this issue on v1.47.3 and reverted to v1.43.5 and it went away.

[steveluscher]

Totally, I get it, @Disperito. Unbounded recursion is super bad news, no matter which way you slice it. What I was trying to get at is that there are two kinds of infinite recursion.

One leads to the JavaScript thread literally locking, and freezing the browser:

while (true) {
  // Browser's dead.
}

The other results in code running as often as it can, and high CPU usage, but doesn't lock the thread; other code can continue to run in the period between yields.

while (true) {
  // Between the time this yields and resumes, other work can get done on the JS thread.
  await someAsyncWork();
}

In any case, we need a general solution for this.

[steveluscher]

Can I ask you a question? Can you count the number of subscriptions that you're making to the RPC in total? I've seen situations where RPCs limit the number of open subscriptions you may hold. When you exceed a certain number, they murder your connection abruptly. See solana-labs/solana#25322 (comment).

[Disperito]

@Disperito. Unbounded recursion is super bad news, no matter which way you slice it. What I was trying to get at is that there are two kinds of infinite recursion.

One leads to the JavaScript thread literally locking, and freezing the browser:

Sorry I have no definite answer. But I have reviewed the logs near when this error occurs, I think there were only less than 10 subscriptions at the same time, even less.

I don't think the large number of subscriptions is the root cause of the bug. It is true that a large number of subscriptions can lead to poor connnection with RPCs, but the point is: your library should recognize this situation and do special handling, such as throwing an exception, or actively unsubscribing

[Jac0xb]

Still getting this error with @solana/web3.js@1.56.2 @steveluscher. Spammed logs until server was killed.

signatureSubscribe error for argument [
'{SOME_SIGNATURE}',
{ commitment: 'confirmed' }
] socket not ready

[steveluscher]

@Jac0xb If you have a reliable repro of this, can you debug it a bit to see if this is the root cause? solana-labs/solana#25578

[Jac0xb]

Not sure the repro steps but happens often with sendConfirmation and possibly log lived connection objects. Haven't seen it before. Happened in Node.js. Looks like there is an onSignature call in there