confirmTransaction causes unhandled rejection

[kallerosenbaum]

Overview

Connection.confirmTransaction() sometimes causes unhandeled rejection if there is an error response from the server. We've seen this happen both with error 429 and 503.

This bug is very similar or exactly the same as the closed bug #3178. I've adapted the reproduction code from that bug here.

Steps to reproduce

Run this code:

// reproduce.ts
import { Connection, clusterApiUrl } from '@solana/web3.js';

class Reproduce {
    private connection: Connection;
    constructor() {
        this.connection = new Connection(clusterApiUrl('devnet'), {
            disableRetryOnRateLimit: true,
            commitment: 'confirmed'
        });
    }

    private async execute(blockhash): Promise<void> {
        console.log('execute');
        try {
            console.log('calling confirmTransaction');
            await this.connection.confirmTransaction({
                signature: '2hFGyLBL91u28EsBjEszm9qruYJm5MVdqBRe8f3FuLGWRJvKE1Dd2wiAwt9gCZJMMENGx17yN7A15c31h2oA6pTm',
                lastValidBlockHeight: blockhash.lastValidBlockHeight,
                blockhash: blockhash.blockhash,
            });
            console.log('confirmTransaction succeeded');
        } catch (error) {
            console.log('confirmTransaction failed', error);
        }
    }

    async run(): Promise<void> {
        let blockhash;
        try {
            blockhash = await this.connection.getLatestBlockhash('finalized');
        } catch (error) {
            console.log('getLatestBlockhash failed', error);
            return;
        }
        for (let i = 0; i < 50; i++) {
            await this.execute(blockhash);
        }
        console.log('DONE');
    }
}

const r = new Reproduce();
r.run();

After a number of iterations we would get a 429 (Too many requests) which causes the program to crash instead of being gracefully handled in the catch block. Here's the output from my test run:

execute
calling confirmTransaction
confirmTransaction failed TransactionExpiredBlockheightExceededError: Signature 2hFGyLBL91u28EsBjEszm9qruYJm5MVdqBRe8f3FuLGWRJvKE1Dd2wiAwt9gCZJMMENGx17yN7A15c31h2oA6pTm has expired: block height exceeded.
    at Connection.confirmTransactionUsingBlockHeightExceedanceStrategy (/home/me/unhandled-rejection/node_modules/@solana/web3.js/src/connection.ts:4059:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Connection.confirmTransaction (/home/me/unhandled-rejection/node_modules/@solana/web3.js/src/connection.ts:3870:14)
    at Reproduce.execute (/home/me/unhandled-rejection/src/command/reproduce.ts:17:13)
    at Reproduce.run (/home/me/unhandled-rejection/src/command/reproduce.ts:37:13) {
  signature: '2hFGyLBL91u28EsBjEszm9qruYJm5MVdqBRe8f3FuLGWRJvKE1Dd2wiAwt9gCZJMMENGx17yN7A15c31h2oA6pTm'
}
execute
calling confirmTransaction
confirmTransaction failed TransactionExpiredBlockheightExceededError: Signature 2hFGyLBL91u28EsBjEszm9qruYJm5MVdqBRe8f3FuLGWRJvKE1Dd2wiAwt9gCZJMMENGx17yN7A15c31h2oA6pTm has expired: block height exceeded.
    at Connection.confirmTransactionUsingBlockHeightExceedanceStrategy (/home/me/unhandled-rejection/node_modules/@solana/web3.js/src/connection.ts:4059:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Connection.confirmTransaction (/home/me/unhandled-rejection/node_modules/@solana/web3.js/src/connection.ts:3870:14)
    at Reproduce.execute (/home/me/unhandled-rejection/src/command/reproduce.ts:17:13)
    at Reproduce.run (/home/me/unhandled-rejection/src/command/reproduce.ts:37:13) {
  signature: '2hFGyLBL91u28EsBjEszm9qruYJm5MVdqBRe8f3FuLGWRJvKE1Dd2wiAwt9gCZJMMENGx17yN7A15c31h2oA6pTm'
}

... and several similar logs until ...

execute
calling confirmTransaction
confirmTransaction failed TransactionExpiredBlockheightExceededError: Signature 2hFGyLBL91u28EsBjEszm9qruYJm5MVdqBRe8f3FuLGWRJvKE1Dd2wiAwt9gCZJMMENGx17yN7A15c31h2oA6pTm has expired: block height exceeded.
    at Connection.confirmTransactionUsingBlockHeightExceedanceStrategy (/home/me/unhandled-rejection/node_modules/@solana/web3.js/src/connection.ts:4059:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Connection.confirmTransaction (/home/me/unhandled-rejection/node_modules/@solana/web3.js/src/connection.ts:3870:14)
    at Reproduce.execute (/home/me/unhandled-rejection/src/command/reproduce.ts:17:13)
    at Reproduce.run (/home/me/unhandled-rejection/src/command/reproduce.ts:37:13) {
  signature: '2hFGyLBL91u28EsBjEszm9qruYJm5MVdqBRe8f3FuLGWRJvKE1Dd2wiAwt9gCZJMMENGx17yN7A15c31h2oA6pTm'
}
execute
calling confirmTransaction
/home/me/unhandled-rejection/node_modules/@solana/web3.js/src/connection.ts:1698
        callback(new Error(`${res.status} ${res.statusText}: ${text}`));
                 ^


Error: 429 Too Many Requests:  {"jsonrpc":"2.0","error":{"code": 429, "message":"Too many requests for a specific RPC call"}, "id": "216f53d5-3b9b-4ff2-9283-318c2f01d867" } 

    at ClientBrowser.callServer (/home/me/unhandled-rejection/node_modules/@solana/web3.js/src/connection.ts:1698:18)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Node.js v22.5.1

Process finished with exit code 1

(The signature 2hFGyLBL91u28EsBjEszm9qruYJm5MVdqBRe8f3FuLGWRJvKE1Dd2wiAwt9gCZJMMENGx17yN7A15c31h2oA6pTm doesn't exist on devnet, but that's beside the point)

My guess is that it's this code in connection.ts that causes this: https://github.com/solana-labs/solana-web3.js/blame/maintenance/v1.x/src/connection.ts#L3937-L3985

That code creates an anonymous async function and calls it without awaiting it. When an exception (due to 429) is thrown from await this.getSignatureStatus(signature) there's nothing to catch it:

        (async () => {
          await subscriptionSetupPromise;
          if (done) return;
          const response = await this.getSignatureStatus(signature);
          if (done) return;
          if (response == null) {
            return;
          }
          const {context, value} = response;
          if (value == null) {
            return;
          }
          if (value?.err) {
            reject(value.err);
          } else {
            switch (commitment) {
              case 'confirmed':
              case 'single':
              case 'singleGossip': {
                if (value.confirmationStatus === 'processed') {
                  return;
                }
                break;
              }
              case 'finalized':
              case 'max':
              case 'root': {
                if (
                  value.confirmationStatus === 'processed' ||
                  value.confirmationStatus === 'confirmed'
                ) {
                  return;
                }
                break;
              }
              // exhaust enums to ensure full coverage
              case 'processed':
              case 'recent':
            }
            done = true;
            resolve({
              __type: TransactionStatus.PROCESSED,
              response: {
                context,
                value,
              },
            });
          }
        })();

Description of bug

This can cause production code to crash unexpectedly. A workaround could be to add the following to your program (but be cautions because the application is in an undefined state):

process.on('unhandledRejection', (error) => {
    // Log the error and don't throw any error from here.
});