Connect hangs before KEXDH_GEX_REPLY in arm container for some remotes #842

te-online · 2019-10-24T21:21:21Z

Summary

This issue only seems to happen when connecting to some remotes and only when running ssh2 on arm systems. I'm referring to the latest version 0.8.5 here.

I tested 3 remotes and ssh2 won't connect to 2 of them, while 1 works just fine. (This might not be enough data to identify the problem, though.) Unfortunately, I don't have access to the SSH configuration of those remote servers and I want to try and support all of them in my app. Is there anything I can do to fix this?

Non-working remotes identify as: SSH-2.0-OpenSSH_7.1 and SSH-2.0-OpenSSH_7.5 FreeBSD-20170903
The working remote identifies as: SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3.

Negotiated key format, cipher and HMAC algorithm are identical, no matter which remote.

Steps to reproduce:

Run docker run -it arm32v6/node:12-buster (I also tried with arm32v6/node:10-alpine and other arm containers)
Run cd ~ && npm add ssh2
Create a file server.js with a basic example

var conn = new Client();
conn.on('ready', function() {
	console.log('Client :: ready');
  conn.exec('uptime', function(err, stream) {
    if (err) throw err;
    stream.on('close', function(code, signal) {
      console.log('Stream :: close :: code: ' + code + ', signal: ' + signal);
      conn.end();
    }).on('data', function(data) {
      console.log('STDOUT: ' + data);
    }).stderr.on('data', function(data) {
      console.log('STDERR: ' + data);
    });
  });
}).connect({
	host: 'example.org',
	user: 'your-username',
	password: 'your-password',
	debug: console.log
});

Results in this debug output and the error event

DEBUG: Local ident: 'SSH-2.0-ssh2js0.4.6'
DEBUG: Client: Trying [redacted] on port 22 ...
DEBUG: Client: Connected
DEBUG: Parser: IN_INIT
DEBUG: Parser: IN_GREETING
DEBUG: Parser: IN_HEADER
DEBUG: Remote ident: 'SSH-2.0-OpenSSH_7.1'
DEBUG: Outgoing: Writing KEXINIT
DEBUG: Parser: IN_PACKETBEFORE (expecting 8)
DEBUG: Parser: IN_PACKET
DEBUG: Parser: pktLen:1036,padLen:10,remainLen:1032
DEBUG: Parser: IN_PACKETDATA
DEBUG: Parser: IN_PACKETDATAAFTER, packet: KEXINIT
DEBUG: Comparing KEXINITs ...
DEBUG: (local) KEX algorithms: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
DEBUG: (remote) KEX algorithms: [email protected],diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1
DEBUG: KEX algorithm: diffie-hellman-group-exchange-sha256
DEBUG: (local) Host key formats: ssh-rsa,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
DEBUG: (remote) Host key formats: ssh-rsa,[email protected],ssh-ed25519,[email protected]
DEBUG: Host key format: ssh-rsa
DEBUG: (local) Client->Server ciphers: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm,[email protected],aes256-gcm,[email protected]
DEBUG: (remote) Client->Server ciphers: [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc
DEBUG: Client->Server Cipher: aes128-ctr
DEBUG: (local) Server->Client ciphers: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm,[email protected],aes256-gcm,[email protected]
DEBUG: (remote) Server->Client ciphers: [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc
DEBUG: Server->Client Cipher: aes128-ctr
DEBUG: (local) Client->Server HMAC algorithms: hmac-sha2-256,hmac-sha2-512,hmac-sha1
DEBUG: (remote) Client->Server HMAC algorithms: [email protected],[email protected],[email protected],[email protected],hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,[email protected],[email protected],hmac-sha1
DEBUG: Client->Server HMAC algorithm: hmac-sha2-256
DEBUG: (local) Server->Client HMAC algorithms: hmac-sha2-256,hmac-sha2-512,hmac-sha1
DEBUG: (remote) Server->Client HMAC algorithms: [email protected],[email protected],[email protected],[email protected],hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,[email protected],[email protected],hmac-sha1
DEBUG: Server->Client HMAC algorithm: hmac-sha2-256
DEBUG: (local) Client->Server compression algorithms: none,[email protected],zlib
DEBUG: (remote) Client->Server compression algorithms: none,[email protected]
DEBUG: Client->Server compression algorithm: none
DEBUG: (local) Server->Client compression algorithms: none,[email protected],zlib
DEBUG: (remote) Server->Client compression algorithms: none,[email protected]
DEBUG: Server->Client compression algorithm: none
DEBUG: Outgoing: Writing KEXDH_GEX_REQUEST
DEBUG: Parser: IN_PACKETBEFORE (expecting 8)
DEBUG: Parser: IN_PACKET
DEBUG: Parser: pktLen:532,padLen:8,remainLen:528
DEBUG: Parser: IN_PACKETDATA
DEBUG: Parser: IN_PACKETDATAAFTER, packet: KEXDH_GEX_GROUP
DEBUG: Outgoing: Writing KEXDH_GEX_INIT
DEBUG: Parser: IN_PACKETBEFORE (expecting 8)
events.js:174
      throw er; // Unhandled 'error' event
      ^

Error: Timed out while waiting for handshake
    at Timeout._onTimeout (/root/node_modules/ssh2/lib/client.js:695:19)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
Emitted 'error' event at:
    at Timeout._onTimeout (/root/node_modules/ssh2/lib/client.js:697:14)
    at ontimeout (timers.js:436:11)
    [... lines matching original stack trace ...]
    at Timer.processTimers (timers.js:223:10)

What I tried already:

Add readyTimeout: 99999
Add algorithms: { ciphers: ['aes128-ctr', 'aes192-ctr', 'aes256-ctr'] }
Use a non-arm container, e.g. node:12-alpine: It works as expected

Is there anything else I can try? Are these remote configurations simply not supported? I'm trying to run a node server using ssh2 and ssh2-sftp-client on a RaspberryPi in the mentioned container.

The text was updated successfully, but these errors were encountered:

mscdex · 2019-10-24T22:41:22Z

Could you be running into this node issue?

te-online · 2019-10-25T17:42:17Z

Indeed, the results of the command mentioned in that issue is interesting:

/ # time node -e "require('crypto').getDiffieHellman('modp18')"
real	12m 10.54s
user	12m 0.72s
sys	0m 5.74s

It takes 12 minutes for this crypto method to run.

Do you think there's a solution to this issue, apart from waiting 12 minutes at each connection?

Edit: I could probably try installing an older version of OpenSSL in the container if that's feasible security-wise.

mscdex · 2019-10-25T20:07:42Z

You could try limiting the kex algorithms to exclude the specific group algorithms and see if that works better. For example, this would match the default kex algorithms being sent as of this writing, except with the 'group*' algorithms excluded:

ssh.connect({
  algorithms: {
    kex: [
      'ecdh-sha2-nistp256',
      'ecdh-sha2-nistp384',
      'ecdh-sha2-nistp521',
      'diffie-hellman-group-exchange-sha256',
    ],
  },
  // ...
});

Also, the 'ecdh-*' and 'group-exchange' algorithms use different crypto APIs, so you might want to test the speed of both types independently before including both.

te-online · 2019-10-27T17:37:30Z

Thanks a lot for your troubleshooting advice! Unfortunately, the remote server quits the connection when I remove diffie-hellman-group-exchange-sha256 from the list and that's apparently one of the “slow” algorithms. So there doesn't seem to be a match with some servers between “fast” algorithms and the list of algorithms accepted by the server.

For the time being, I'm just going to switch to a different architecture and accept this as a current limitation of arm devices. But maybe that issue will be solved in node or openssl at some point (if cryptographically possible).

cushdan · 2021-08-18T14:05:11Z

I ran into this issue (timeouts >3min) recently when trying to run the ssh2-sftp-client on an aws lambda function. Removing the slow algorithm has worked so far

 algorithms: {
      kex: {
        remove: [
          'diffie-hellman-group-exchange-sha256'
        ]
      }
    }

mattisdada · 2021-11-23T00:13:50Z

I ran into this issue (timeouts >3min) recently when trying to run the ssh2-sftp-client on an aws lambda function. Removing the slow algorithm has worked so far
 algorithms: {
      kex: {
        remove: [
          'diffie-hellman-group-exchange-sha256'
        ]
      }
    }

This note helped me immensely, it seems that this is a problematic kex in Lambda. May be worth a note in a README or FAQ or something?

My environment:

Node 14
AWS Lambda / Amazon Linux 2
Remote Server Kex: diffie-hellman-group1-sha1,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha256

My fix was to do: algorithms: { kex: ["diffie-hellman-group1-sha1", "diffie-hellman-group14-sha1"] }. The remove object key is not part of the typescript definition, so that may have been an old feature.

Hope this helps some other poor soul! :)

mscdex · 2021-11-23T01:34:27Z

The remove object key is not part of the typescript definition, so that may have been an old feature.

It's the other way around. v1.x added that feature, so the typescript definition is probably not up-to-date for v1.x.

mattisdada · 2021-11-23T22:45:07Z

The remove object key is not part of the typescript definition, so that may have been an old feature.

It's the other way around. v1.x added that feature, so the typescript definition is probably not up-to-date for v1.x.

Ah sorry about that, you're absolutely correct, woops! I still think the Lambda problem should still be highlighted somewhere though

39otrebla · 2022-05-08T14:07:43Z

I ran into this issue (timeouts >3min) recently when trying to run the ssh2-sftp-client on an aws lambda function. Removing the slow algorithm has worked so far
 algorithms: {
      kex: {
        remove: [
          'diffie-hellman-group-exchange-sha256'
        ]
      }
    }
This note helped me immensely, it seems that this is a problematic kex in Lambda. May be worth a note in a README or FAQ or something?

My environment:

Node 14

AWS Lambda / Amazon Linux 2

Remote Server Kex: diffie-hellman-group1-sha1,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha256

My fix was to do: algorithms: { kex: ["diffie-hellman-group1-sha1", "diffie-hellman-group14-sha1"] }. The remove object key is not part of the typescript definition, so that may have been an old feature.

Hope this helps some other poor soul! :)

This worked for me too. Using ssh2-sftp-client version 7.x.x on a custom-built ARM Docker image based on default Lambda image for Node14.

Thank you mate, you literally saved me hours of headache.

mrobinsonlumint · 2023-03-17T18:53:00Z

This thread was extremely helpful and saved me hours! Thank you!!
Out of curiosity, does anyone know why AWS Lambda cannot support this slow algorithm (diffie-hellman-group-exchange-sha256) ? Or future plans for it to be supported?

te-online closed this as completed Oct 27, 2019

ds3783 mentioned this issue Apr 20, 2023

Client times out after Received DH GEX Group #1290

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Connect hangs before KEXDH_GEX_REPLY in arm container for some remotes #842

Connect hangs before KEXDH_GEX_REPLY in arm container for some remotes #842

te-online commented Oct 24, 2019

mscdex commented Oct 24, 2019

te-online commented Oct 25, 2019 •

edited

Loading

mscdex commented Oct 25, 2019

te-online commented Oct 27, 2019

cushdan commented Aug 18, 2021

mattisdada commented Nov 23, 2021 •

edited

Loading

mscdex commented Nov 23, 2021

mattisdada commented Nov 23, 2021

39otrebla commented May 8, 2022

mrobinsonlumint commented Mar 17, 2023

Connect hangs before KEXDH_GEX_REPLY in arm container for some remotes #842

Connect hangs before KEXDH_GEX_REPLY in arm container for some remotes #842

Comments

te-online commented Oct 24, 2019

Summary

Steps to reproduce:

What I tried already:

mscdex commented Oct 24, 2019

te-online commented Oct 25, 2019 • edited Loading

mscdex commented Oct 25, 2019

te-online commented Oct 27, 2019

cushdan commented Aug 18, 2021

mattisdada commented Nov 23, 2021 • edited Loading

mscdex commented Nov 23, 2021

mattisdada commented Nov 23, 2021

39otrebla commented May 8, 2022

mrobinsonlumint commented Mar 17, 2023

te-online commented Oct 25, 2019 •

edited

Loading

mattisdada commented Nov 23, 2021 •

edited

Loading