cf ssh stops after 2GB transferred

[saliceti]

Command

• Any cf ssh command involving uploading more than 2GB data
• Ex: dd if=/dev/zero bs=1024k count=5000 | pv | ./cf ssh <app-name> -c "cat > /dev/null"
  (pv is only there to show progress and amount of data transferred)

What occurred

FAILED 0:00:35 [60.8MB/s] [                                                                                                            <=>                                               ]
Error: wait: remote command exited without exit status or exit signal
   2GB 0:00:35 [57.5MB/s] [                                                                                                               <=>                                            ]     

What you expected to occur

5000+0 records in0.7MB/s] [                                     <=>                                                                                                                      ]
5000+0 records out
5242880000 bytes (5.2 GB) copied, 88.5242 s, 59.2 MB/s
4.88GB 0:01:28 [56.5MB/s] [                                  <=>                                                                                                                         ]

CLI Version

• Works with 6.23.1
• Fails with 6.24.0
• Fails with 6.25.0

CC API Endpoint Version

• API version: 2.75.0
• Patched diego 1.8.1 version with updated crypto library (see below)

CF Trace

CF_TRACE doesn't look interesting as the ssh request is not logged.

Platform & Shell Details

Same behaviour on:

• Ubuntu 14.04.5 LTS
• MacOS 10.12.3

Any other relevant information

This was discovered when investigating https://github.com/cloudfoundry/diego-ssh/issues/31
The 1GB issue was solved by updating the crypto library in diego-release. But now we're hitting this error that looks related to cf cli.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/142139109

The labels on this github issue will be updated when the story is started.

[dkoper]

@saliceti Thanks for all those details!
We didn't make any changes around the cf ssh command, but we did start building with Golang 1.7.5 in cf CLI 6.24.0.
If you have a patched diego environment, any chance you could build the 6.25.0 CLI with Golang 1.7.4 and check if that still fails?

[saliceti]

Still fails with 1.7.4:

# dd if=/dev/zero count=2500 bs=1024k | pv | ./cli ssh healthcheck -c "cat > /dev/null"
panic: ssh: no key material for msgNewKeys                                         ]

goroutine 899 [running]:
panic(0xb7d2e0, 0xc420371200)
	/Users/colin/.gvm/gos/go1.7.4/src/runtime/panic.go:500 +0x1a1
code.cloudfoundry.org/cli/vendor/golang.org/x/crypto/ssh.(*connectionState).writePacket(0xc42000c268, 0xc4203c46c0, 0x108dd60, 0xc420017410, 0xc420790000, 0x8009, 0x8009, 0x0, 0x0)
	/Users/colin/.gvm/pkgsets/go1.7.4/cf-cli/src/code.cloudfoundry.org/cli/vendor/golang.org/x/crypto/ssh/transport.go:163 +0x226
code.cloudfoundry.org/cli/vendor/golang.org/x/crypto/ssh.(*transport).writePacket(0xc42000c200, 0xc420790000, 0x8009, 0x8009, 0x0, 0x0)
	/Users/colin/.gvm/pkgsets/go1.7.4/cf-cli/src/code.cloudfoundry.org/cli/vendor/golang.org/x/crypto/ssh/transport.go:144 +0x77
code.cloudfoundry.org/cli/vendor/golang.org/x/crypto/ssh.(*handshakeTransport).pushPacket(0xc420784140, 0xc420790000, 0x8009, 0x8009, 0x0, 0x0)
	/Users/colin/.gvm/pkgsets/go1.7.4/cf-cli/src/code.cloudfoundry.org/cli/vendor/golang.org/x/crypto/ssh/handshake.go:206 +0x51
code.cloudfoundry.org/cli/vendor/golang.org/x/crypto/ssh.(*handshakeTransport).kexLoop(0xc420784140)
	/Users/colin/.gvm/pkgsets/go1.7.4/cf-cli/src/code.cloudfoundry.org/cli/vendor/golang.org/x/crypto/ssh/handshake.go:286 +0x303
created by code.cloudfoundry.org/cli/vendor/golang.org/x/crypto/ssh.newClientTransport
	/Users/colin/.gvm/pkgsets/go1.7.4/cf-cli/src/code.cloudfoundry.org/cli/vendor/golang.org/x/crypto/ssh/handshake.go:113 +0x2bb
   2GB 0:01:49 [18.7MB/s] [          <=>                                           ]