List files from remote directory fails with Serv-U

[comtel2000]

Hi, if I try to list files from a remote directory (sample: export/files) I get a exception:

net.schmizz.sshj.sftp.SFTPException: No such file.
    at net.schmizz.sshj.sftp.Response.error(Response.java:113)
    at net.schmizz.sshj.sftp.Response.ensurePacketTypeIs(Response.java:91)
    at net.schmizz.sshj.sftp.SFTPEngine.openDir(SFTPEngine.java:153)
    at net.schmizz.sshj.sftp.SFTPClient.ls(SFTPClient.java:61)
    at net.schmizz.sshj.sftp.SFTPClient.ls(SFTPClient.java:56)

But only with Serv-U (tested with: Serv-U_15.1.4.6,Serv-U_15.1.3.43) and SSHClient.newSFTPClient
and with StatefulSFTPClient sftp = new StatefulSFTPClient(engine.init()); it works very well.
I tested it also with other SFTP servers like CoreFTP, Windows embedded SFTP and both ways are ok.
Only with CoreFTP (mini-server 2.04) the SFTPEngine.close(); cause a TimeoutException all the time.

    try (SSHClient ssh = new SSHClient()) {
      ssh.addHostKeyVerifier(new PromiscuousVerifier());
      ssh.connect(host, port);
      ssh.authPassword(user, pwd);

      SFTPEngine engine = new SFTPEngine(ssh);
      StatefulSFTPClient sftp = new StatefulSFTPClient(engine.init());
      // auto close run into timeout exception with CoreFTP mini SFTP
      // try (StatefulSFTPClient sftp = new StatefulSFTPClient(engine.init())) {
      logger.info("found {} remote files", sftp.ls(incomingRemotePath).size());
    } catch (Exception e) {
      logger.error(e.getMessage(), e);
    }

    try (SSHClient ssh = new SSHClient()) {
      ssh.addHostKeyVerifier(new PromiscuousVerifier());
      ssh.connect(host, port);
      ssh.authPassword(user, pwd);

      SFTPClient sftp = ssh.newSFTPClient();
      // newSFTPClient doesn't work with Serv-U SFTP
      logger.info("found {} remote files", sftp.ls(incomingRemotePath).size());
    } catch (Exception e) {
      logger.error(e.getMessage(), e);
    }

[hierynomus]

Which version of SSHJ are you using?

So let me get this straight, there are 2 different problems here:

• Serv-U FTP does not work with sshClient.newSFTPClient(), but it works with new StatefulSFTPClient(..)?
• Using CoreFTP, you get a TimeoutException when SFTPEngine.close() is being called.

Do you have some additional (DEBUG/TRACE) logging for the exceptions?

[comtel2000]

Yes, there are 2 different issues, but I guess the TimeoutException looks like a CoreFTP issue.
I've tested it with the latest version of SSHJ 0.17.2 and previous 0.16.0.
Here is the Serv-U log with newSFTPClient():

[31] Tue 30Aug16 10:01:03 - (000008) SSH2_MSG_USERAUTH_SUCCESS: Login erfolgreich
[30] Tue 30Aug16 10:01:03 - (000008) SSH_FXP_INIT: Client Version 3 (SSHJ_0_17_2)
[31] Tue 30Aug16 10:01:03 - (000008) SSH_FXP_VERSION: Server Version 3
[30] Tue 30Aug16 10:01:03 - (000008) {0x00000001} SSH_FXP_OPENDIR
[31] Tue 30Aug16 10:01:03 - (000008) {0x00000001} SSH_FX_NO_SUCH_FILE: No such file.
[31] Tue 30Aug16 10:01:03 - (000008) SSH_MSG_DISCONNECT: client has requested a disconnect.  Reason code: 11
[02] Tue 30Aug16 10:01:03 - (000008) Sitzung geschlossen
[02] Tue 30Aug16 10:01:03 - (000008) Benutzer "csdb_inte_0016" abgemeldet
[02] Tue 30Aug16 10:09:53 - (000006) Inaktivitäts-Timeout der Sitzung
[02] Tue 30Aug16 10:09:53 - (000006) Sitzung geschlossen

and with StatefulSFTPClient:

[02] Tue 30Aug16 11:06:27 - User has connected from 127.0.0.1 (local address 127.0.0.1, port 22)
[31] Tue 30Aug16 11:06:27 - (000019) SSH2_MSG_USERAUTH_SUCCESS: Login erfolgreich
[30] Tue 30Aug16 11:06:27 - (000019) SSH_FXP_INIT: Client Version 3 (SSHJ_0_17_2)
[31] Tue 30Aug16 11:06:27 - (000019) SSH_FXP_VERSION: Server Version 3
[30] Tue 30Aug16 11:06:27 - (000019) {0x00000001} SSH_FXP_REALPATH: Pfad: "."
[31] Tue 30Aug16 11:06:27 - (000019) {0x00000001} SSH_FXP_NAME: Pfad: "/C:/Temp/sftp"
[30] Tue 30Aug16 11:06:27 - (000019) {0x00000002} SSH_FXP_OPENDIR: Pfad: "/C:/Temp/sftp/0013FP03/from_csdb"
[31] Tue 30Aug16 11:06:27 - (000019) {0x00000002} SSH_FXP_HANDLE: Verzeichnis Handle: 0x00000000
[32] Tue 30Aug16 11:06:27 - (000019) {0x00000003} SSH_FXP_READDIR: Handle: 0x00000000
[33] Tue 30Aug16 11:06:27 - (000019) {0x00000003} SSH_FXP_NAME: 6 Elemente gefunden
[32] Tue 30Aug16 11:06:27 - (000019) {0x00000004} SSH_FXP_READDIR: Handle: 0x00000000
[31] Tue 30Aug16 11:06:27 - (000019) {0x00000004} SSH_FX_EOF: End of file.
[30] Tue 30Aug16 11:06:27 - (000019) {0x00000005} SSH_FXP_CLOSE: Handle: 0x00000000
[31] Tue 30Aug16 11:06:27 - (000019) {0x00000005} SSH_FX_OK: Success.
[31] Tue 30Aug16 11:06:27 - (000019) SSH_MSG_DISCONNECT: client has requested a disconnect.  Reason code: 11
[02] Tue 30Aug16 11:06:27 - (000019) Sitzung geschlossen

[comtel2000]

This is the timeout exception for CoreFTP mini-server 30sec. after closing to sftpclient (should I open a new ticket?):

20:03:34.868 [main] DEBUG net.schmizz.sshj.connection.channel.direct.SessionChannel - Sending close
20:03:34.868 [main] TRACE net.schmizz.sshj.transport.Encoder - Encoding packet #13: 61 00 00 00 00
20:03:34.868 [main] DEBUG net.schmizz.concurrent.Promise - Awaiting <<chan#0 / close>>
20:04:04.885 [main] INFO net.schmizz.sshj.transport.TransportImpl - Disconnected - BY_APPLICATION
20:04:04.886 [main] DEBUG net.schmizz.sshj.connection.ConnectionImpl - Notified of net.schmizz.sshj.transport.TransportException: [BY_APPLICATION] Disconnected
20:04:04.888 [main] DEBUG net.schmizz.sshj.connection.channel.direct.SessionChannel - Channel #0 got notified of net.schmizz.sshj.transport.TransportException: [BY_APPLICATION] Disconnected
20:04:04.889 [main] DEBUG net.schmizz.sshj.connection.ConnectionImpl - Forgetting `session` channel (#0)
20:04:04.889 [main] DEBUG net.schmizz.concurrent.Promise - Setting <<chan#0 / close>> to `SOME`
20:04:04.889 [main] DEBUG net.schmizz.sshj.transport.TransportImpl - Sending SSH_MSG_DISCONNECT: reason=[BY_APPLICATION], msg=[]
20:04:04.889 [main] TRACE net.schmizz.sshj.transport.Encoder - Encoding packet #14: 01 00 00 00 0b 00 00 00 00 00 00 00 00
20:04:04.890 [main] DEBUG net.schmizz.concurrent.Promise - Setting <<transport close>> to `SOME`
20:04:04.890 [reader] DEBUG net.schmizz.sshj.transport.Reader - Stopping
20:04:04.893 [main] ERROR de.pharmacontrol.xml.arvato.ServUBugDemo - Timeout expired
net.schmizz.sshj.connection.ConnectionException: Timeout expired
    at net.schmizz.sshj.connection.ConnectionException$1.chain(ConnectionException.java:32)
    at net.schmizz.sshj.connection.ConnectionException$1.chain(ConnectionException.java:26)
    at net.schmizz.concurrent.Promise.retrieve(Promise.java:139)
    at net.schmizz.concurrent.Event.await(Event.java:103)
    at net.schmizz.sshj.connection.channel.AbstractChannel.close(AbstractChannel.java:245)
    at net.schmizz.sshj.sftp.SFTPEngine.close(SFTPEngine.java:247)
    at net.schmizz.sshj.sftp.SFTPClient.close(SFTPClient.java:254)
Caused by: java.util.concurrent.TimeoutException: Timeout expired
    ... 6 common frames omitted

[comtel2000]

After testing a couple of relative paths like "./export", "~/export" there is only one working solution for Serv-U (on windows env.). I've to canonicalize(".") and send the absolute path like in StatefulSFTPClient. I guess this is the reason for the separate StatefulSFTPClient? I couldn't find any samples or description (javadoc) when to use stateful or default.

    try (SSHClient ssh = new SSHClient()) {
      ssh.addHostKeyVerifier(new PromiscuousVerifier());
      ssh.connect(host, port);
      ssh.authPassword(user, pwd);
      SFTPClient sftp = ssh.newSFTPClient();
      // Serv-U req. a absolute path ("/C:/temp" + "/" + "export")
      String cwd = sftp.canonicalize(".") + "/" + incomingRemotePath;
      logger.info("found {} remote files at {}", sftp.ls(cwd).size(), cwd);
    } catch (Exception e) {
      logger.error(e.getMessage(), e);
    }

[dajoropo]

Which version of SSHJ are you using?

So let me get this straight, there are 2 different problems here:

* Serv-U FTP does not work with `sshClient.newSFTPClient()`, but it works with `new StatefulSFTPClient(..)`?

Hi @hierynomus

I had same issue here. An application, that was working correctly since some months, stopped working after the customer switched FTP server to start using Serv-U. I'm using SSHJ version 0.27.0. The exception (same as previous one) was thrown when executing ls().

net.schmizz.sshj.sftp.SFTPException: No such file.

at net.schmizz.sshj.sftp.Response.error(Response.java:140) [sshj-0.27.0.jar:0.27.0]

at net.schmizz.sshj.sftp.Response.ensurePacketTypeIs(Response.java:117) [sshj-0.27.0.jar:0.27.0]

at net.schmizz.sshj.sftp.SFTPEngine.openDir(SFTPEngine.java:161) [sshj-0.27.0.jar:0.27.0]

at net.schmizz.sshj.sftp.SFTPClient.ls(SFTPClient.java:57) [sshj-0.27.0.jar:0.27.0]

at net.schmizz.sshj.sftp.SFTPClient.ls(SFTPClient.java:52) [sshj-0.27.0.jar:0.27.0]

Changing our application to create a Stateful FTP client instead of an Standard one solved the issue. I made a method similar to the one implemented here https://stackoverflow.com/questions/50229081/sshj-how-to-use-statefulsftpclient to mimic the isConnected isAuthenticated calls from the Standard FTP client.

I didn't have any issue with the paths, they were used in the same way in both servers, so I think there is no problem there. I think the Stateful for Serv-U is also not a bug if it is documented somewhere.

I made a pull request in case you want to include that method.
Cheers,
Daniel

[hierynomus]

Closing this as the added method in #603 should fix this.