425 Unable to build data connection: TLS session of data connection not resumed

[ManuelReschke]

NOTE! YES ITS A NOT A RCLONE PROBLEM, they use this lib here and this is broken and reproducible.

Describe the bug
Using rclone to connect to an FTPS - FileZilla Server. Error
425 Unable to build data connection: TLS session of data connection not resumed

To Reproduce

Setup FileZilla Server with v1.7.2, mount this FTP with the given command. Try to transfer diffrent files. Use rclone with ls or copy or mount and try to send file.

Expected behavior
A working connection!

FTP server

• Filezilla FTP Server v1.7.2

Debug output
All infos here: https://forum.rclone.org/t/rclone-mount-425-unable-to-build-data-connection-tls-session-of-data-connection-not-resumed/40038

Additional context
rclone ls ftps-nas: -vv
2023/07/17 15:28:44 DEBUG : rclone: Version "v1.63.0" starting with parameters ["rclone" "ls" "ftps-nas:" "-vv"]
2023/07/17 15:28:44 DEBUG : Creating backend with remote "ftps-nas:"
2023/07/17 15:28:44 DEBUG : Using config file from "/home/dev/.config/rclone/rclone.conf"
2023/07/17 15:28:44 DEBUG : ftp://-MYDOMAIN-:21: Connecting to FTP server
2023/07/17 15:28:44 DEBUG : ftp://:21: dial("tcp",":21")
2023/07/17 15:28:44 DEBUG : ftp://:21: > dial: conn=*fshttp.timeoutConn, err=
2023/07/17 15:28:44 DEBUG : ftp://:21: dial("tcp","95.90.129.161:65349")
2023/07/17 15:28:45 DEBUG : ftp://:21: > dial: conn=*tls.Conn, err=
2023/07/17 15:28:45 DEBUG : ftp://:21: Connecting to FTP server
2023/07/17 15:28:45 DEBUG : ftp://:21: Connecting to FTP server
2023/07/17 15:28:45 DEBUG : ftp://:21: Connecting to FTP server
2023/07/17 15:28:45 DEBUG : ftp://:21: dial("tcp",":21")
2023/07/17 15:28:45 DEBUG : ftp://:21: Connecting to FTP server
2023/07/17 15:28:45 DEBUG : ftp://:21: Connecting to FTP server
2023/07/17 15:28:45 DEBUG : ftp://:21: Connecting to FTP server
2023/07/17 15:28:45 DEBUG : ftp://:21: Connecting to FTP server
2023/07/17 15:28:45 DEBUG : ftp://:21: dial("tcp",":21")
2023/07/17 15:28:45 DEBUG : ftp://:21: dial("tcp",":21")
2023/07/17 15:28:45 DEBUG : ftp://:21: > dial: conn=*fshttp.timeoutConn, err=
2023/07/17 15:28:45 DEBUG : ftp://:21: dial("tcp","95.90.129.161:65385")

[bryancua]

This is not an issue with this package. You need to setup TLS resumption on your &tls.Config:

• ServerName must be correct
• SessionTicketsDisabled must be false (this is default)
• ClientSessionCache must be non-nil (e.g., tls.NewLRUClientSessionCache(0))

[ManuelReschke]

Ok thanks for the answer.

[ncw]

@bryancua wrote:

This is not an issue with this package. You need to setup TLS resumption on your &tls.Config:

1. ServerName must be correct
2. SessionTicketsDisabled must be false (this is default)
3. ClientSessionCache must be non-nil (e.g., tls.NewLRUClientSessionCache(0))

Hi rclone maintainer here!

Here is what rclone's tls.Config looks like

	if opt.TLS || opt.ExplicitTLS {
		tlsConfig = &tls.Config{
			ServerName:         opt.Host,
			InsecureSkipVerify: opt.SkipVerifyTLSCert,
		}
		if opt.TLSCacheSize > 0 {
			tlsConfig.ClientSessionCache = tls.NewLRUClientSessionCache(opt.TLSCacheSize)
		}
		if opt.DisableTLS13 {
			tlsConfig.MaxVersion = tls.VersionTLS12
		}
	}

opt.TLSCacheSize defaults to non zero

Here is the actual tls.Config as printed from the users configuration from ( rclone/rclone#7234 ) which appears to satisfy your 3 criteria.

&tls.Config{
	Rand:io.Reader(nil),
	Time:(func() time.Time)(nil),
	Certificates:[]tls.Certificate(nil),
	NameToCertificate:map[string]*tls.Certificate(nil),
	GetCertificate:(func(*tls.ClientHelloInfo) (*tls.Certificate, error))(nil),
	GetClientCertificate:(func(*tls.CertificateRequestInfo) (*tls.Certificate, error))(nil),
	GetConfigForClient:(func(*tls.ClientHelloInfo) (*tls.Config, error))(nil),
	VerifyPeerCertificate:(func([][]uint8, [][]*x509.Certificate) error)(nil),
	VerifyConnection:(func(tls.ConnectionState) error)(nil),
	RootCAs:(*x509.CertPool)(nil),
	NextProtos:[]string(nil),
	ServerName:"io19.blackpig.net",
	ClientAuth:0,
	ClientCAs:(*x509.CertPool)(nil),
	InsecureSkipVerify:false,
	CipherSuites:[]uint16(nil),
	PreferServerCipherSuites:false,
	SessionTicketsDisabled:false,
	SessionTicketKey:[32]uint8{},
	ClientSessionCache:(*tls.lruSessionCache)(0xc00097bd20),
	MinVersion:0x0,
	MaxVersion:0x0,
	CurvePreferences:[]tls.CurveID(nil),
	DynamicRecordSizingDisabled:false,
	Renegotiation:0,
	KeyLogWriter:io.Writer(nil),
	mutex:sync.RWMutex{},
	sessionTicketKeys:[]tls.ticketKey(nil),
	autoSessionTicketKeys:[]tls.ticketKey(nil),
}

Any ideas as to what is missing?

Thank you

[ncw]

Here is my understanding of what TLS session of data connection not resumed actually means:

As an extra security feature some FTP servers (eg FileZilla) require that the data connection re-use the same TLS connection as the control connection. This is a good thing for security. However that message means that the server detected that the TLS connection wasn't the same (ie resumed).

This only happens when rclone is using more than one connection for transfers, so I conjecture that either rclone or this library is mixing up the TLS connections when there are more than one. I can't see an obvious way for rclone to do this since all rclone does is call Stor or RetrFrom so I think the problem must be deeper.

All this library does is call net.Dialer with the tls.Config as passed in. I think the problem must be here - the net.Dialer is resuming the wrong TLS connection (since there will be several in the cache) and the FTP server is complaining about that.

The obvious solution to this is to pass in a different TLS session cache for each connection rather than re-using it for each connection like you would with HTTP. I'll try this approach in rclone (see rclone/rclone#7234 (comment) )

If this works then a note in the docs for the TLS dialling functions would be a good idea, or we could get then to rebuild the TLS session cache each time it is used.

[ncw]

The obvious solution to this is to pass in a different TLS session cache for each connection rather than re-using it for each connection like you would with HTTP.

This approach does work according to my tester and fixes the problem.

Shall I prepare a PR to update the docs @ManuelReschke ?

I'll also submit another PR which enables this to be fixed when using an external dialer which is what rclone needs.

[bryancua]

Here is my understanding of what TLS session of data connection not resumed actually means:

As an extra security feature some FTP servers (eg FileZilla) require that the data connection re-use the same TLS connection as the control connection. This is a good thing for security. However that message means that the server detected that the TLS connection wasn't the same (ie resumed).

This only happens when rclone is using more than one connection for transfers, so I conjecture that either rclone or this library is mixing up the TLS connections when there are more than one. I can't see an obvious way for rclone to do this since all rclone does is call Stor or RetrFrom so I think the problem must be deeper.

All this library does is call net.Dialer with the tls.Config as passed in. I think the problem must be here - the net.Dialer is resuming the wrong TLS connection (since there will be several in the cache) and the FTP server is complaining about that.

The obvious solution to this is to pass in a different TLS session cache for each connection rather than re-using it for each connection like you would with HTTP. I'll try this approach in rclone (see rclone/rclone#7234 (comment) )

If this works then a note in the docs for the TLS dialling functions would be a good idea, or we could get then to rebuild the TLS session cache each time it is used.

I somehow didn't realize that TLS config objects can be shared between different connections, and have always just assumed that each connection should use its own config object (and therefore, cache object) by default. I wonder if it has to do with the way the cache "caches" connections.

[ncw]

I somehow didn't realize that TLS config objects can be shared between different connections, and have always just assumed that each connection should use its own config object (and therefore, cache object) by default. I wonder if it has to do with the way the cache "caches" connections.

Each connection does have its own config. It is just if you pass the same tls.Config in to ftp.DialWithTLS or ftp.DialWithExplicitTLS each time you make a new connection then they will share the TLS cache.

This is obvious in retrospect so I was thinking of a little addition to the docs like this:

diff --git a/ftp.go b/ftp.go
index 008b0c5..90e6a23 100644
--- a/ftp.go
+++ b/ftp.go
@@ -278,6 +278,13 @@ func DialWithContext(ctx context.Context) DialOption {
 // If called together with the DialWithDialFunc option, the DialWithDialFunc function
 // will be used when dialing new connections but regardless of the function,
 // the connection will be treated as a TLS connection.
+//
+// Note that if you set `tlsConfig.ClientSessionCache` then you
+// **must** provide a new cache each time you use this option. If you
+// don't then servers implementing TLS 1.3 session resumption will
+// refuse some of the connections with an error like this:
+//
+//	425 Unable to build data connection: TLS session of data connection not resumed.
 func DialWithTLS(tlsConfig *tls.Config) DialOption {
 	return DialOption{func(do *dialOptions) {
 		do.tlsConfig = tlsConfig
@@ -286,6 +293,13 @@ func DialWithTLS(tlsConfig *tls.Config) DialOption {
 
 // DialWithExplicitTLS returns a DialOption that configures the ServerConn to be upgraded to TLS
 // See DialWithTLS for general TLS documentation
+//
+// Note that if you set `tlsConfig.ClientSessionCache` then you
+// **must** provide a new cache each time you use this option. If you
+// don't then servers implementing TLS 1.3 session resumption will
+// refuse some of the connections with an error like this:
+//
+//	425 Unable to build data connection: TLS session of data connection not resumed.
 func DialWithExplicitTLS(tlsConfig *tls.Config) DialOption {
 	return DialOption{func(do *dialOptions) {
 		do.explicitTLS = true

[LeonB]

I'm also having the issue with the 425 error but with a very simple client:

package main

import (
	"crypto/tls"
	"log"
	"os"

	"github.com/jlaffaye/ftp"
)

func main() {
	var opts []ftp.DialOption
	config := &tls.Config{
		InsecureSkipVerify: true,
		ClientSessionCache: tls.NewLRUClientSessionCache(1),
	}
	opts = append(opts, ftp.DialWithExplicitTLS(config))
	opts = append(opts, ftp.DialWithDebugOutput(os.Stdout))

	conn, err := ftp.Dial(os.Getenv("FTP_ADDR"), opts...)
	if err != nil {
		log.Fatal(err)
	}

	err = conn.Login(os.Getenv("FTP_USERNAME"), os.Getenv("FTP_PASSWORD"))
	if err != nil {
		log.Fatal(err)
	}

	ee, err := conn.List("")
	if err != nil {
		log.Fatal(err)
	}

	log.Fatal(ee)
}

220 Please visit https://filezilla-project.org/
AUTH TLS
234 Using authentication type TLS.
USER *****
331 Please, specify the password.
PASS *****
230 Login successful.
FEAT
211-Features:
 MDTM
 REST STREAM
 SIZE
 MLST type*;size*;modify*;perm*;
 MLSD
 AUTH SSL
 AUTH TLS
 PROT
 PBSZ
 UTF8
 TVFS
 EPSV
 EPRT
 MFMT
211 End
TYPE I
200 Type set to I
OPTS UTF8 ON
202 UTF8 mode is always enabled. No need to send this command
PBSZ 0
200 PBSZ=0
PROT P
200 Protection level set to P
EPSV
229 Entering Extended Passive Mode (|||49350|)
MLSD
150 About to start data transfer.
425 Unable to build data connection: TLS session of data connection not resumed.
2024/01/12 12:55:12 1 error occurred:
	* 425 Unable to build data connection: TLS session of data connection not resumed.

go.mod

	github.com/jlaffaye/ftp v0.2.1-0.20240111193146-a93533d16c6c

Is there still an issue with the tls connection in the library?

[ncw]

@LeonB as far as I know the library is working properly now.

However you've told the cache to only cache one connection - that might be the problem as when you start a transfer you are starting a second connection. I use 32 here in rclone.

  `ClientSessionCache: tls.NewLRUClientSessionCache(1),`