Slow file move within the same bucket

[michaltarana]

Hi, I am using s3fs 2024.9.0 to handle files on an S3 bucket operated on a CEPH cluster. One of the operations I need to perform is renaming a file, i.e. move within the same bucket, using

S3FileSytem_instance.mv('bucket/old_name', 'bucket/new_name')

While it works fine for smaller files, the files over 5GB move very slowly and I get back a ReadTimeOutError exception from botocore.

The same operation on the same files, using aws-cli mv takes only several seconds. However, s3fs is not done even after 15 minutes when the read_timeout exception is raised. Of course, I could try to increase the read_timeout config_kwarg. However, I am wondering, why is the s3fs so much slower with this operation on a large file than aws cli?

I would be very thankful for any insight.