Error when assembling chunks, status code 504 when using S3

[dronesportugal]

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

Steps to reproduce

1. Upload a big file, in my case 9GB .zip file
2. Error will show "Error when assembling chunks, status code 504"
3. File is locked and unable to be deleted "Failed to load resource: the server responded with a status of 423 ()"

Expected behaviour

Tell us what should happen
Expected file to be upload without errors

Actual behaviour

Tell us what happens instead
Error 504 assembling chunks and file is locked, unable to move or delete

Server configuration

Operating system:
Ubuntu 18.04
Web server:

Database:
Type: pgsql

Version: PostgreSQL 10.10 (Ubuntu 10.10-0ubuntu0.18.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0, 64-bit

Size: 14.4 MB
PHP version:
PHP
Version: 7.2.24

Memory Limit: 512 MB

Max Execution Time: 3600

Upload max size: 1000 MB
Nextcloud version: (see Nextcloud admin page)

Updated from an older Nextcloud/ownCloud or fresh install:

Where did you install Nextcloud from:
Professionaly install
Daniel Hansson
T&M Hansson IT AB
https://www.hanssonit.se

and installed from Unraid app, 2 different sources, same issue
Signing status:

Signing status

Login as admin user into your Nextcloud and access 
http://example.com/index.php/settings/integrity/failed 
paste the results here.

List of activated apps:

App list

If you have access to your command line run e.g.:
sudo -u www-data php occ app:list
from within your Nextcloud installation folder

Nextcloud configuration:

Config report

If you have access to your command line run e.g.:
sudo -u www-data php occ config:list system
from within your Nextcloud installation folder

or 

Insert your config.php content here. 
Make sure to remove all sensitive content such as passwords. (e.g. database password, passwordsalt, secret, smtp password, …)

Are you using external storage, if yes which one: local/smb/sftp/...

Are you using encryption: yes/no

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...

LDAP configuration (delete this part if not used)

LDAP config

With access to your command line run e.g.:
sudo -u www-data php occ ldap:show-config
from within your Nextcloud installation folder

Without access to your command line download the data/owncloud.db to your local
computer or access your SQL server remotely and run the select query:
SELECT * FROM `oc_appconfig` WHERE `appid` = 'user_ldap';


Eventually replace sensitive data as the name/IP-address of your LDAP server or groups.

Client configuration

Browser:
Google Chrome latest version
Operating system:
Windows 10

Logs

Web server error log

Web server error log

Insert your webserver log here

Nextcloud log (data/nextcloud.log)

Nextcloud log

Insert your Nextcloud log here

Browser log

Browser log

Insert your browser log here, this could for example include:

a) The javascript console log
b) The network log
c) ...

[GuyPaddock]

@Corrupto what web server are you using? What type of PHP (Apache or FPM)?

[dronesportugal]

@Corrupto what web server are you using? What type of PHP (Apache or FPM)?

PHP FPM 7.2
Nextcloud 17.0.1 stable

Hope to answer everything you asked.

[GuyPaddock]

@Corrupto We were seeing this too... because of the size of the file involved, nginx is timing-out before PHP is done assembling the final file.

With the current version of Nextcloud, you would want to adjust your NGinx configuration to increase timeouts to give NC more time to assemble files after upload:

        fastcgi_connect_timeout 60;
        fastcgi_send_timeout 1800;
        fastcgi_read_timeout 1800;

This allows up to 30 mins of time to pass. It probably doesn't need to be quite that high, but it all depends on how fast your server disk I/O is.

It'd be great if a future version of Nextcloud had a daemon to handle this kind of task out-of-proc (like Seafile does), since larger files are becoming pretty normal.

[dronesportugal]

@Corrupto We were seeing this too... because of the size of the file involved, nginx is timing-out before PHP is done assembling the final file.

With the current version of Nextcloud, you would want to adjust your NGinx configuration to increase timeouts to give NC more time to assemble files after upload:

        fastcgi_connect_timeout 60;
        fastcgi_send_timeout 1800;
        fastcgi_read_timeout 1800;

This allows up to 30 mins of time to pass. It probably doesn't need to be quite that high, but it all depends on how fast your server disk I/O is.

It'd be great if a future version of Nextcloud had a daemon to handle this kind of task out-of-proc (like Seafile does), since larger files are becoming pretty normal.

Can you let me know the location of the Nginx configuration? its not on the nextcloud folder. Maybe on /etc/php/7.2/fpm? wish file?

[GuyPaddock]

@Corrupto It depends on what nginx containers you are using... In our case, it's located in /etc/nginx/nginx.conf

[zarmhast]

Issue still reproducible in Nextcloud 20.0

I cannot set nginx/PHP-fpm timeouts to high values, as it only shifts the problem, instead of solving it, and at some point I start getting network disconnects instead of 504.

My proposed solution would be to mark files as ready to be reassembled, and handle the lengthy reassembly process in a cron task (cron.php). It seems a simple enough change, yet resulting in a way more stable file server. Especially considering that it is not that easy to add a daemon to a PHP-FPM server.

[SMH17]

It returns this error with big files upload even when actually they are correctly uploaded (tested checksum, no integrity issues, after upload in processing task it is triggered timeout event while uploaded file chucks are in PHP temp folder with assembling still in progress).
This needs to be addressed, Nextcloud should keep connection alive till the chucks are fully assembled, preventing these timeout events and related error. Setting huge timeouts isn't a good practice.

Configurations where I have reproduced this issue

[1]
OS: Debian Buster (ARM 64bit)
PHP: 7.3
Apache: 2.4
Nextcloud: 20.0.4

File size uploaded with this error 12GB, 33GB
Upload mode: Web interface drag-and-drop (Chromium browser 87.0.4280.88)

[2]
OS: Debian Buster (x86-64)
PHP: 7.4
Nginx: 1.18
Nextcloud: 20.0.4

File size uploaded with this error 40GB
Upload mode: Web interface drag-and-drop (Chromium browser 87.0.4280.88)

[brtptrs]

After adjusting the timeouts we have the situation that big uploads end with "Error when assembling chunks, status code 504" errors only to have the files appear "magically" in nextcloud after some time. It seems that die the feedback on screen can also be misleading.

[kiplingw]

I confirm this with 21.0.1.1. I hope we won't have to upgrade to version 22 in order to have this fixed, since some of us on shared hosting don't have access to the newer MySQL required to run it.

[voyager-hr]

Did anyone manage to find solution for this?

In short I can upload, it seems, any size file, but when it says processing files "it fails with that 504 error" but file IS there and matches checksum of original file.

I thought it might be chunks size (default to 10MB) - I coudn't for some reason get it changed to different value so I changed code in PHP to 100 * 1024 * 1024 instead of default 10 * 1024 * 1024. But problem still persists.

Any other ideas? I tried now adding that timeout parameter but I somehow doubt that is the reason because I tested it is QUITE close to 2.5GB wehere it fails for me ... so 2.3GB I can upload no problem (either on LAN 1G speed or 10mbit adsl line) but 2.6 fails (that is why I thought maybe it can handle only 256 or close chunks).

I am running it inside docker container using SSD as storage and image linuxserver/nextcloud:amd64-version-21.0.1

[voyager-hr]

OK I think I have resolved this after hours of debugging - when I got desperate and posted comment above and resumed investigation of the issue.

I tried mounting upload and destination directories separatly as ramdisk (tmpfs), then both, but benefit was neglible.

So after realizing it is some kind of timeout and using my kitchen timer found it at 60 sec.

I found this to resolve my issue:

In nc container I edited /config/nginx/site-confs default site config for nginx and ADDED (just bold part: fastcgi_read_timeout 3600s;):

    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_param PATH_INFO $path_info;
    fastcgi_param HTTPS on;

    **# 20210523 Dorijan - default is 60s and assembling uploaded chunks takes longer than 60 seconds
    fastcgi_read_timeout 3600s;**

    fastcgi_param modHeadersAvailable true;         # Avoid sending the security headers twice
    fastcgi_param front_controller_active true;     # Enable pretty urls
    fastcgi_pass php-handler;

[SMH17]

@voyager-hr This is a bug that must be fixed Nextcloud side, keeping connection alive till the completion, rather than increasing timeout. As I have said above, using huge timeout in order to workaround the issue isn't a good practice.

[szaimen]

If you are using Nginx, did you make sure to configure it correctly?
https://docs.nextcloud.com/server/stable/admin_manual/installation/nginx.html?highlight=nginx

[SMH17]

The server is configured correctly and the bug is caused by the fact Nextcloud doesn't keep connection alive till the completion, so when a very large file is uploaded during the time needed to assembly all chunks it goes in timeout.

[szaimen]

Hm... I am not sure if this is fixable by us since it looks like it runs into a timeout issue from the webserver side if it takes too long.

cc @nextcloud/server-triage for more input on this.

[zarmhast]

Hi, please update to 24.0.8 or better 25.0.2 and report back if it fixes the issue. Thank you!

Since about 23, don't remember the version, it is no longer possible to upload large files because float is not integer.

Because of that inability to handle large files any longer and because it is marked as "won't fix" I have long uninstalled nextcloud. IMHO, due to its inability to handle large files at all (runtime type exception when handling file size), the chunking issue is no longer relevant.

[szaimen]

Handling large files is not a problem at all with e.g. https://github.com/nextcloud/all-in-one. So it is really only a configuration issue when it doesn't work in my experience.

[BlueH2O]

Hi, please update to 24.0.8 or better 25.0.2 and report back if it fixes the issue. Thank you!

I am on 25.0.2 and the issue persists.

[szaimen]

Die you follow all the steps of https://docs.nextcloud.com/server/stable/admin_manual/configuration_files/big_file_upload_configuration.html to enable large file uploads?

[shibco]

Hi, please update to 24.0.8 or better 25.0.2 and report back if it fixes the issue. Thank you!

I am on 25.0.3 and have this issue also.

Die you follow all the steps of https://docs.nextcloud.com/server/stable/admin_manual/configuration_files/big_file_upload_configuration.html to enable large file uploads?

Followed to the letter, and still encountered this issue on any file above 1GB. This suggestion worked: #17992 (comment)

[despens]

@szaimen Following the provided documentation, which isn't the most clear on all the settings described, is only able to mitigate the issue in certain cases. As many users report, there are issues with timeouts when uploading large files. Timeouts are particularly hard to reproduce, especially with networked storage, or with slow connections in between uploading user and receiving nextcloud server. The solution is to not make operations that potentially take a long time dependent on a single HTTP request being upheld for the length of that operation. In the case of large files, the timeouts can also be caused by network settings of the uploading user. You cannot tell everyone to switch internet providers or get another job where the company network is configured differently 😉 Instead, nextcloud needs to work on being more resistant against timeouts that are always possible. They just surface most for users working with large files.

Now even if increasing timeout settings and the max post request size across a dozen config files on nextcloud infrastructure would solve all uploading issues, which, as explained above, it doesn't, having such long timeouts is bad practice for performance and security. It makes it possible for a malicious client to just keep loads of HTTP requests active without sending much data at all, quickly reaching the limit of the threads the webserver or reverse proxy can handle. This then prevents regular users from using the service at all.

So, please register this issue as what it is and give it the priority it deserves. If this behavior was fixed I could stop renting temporary VMs located in close proximity to my nextcloud deployment just to upload large files to it.

[dronesportugal]

I was forced to switch to community version of ownCloud and it works perfectly without any changes to the default code. Uploaded and download 100+ GB zip from webpage and a lot of sync files 4+ GB and never fails.
Since nextcloud was forked from owncloud, digging into their code would probably fix this.

[szaimen]

This suggestion worked: #17992 (comment)

This only prooves that I am right that you need to adjust the values that are mentioned in https://docs.nextcloud.com/server/stable/admin_manual/configuration_files/big_file_upload_configuration.html. If you dont want to adjust the values on your instance manually, feel free to switch to https://github.com/nextcloud/all-in-one where things should work out of the box.

[despens]

@szaimen I think there is a general misunderstanding. While changing these values helps some users, it doesn't help others. There would be no reason for users to change values if all nextcloud clients would use chunked uploads and the re-assembly of the chunks into one file wouldn't be dependent on a single HTTP request not timing out.

Even the AIO instructions mention

Cloudflare only supports uploading files up to 100 MB in the free plan, if you try to upload bigger files you will get an error (413 - Payload Too Large).

As long as the maximum file size that can be put on nextcloud is restricted by the maximum size of a POST or PUT request, this problem will persist.

[szaimen]

Actually all Nextcloud clients should indeed be able to use chunked uploads. This is more a general hint for e.g. public uploads. I'll adjust the doc of AIO accordingly. See nextcloud/all-in-one#1843

Also this is actually a problem of your infrastructure if clouflare does not supports uploads bigger than 100MB or rather a feature request to use chunked uploads also in the public upload case.

[despens]

Thanks for changing the AIO documentation @szaimen. It is true that even nextcloud's basic browser uploader in standard configuration splits large files into 10MB chunks, except for the public file drop situation.

This is great because the data transfer part of an upload is very unlikely to time out then. A client could even be smart enough that if a single chunk transfer times out it would be retried with a smaller chunk size, which I believe the desktop client already does.

Now once the chunks are transferred to nextcloud, they are re-assembled into a single file. Depending on the storage used as well as the number and size of the chunks, this process can take quite some time. The re-assembling must be finished within the timeout set for a single HTTP request. Probably since nextcloud doesn't send any data to a client during a lengthy re-assembly process, proxies and/or clients are likely to discard the connection, typically to protect from (unintentional) DOS. Discarding the connection will fail the upload. The time for re-assembly can increase massively if nextcloud is configured to use networked storage (like iSCSI, NFS, SFTP, S3) and moves chunks and final file in between remote storage and local temp storage several times.

As a user, currently I have two choices to transfer a large file:

1. Use a client that supports chunks (nextcloud browser interface or desktop client), but that will time out on the re-assembly part, after which even a successful upload is discarded.
2. Use a client that doesn't use chunked uploads (such as rclone or a WebDAV client), which will time out on the actual data transfer part.

Changing timeouts and chunk sizes in configuration can mitigate the problem under certain conditions, while degrading security and performance in some cases, but it is not a reliable solution. Ideally clients, and in particular the desktop client, would be able to wait for the re-assembling of chunks and register a successful upload.

Currently this doesn't seem to be implemented, or is not working. This is a problem for users that have to deal with large files. It will hardly affect users that deal with typical office documents, but those working with audio, video, large documents for print, disk images, etc, will have a very hard time.

I'm one of the folks who has their nextcloud configuration cranked up to 11 regarding timeouts, request body sizes, etc, and still am unable to get my 8 GB digital video file uploaded, while I have no issues to, for instance, upload it to my peertube instance which is self-hosted just the same.

When exchanging large files with collaborators I would love to avoid having to use WeTransfer or DropBox or other proprietary and centralized tools, but sometimes I have to rely on them, because these services figured out how to deal with large files reliably. It would also be amazing for people currently using CreativeCloud to move to sovereign storage with nextcloud. But as things stand I would only be able to recommend nextcloud for collaborating on light office files.

[shibco]

This jumped out at me, from @despens's comment above:

I'm one of the folks who has their nextcloud configuration cranked up to 11 regarding timeouts, request body sizes, etc, and still am unable to get my 8 GB digital video file uploaded, while I have no issues to, for instance, upload it to my peertube instance which is self-hostes just the same.

We find ourselves in exactly the same situation, and this makes remarks like #17992 (comment) so frustrating. Our organisation uses a large S3 configuration provided as a managed Openstack instance, and we run individual compartmentalised VMs, all running the same VM image. We are able to effortlessly send 10-12GB files to Peertube out of the box, but experience easily reproduce-able and consistent failures on Nextcloud. Peertube is built by a team of one paid individual, and completely staffed by voluntary contributors beyond that financially supported developer.

What's worse, is that the upload method via Desktop client described by @despens has additional ironic bugs that are intrinsically linked to this issue: Error messages are truncated and unreadable on chunk failure, which confuses users. In certain circumstances, the UI will even reset after a period of time and display a large "✅ ALL FILES SYNCED" interface to the user, despite a very clear and serious discrepancy between the client and server directory structure.

[despens]

To illustrate the agonizing everyday impact of this bug:

• I'm syncing my local projects directory with nextcloud using the nextcloud desktop client.
• Upon startup the client starts scanning the projects directory and uploads any changes. The upload process also starts for 8 files that always fail to upload because they are too large for how nextcloud handles uploads. They upload completely in chunks, then the assembling of chunks fails.
• The large files will be attempted to upload again every single sync cycle. So if I change a 100 bytes text file, it will only be synced after the attempt to upload all the large files has failed again.
• Depending on local network speed this can take hours. The process is wasting incredible amounts of bandwidth and CPU cycles on the server, and generates an avalanche of truncated error messages in the client that, when clicked on, don't show any useful information; for instance the error shows internal chunk names instead of the user-facing name of the file that generated the problem.

In short, this bug is not only affecting large files, but also

• every other file that is supposed to be synced
• every other user on the local network that wants to do legitimate stuff when nextcloud is taking most of the bandwidth
• every other user trying to interact with the nextcloud server, because it is constantly busy with attempted re-uploads.

💀

[shibco]

I can reproduce this too. Further to this, when a user is experiencing this issue, it will hang the web interface for all users (for example, the nextcloud server takes 30+ seconds to even respond to an http request), and furthermore this bug destroys file versioning over S3.

[Sokolaizer]

So guys, can there be a possibility to rise the priority of the task? Because it seems to me that this is quite a sensitive issue :)

[zarmhast]

This is no longer an issue for me since nextcloud dropped support for large files altogether. Now trying to upload anything larger than 2GB triggers a 500 because float is not an integer. Anyhow was long overdue moving to a different file sharing system that actually works.

[pierreozoux]

Now you can upload to s3 with chuncks directly, so this bug should be closed I think. If anybody can confirm?
(this is on our backlog to try it out)

[solracsf]

This shouldn't be an issue anymore with:

• S3 backend that supports MultiPart uploads
• NC 27+ : Use MultipartUpload for uploading chunks to s3 #27034

This allows chunked upload streaming the chunks directly to S3 so that the final MOVE request no longer needs to assemble the final file on the Nextcloud server.