Performance on large collections

[amw]

I'm trying to backup large file collection. About two million files. Why does mt-aws-glacier does not upload files as it finds them on disk and instead insists on creating a list of every file in the target directory first?

I see it reports it found new files every 1000 of them, but I don't see any uploads starting:

PID 19644 Started worker
PID 19645 Started worker
Found 1000 local files
Found 2000 local files
Found 3000 local files
Found 4000 local files
Found 5000 local files
Found 6000 local files
Found 7000 local files
Found 8000 local files
Found 9000 local files
Found 10000 local files
Found 11000 local files
Found 12000 local files
Found 13000 local files
Found 14000 local files
Found 15000 local files
Found 16000 local files
Found 17000 local files
Found 18000 local files

I can't reasonably wait until it loads a list of the whole 2 million files into memory first. I could use filters to select a subset of files at first (hopefully it doesn't load files not covered by /root/reference), but it would be so much better if it just checked file, checked it's status in index, start uploading, proceed further when there's a free spot on the upload queue.

What do you think?

[vsespb]

Probably yes. That's not easy to implement currently. Either file finder should be moved to different process or file finder code converted to iterator and this iterator should not do any slow operations as main process controls child processes and should respond instant. Also some compications for --delete-removed etc. I think I prefer move it to different process. But again, there is nothing like stream of files implemented in internal communication protocol.

I can mark it as enhancement, can't tell when I implement it. But looks useful.

I could use filters to select a subset of files at firs

see also this part of doc:

5) In some cases, to reduce disk IO, directory traversal into excluded directory can be stopped. This only can happen when mtglacier absolutely sure that it won't break behavior (4) described above. Currently it's guaranteed that traversal stop only in case when:

A directory match EXCLUDE rule without '!' prefix, ending with '/' or '**', or empty rule

AND there are no INCLUDE rules before this EXCLUDE RULE

[vsespb]

Also, try use --max-number-of-files - that's probably the solution for now.

[vsespb]

and limiting by file filtering is helping if you have a so huge journal which does not fit into memory (filtering save memory in this case).

[ipernet]

Hi

I +1 this enhancement.

Here I have 800k files, it takes 45 minutes to only list those files. Given my unstable internet connection, it's almost a no-go for using mt-aws-glacier in this case (restarting the process takes 45m again :/ ).

[vsespb]

Given my unstable internet connection, it's almost a no-go for using mt-aws-glacier in this case

In case of network errors mt-aws-glacier should retry request (at least 100 times), so you don't have to restart. It should work without need of restart in this case. If it crashes instead of retry - pls report a bug.

[ipernet]

Hi @vsespb

Agreed.

FYI, I manage to fail 100+ times some requests. Which cause others "tasks" to stop (see below logs). Looks like the counter is global and not per upload task? Don't get why some workers are always out and others works (will try to decrease concurrency from 4 to 1).

I'll check also why I get unexpected SIGINT signal on the process despite I run it using nohup <cmd> & (on a Synology, Busybox 1.16.1, ash shell). Will let you know any findings.

Thanks,

PID 17658 HTTP 408 This might be normal. Will retry (11 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (12 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (11 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (12 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (202 seconds spent for request)
PID 17659 Uploaded part for XXXXXXX.psd at offset [0]
PID 17658 HTTP 408 This might be normal. Will retry (11 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (11 seconds spent for request)
PID 17660 Uploaded part for XXXXXXX.psd at offset [67108864]
PID 17657 HTTP 408 This might be normal. Will retry (10 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (12 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (10 seconds spent for request)
PID 17659 Uploaded part for XXXXXXX.psd at offset [134217728]
PID 17658 HTTP 408 This might be normal. Will retry (13 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (10 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (398 seconds spent for request)
PID 17660 Uploaded part for XXXXXXX.psd at offset [201326592]
PID 17658 HTTP 408 This might be normal. Will retry (12 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (10 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (11 seconds spent for request)
PID 17659 Uploaded part for XXXXXXX.psd at offset [268435456]
PID 17658 HTTP 408 This might be normal. Will retry (14 seconds spent for request)
PID 17660 HTTP 408 This might be normal. Will retry (519 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (207 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (16 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (15 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (12 seconds spent for request)
PID 17659 Uploaded part for XXXXXXX.psd at offset [402653184]
PID 17657 HTTP 408 This might be normal. Will retry (12 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (13 seconds spent for request)
PID 17660 HTTP 408 This might be normal. Will retry (580 seconds spent for request)
ERROR (child 17658): Request was not successful after 100 retries

EXIT on SIGCHLD (exit_code 1)

[vsespb]

Looks like the counter is global and not per upload task?

no, it's per task!

Don't get why some workers are always out and others works

not sure. that's interesting. never saw this. some workers (PIDs) or some files?

I'll check also why I get unexpected SIGINT signal on the process despite I run it using nohup

nohup should prevent SIGHUP, not SIGINT

will try to decrease concurrency from 4 to 1

yes, you need decrease concurrency and part size. see docs: https://github.com/vsespb/mt-aws-glacier#limitations

I believe HTTP 408 is problem on Amazon side (I reproduced it using cURL), but they don't seem to care https://forums.aws.amazon.com/thread.jspa?messageID=511490&#511490

[ipernet]

Thanks for the support @vsespb. Your ticket about HTTP408 is interesting, I'll try some tweaking too.

I have attached the full log for your information, but I definitively need to decrease my concurrency and partsize first to ensure better success.

$ nohup mtglacier sync --config glacier.cfg --dir /volume1/Production/ --vault XXXXXXX --journal journal.log --exclude '@eadir/' --concurrency 4 --partsize 64 --timeout 60 &


MT-AWS-Glacier, Copyright 2012-2014 Victor Efimov http://mt-aws.com/ Version 1.120

PID 17657 Started worker
PID 17658 Started worker
PID 17659 Started worker
PID 17660 Started worker
Skip .DS_Store
Skip Thumbs.db
Skip Local Disk (D) - Shortcut.lnk
Skip skp_long.mp4
Skip .apdisk
Skip RecentPlaces.lnk
Skip Roger&Gallet/RetG-LOGO-21-02-13.pdf
Skip Roger&Gallet/Thumbs.db
Skip Roger&Gallet/2014/Thumbs.db
Skip Roger&Gallet/2014/Fleur de Figuier/Wireframes/R&G_Fleur_de_Figuier_2.rp
Skip Roger&Gallet/2014/Fleur de Figuier/Wireframes/R&G_Fleur_de_Figuier_3.rp
Skip Roger&Gallet/2014/Fleur de Figuier/Wireframes/R&G_Fleur_de_Figuier_Anglais.rp
Skip Roger&Gallet/2014/Fleur de Figuier/Wireframes/R&G_Fleur_de_Figuier_1.rp
Skip Files/2305.png
Skip Files/130014-03-ROGER_et_GALLET_Fleur_Figuier.tif
Skip Files/A84692_210X297_071_Figuier_INDICATION COULEUR.PDF
Skip Files/A84692_210X297_071_Figuier_V2B.pdf
Skip Files/Bodoni CE.zip
Skip Files/crème douche Fleur de Figuier 200 mL.tif
Skip Files/DPFR 2014_R&G_Huile Gourmande Fleur de Figuier-2.jpg
Skip Files/DPFR 2014_R&G_Huile Gourmande Fleur de Figuier-6.jpg
Skip Files/Eau fraîche Fleur de figuier_2014.tif
Skip Files/ecce-logo.zip
Skip Files/EPIV_Fleur_de_figuier_02_2.mp4
Skip Files/FDF - 50ml offert.png
Skip Files/Flacon 100ml Fleur de Figuier.tif
Skip Files/HD Huile Gourmande Fleur de Figuier.tif
Skip Files/HFJ_Gotham_1.zip
Skip Files/Huile Gourmande Fleur de Figuier.jpg
Found 1000 local files
[...]
Found 827000 local files
PID 17659 Created an upload_id cQKVU6J1wiWMmE9ajzhcUYdz7DaQ2rbwOZNvjall9waI9epXK2g4nLXpwff9zZMAD9E3ctXZ-GWAAamGcu5Lp4K9RT-O
PID 17657 Created an upload_id yatxVBw-x8_3T3S9x9z3ZW7XOIRTFUUdZO65UlyQYUP1uqScBI_w9HxIx3X0Ul8EiNG1rSLFKyEqhhOSK3tGd-bDTOyY
PID 17658 Created an upload_id jPLNCtu_Xo4d5eK3TD0oODbXs2t-5RpN9ARSlwYzqDJPPUCVwJCVJqWKKgONEohadXG5LRhWIS-V3FCalXTMSaLL-Que
PID 17660 Created an upload_id Xjl3byHt5NEicPyq9kC5V-idJPD6NYkKXUsG-EHPFZJv9EGZ0skfpwJ616pGx8q-qKxI221QO-fJOpBYGlXeXhL08QxM
PID 17658 HTTP 408 This might be normal. Will retry (500 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (109 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (6 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (701 seconds spent for request)
PID 17660 HTTP 408 This might be normal. Will retry (684 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (62 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (10 seconds spent for request)
PID 17660 HTTP 408 This might be normal. Will retry (38 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (44 seconds spent for request)
PID 17660 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17660 HTTP 408 This might be normal. Will retry (11 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (34 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (62 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (51 seconds spent for request)
PID 17660 HTTP 408 This might be normal. Will retry (107 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (46 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (204 seconds spent for request)
PID 17659 Uploaded part for Files/Figuier 2014 SP.psd at offset [0]
PID 17659 Uploaded part for Files/Lait Fleur de Figuier.tif at offset [67108864]
PID 17658 HTTP 408 This might be normal. Will retry (888 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (797 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (41 seconds spent for request)
PID 17660 HTTP 408 This might be normal. Will retry (956 seconds spent for request)
PID 17659 HTTP 408 This might be normal. Will retry (812 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (29 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (8 seconds spent for request)
PID 17660 HTTP 408 This might be normal. Will retry (48 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (88 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (54 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (8 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (24 seconds spent for request)
PID 17660 HTTP 408 This might be normal. Will retry (98 seconds spent for request)
PID 17659 HTTP 408 This might be normal. Will retry (154 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (98 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (103 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (159 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (146 seconds spent for request)
PID 17660 HTTP 408 This might be normal. Will retry (296 seconds spent for request)
PID 17660 HTTP 408 This might be normal. Will retry (24 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (15 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (220 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (9 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (8 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (46 seconds spent for request)
PID 17659 HTTP 408 This might be normal. Will retry (571 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (158 seconds spent for request)
PID 17660 HTTP 408 This might be normal. Will retry (238 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (8 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (83 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (12 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (489 seconds spent for request)
PID 17660 HTTP 408 This might be normal. Will retry (530 seconds spent for request)
PID 17660 HTTP 408 This might be normal. Will retry (29 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (71 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (112 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (8 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17659 Uploaded part for Files/figuier 2014 extreme horizontal CMJN.psd at offset [67108864]
PID 17658 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (334 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (19 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (11 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (11 seconds spent for request)
PID 17660 HTTP 408 This might be normal. Will retry (557 seconds spent for request)
PID 17660 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (278 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (154 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (8 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (10 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (22 seconds spent for request)
PID 17659 Uploaded part for Files/figuier 2014 extreme horizontal CMJN.psd at offset [134217728]
PID 17658 HTTP 408 This might be normal. Will retry (9 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (18 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (23 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (23 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (21 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (11 seconds spent for request)
PID 17660 Uploaded part for Files/Lait Fleur de Figuier.tif at offset [0]
PID 17660 Finished Files/Lait Fleur de Figuier.tif hash [0ac01f09678f64bf8403a7ec1d2fda758ef5ec004c3fc815490a4b4d9fc29268] archive_id [UMSeCggPa_ilBCv4tm_PgkIhdLMx7-QkGP3yKTEZtp5zc3Y
RJ8BcuIqJC3-GOAOn1mYrOjvfVTc-k1_kt6UTry4oGyhzgmHyrAwOQVy00kl0NvPutAkzjxorNZee3_iSDOT3hgNUIA]
PID 17658 HTTP 408 This might be normal. Will retry (90 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (958 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (18 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (59 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (12 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (13 seconds spent for request)
PID 17659 Uploaded part for Files/figuier 2014 extreme horizontal CMJN.psd at offset [201326592]
PID 17658 HTTP 408 This might be normal. Will retry (13 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (12 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (10 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (10 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (11 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (13 seconds spent for request)
PID 17660 Uploaded part for Files/figuier 2014 extreme horizontal CMJN.psd at offset [268435456]
PID 17657 HTTP 408 This might be normal. Will retry (327 seconds spent for request)
PID 17659 Uploaded part for Files/figuier 2014 extreme horizontal CMJN.psd at offset [335544320]
PID 17657 HTTP 408 This might be normal. Will retry (132 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (13 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (9 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (8 seconds spent for request)
PID 17660 Uploaded part for Files/figuier 2014 extreme horizontal CMJN.psd at offset [402653184]
PID 17658 HTTP 408 This might be normal. Will retry (11 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (510 seconds spent for request)
PID 17659 Uploaded part for Files/figuier 2014 extreme horizontal CMJN.psd at offset [469762048]
PID 17658 HTTP 408 This might be normal. Will retry (15 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (8 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (14 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (519 seconds spent for request)
PID 17660 Uploaded part for Files/figuier 2014 extreme horizontal CMJN.psd at offset [536870912]
PID 17657 HTTP 408 This might be normal. Will retry (129 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (12 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17659 Uploaded part for Files/figuier 2014 extreme horizontal CMJN.psd at offset [603979776]
PID 17657 HTTP 408 This might be normal. Will retry (64 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (12 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (68 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (13 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (12 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (67 seconds spent for request)
PID 17660 Uploaded part for Files/figuier 2014 extreme horizontal CMJN.psd at offset [671088640]
PID 17659 Uploaded part for Files/figuier 2014 extreme horizontal CMJN.psd at offset [738197504]
PID 17658 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (137 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (8 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (13 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (58 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (12 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (130 seconds spent for request)
PID 17660 Uploaded part for Files/figuier 2014 extreme horizontal CMJN.psd at offset [805306368]
PID 17658 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17660 Uploaded part for Files/figuier 2014 extreme horizontal CMJN.psd at offset [939524096]
PID 17658 HTTP 408 This might be normal. Will retry (16 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (327 seconds spent for request)
PID 17659 Uploaded part for Files/figuier 2014 extreme horizontal CMJN.psd at offset [872415232]
PID 17658 HTTP 408 This might be normal. Will retry (14 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (131 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (12 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (134 seconds spent for request)
PID 17660 Uploaded part for Files/Figuier 2014 SP.psd at offset [134217728]
PID 17658 HTTP 408 This might be normal. Will retry (12 seconds spent for request)
PID 17659 Uploaded part for Files/Figuier 2014 SP.psd at offset [201326592]
PID 17658 HTTP 408 This might be normal. Will retry (12 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (9 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (513 seconds spent for request)
PID 17659 HTTP 408 This might be normal. Will retry (394 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (16 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (136 seconds spent for request)
PID 17660 Uploaded part for Files/Figuier 2014 SP.psd at offset [268435456]
PID 17657 HTTP 408 This might be normal. Will retry (135 seconds spent for request)
PID 17660 HTTP 408 This might be normal. Will retry (64 seconds spent for request)
PID 17659 HTTP 408 This might be normal. Will retry (203 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (23 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (14 seconds spent for request)
PID 17659 HTTP 408 This might be normal. Will retry (387 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (15 seconds spent for request)
PID 17659 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (465 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (16 seconds spent for request)
PID 17660 Uploaded part for Files/Figuier 2014 SP.psd at offset [402653184]
PID 17660 HTTP 408 This might be normal. Will retry (15 seconds spent for request)
PID 17659 HTTP 408 This might be normal. Will retry (384 seconds spent for request)
PID 17660 HTTP 408 This might be normal. Will retry (8 seconds spent for request)
PID 17659 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (140 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (21 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (6 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (74 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (18 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (10 seconds spent for request)
PID 17660 Uploaded part for Files/Figuier 2014 SP.psd at offset [469762048]
PID 17659 Uploaded part for Files/Figuier 2014 SP.psd at offset [335544320]
PID 17657 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (11 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (9 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (9 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (328 seconds spent for request)
PID 17660 Uploaded part for Files/Figuier 2014 SP.psd at offset [536870912]
PID 17659 Uploaded part for Files/Figuier 2014 SP.psd at offset [603979776]
PID 17660 Uploaded part for Files/Figuier 2014 SP.psd at offset [671088640]
PID 17658 HTTP 408 This might be normal. Will retry (11 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (12 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (11 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (12 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (202 seconds spent for request)
PID 17659 Uploaded part for Files/figuier 2014 horizontal CMJN.psd at offset [0]
PID 17658 HTTP 408 This might be normal. Will retry (11 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (11 seconds spent for request)
PID 17660 Uploaded part for Files/figuier 2014 horizontal CMJN.psd at offset [67108864]
PID 17657 HTTP 408 This might be normal. Will retry (10 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (12 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (10 seconds spent for request)
PID 17659 Uploaded part for Files/figuier 2014 horizontal CMJN.psd at offset [134217728]
PID 17658 HTTP 408 This might be normal. Will retry (13 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (10 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (398 seconds spent for request)
PID 17660 Uploaded part for Files/figuier 2014 horizontal CMJN.psd at offset [201326592]
PID 17658 HTTP 408 This might be normal. Will retry (12 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (10 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (11 seconds spent for request)
PID 17659 Uploaded part for Files/figuier 2014 horizontal CMJN.psd at offset [268435456]
PID 17658 HTTP 408 This might be normal. Will retry (14 seconds spent for request)
PID 17660 HTTP 408 This might be normal. Will retry (519 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (207 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (16 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (15 seconds spent for request)
PID 17657 HTTP 408 This might be normal. Will retry (7 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (12 seconds spent for request)
PID 17659 Uploaded part for Files/figuier 2014 horizontal CMJN.psd at offset [402653184]
PID 17657 HTTP 408 This might be normal. Will retry (12 seconds spent for request)
PID 17658 HTTP 408 This might be normal. Will retry (13 seconds spent for request)
PID 17660 HTTP 408 This might be normal. Will retry (580 seconds spent for request)
ERROR (child 17658): Request was not successful after 100 retries

EXIT on SIGCHLD (exit_code 1)

Cheers,