SF#147 resuming a multipart upload only skips up to 1000 existing files/parts

[mdomsch]

(From https://sourceforge.net/p/s3tools/bugs/147/)
Milestone: Enhancement_request
Status: open
Owner: nobody
Labels: None
Priority: 5
Updated: 3 hours ago
Created: 3 hours ago
Creator: anierman
Private: No

A multipart upload failed after several thousand 15MB files/parts had been uploaded.

After restarting the upload with put and --upload-id only the first 1000 files/parts were skipped due to md5/size matches. Re-uploading began at file/part 1001.

The issue is that list_multipart in S3.py only returns 1000 records at most, see:
http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadListParts.html

To skip all of the appropriate matches--past 1000--IsTruncated could be detected in the response and appropriate output retrieved in a loop (making use of part-number​-marker).

[mdomsch]

Same root problem as #414

[BobPusateri]

Ran across this issue today. Highly annoying. Will use larger upload chunks in the future so hopefully all my files have less than 1000 parts, but this should really be addressed.

[fviard]

@BobPusateri Indeed it is a real issue and it is the one that I'm currently working on, so it should be fixed soon.

[fviard]

@BobPusateri FYI, this issue is now fixed in MASTER and will be in the next release 2.2.0.

[jtbandes]

Thanks for fixing this! Would be great if it made it into a release at some point 😂 But this fix saved my ~7500-part upload after a power outage. Even cooler that it was able to resume from an upload I started with aws s3 cp. 💖