feat: Add basic support for Multipart Upload #2
Conversation
test/multipart_test.gleam
Outdated
| |> list.map(fn(pair) { | ||
| let #(body, part_number) = pair | ||
| let assert Ok(res) = | ||
| upload_part.request(bucket:, key:, upload_id:, part_number:, body:) |
There was a problem hiding this comment.
should I actually perform these uploads in parallel so that we have an example use case?
test/multipart_test.gleam
Outdated
| }) | ||
|
|
||
| // Complete the multipart upload | ||
| // NOTE: The Parts must be sorted in ascending order for this operation |
There was a problem hiding this comment.
It's too confusing. I'm going to remove this note, and perhaps make complete_multipart_upload.request() sort the parts argument to prevent the error case I'm alluding to in this note.
The reason for sorting is that when calling complete_multipart_upload.request(), the parts argument must be ordered by the part_number, e.g. [Part(.., part_number: 1), Part(.., part_number: 2), Part(.., part_number: 3)], otherwise the S3 server would respond to such request with an error.
There was a problem hiding this comment.
done in 71ba290, but maybe I should skip sorting and let the request fail when the user gives us incorrect parts?
| bucket: String, | ||
| key: String, | ||
| upload_id: String, | ||
| part_number: Int, |
There was a problem hiding this comment.
From UploadPart docs (https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html):
Part numbers can be any number from 1 to 10,000, inclusive. A part number uniquely identifies a part and also defines its position within the object being created. If you upload a new part using the same part number that was used with a previous part, the previously uploaded part is overwritten.
There was a problem hiding this comment.
Could you add this a doc comment please 🙏
| import gleam/list | ||
| import gleam/string | ||
| import gleeunit/should | ||
| import helpers |
There was a problem hiding this comment.
Please add tests for the various possible errors
test/multipart_test.gleam
Outdated
| let assert Ok(res) = | ||
| get_object.request(bucket, key) | ||
| |> get_object.build(helpers.creds) | ||
| |> httpc.send_bits | ||
| let assert Ok(get_object.Found(contents)) = get_object.response(res) |
There was a problem hiding this comment.
Move this to a function in helpers.gleam please
There was a problem hiding this comment.
I'm not sure what you mean. Should I make a helper that takes a bucket, key, creds, and a list of parts as BitArrays, and verifies the object?
There was a problem hiding this comment.
Yeah! helper.get_object or something which returns the contents
test/multipart_test.gleam
Outdated
| should.not_equal(res.upload_id, "") | ||
|
|
||
| // Upload some parts (they can be sent in parallel) | ||
| // NOTE: The minimum Part size for multipart upload is 5MiB (hardcoded in minio) |
There was a problem hiding this comment.
It looks like part 3 is smaller than that.
There was a problem hiding this comment.
That's not a problem. I've updated the note in b0d5139
Maybe it's better to let the user get the error from S3, an incorrect order might be caused by a bug somewhere in the user's code
|
Done, I also updated the readme |
This PR adds support for
CreateMultipartUpload,UploadPartandCompleteMultipartUploadS3 APIs. This lets the users of this library upload large files more quickly, by uploading multiple chunks in parallel.There are other Multipart Upload APIs that could be implemented in the future:
The full list of these APIs can be found on MinIO's docs: https://min.io/docs/minio/linux/reference/s3-api-compatibility.html#multipart-uploads