Skip to content
This repository has been archived by the owner on Aug 2, 2023. It is now read-only.

Added the ability to upload files larger than 4MB #2180

Merged
merged 6 commits into from
Mar 23, 2018
+51 −19
Merged

Added the ability to upload files larger than 4MB #2180

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
c8d31c4
Bug fix in BufferWriter
KrzysztofCwalina Mar 20, 2018
8ee6979
Merge branch 'master' of https://github.com/dotnet/corefxlab
KrzysztofCwalina Mar 22, 2018
33da66f
Merge branch 'master' of https://github.com/dotnet/corefxlab
KrzysztofCwalina Mar 22, 2018
ee23d6b
Merge branch 'master' of https://github.com/dotnet/corefxlab
KrzysztofCwalina Mar 22, 2018
40bec88
AzCopyCore: Added the ability to upload files larger than 4MB
KrzysztofCwalina Mar 22, 2018
474d53f
PR review feedback
KrzysztofCwalina Mar 23, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion samples/AzCopyCore/AzCopyCore/Helpers/PipelinesExtensions.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ public static async Task WriteAsync(this PipeWriter writer, Stream stream, long
while (bytesToWrite > 0)
{
Memory<byte> buffer = writer.GetMemory();
if(buffer.Length > bytesToWrite)
if (buffer.Length > bytesToWrite)
{
buffer = buffer.Slice(0, (int)bytesToWrite);
}
Expand Down
4 changes: 2 additions & 2 deletions samples/AzCopyCore/AzCopyCore/Program.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,7 @@ static async ValueTask<bool> CopyLocalFileToStorageFile(StorageClient client, st
long bytesLeft = bytes.Length;
long index = 0;
int length = 1024 * 1024 * 4;
while (true)
while (bytesLeft > 0)
{
if (bytesLeft < length) length = (int)bytesLeft;
var putRequest = new PutRangeRequest(storagePath, bytes, index, length);
Expand All @@ -212,7 +212,7 @@ static async ValueTask<bool> CopyLocalFileToStorageFile(StorageClient client, st
}
index += length;
bytesLeft -= length;
if (bytesLeft == 0) break;
Debug.Assert(bytesLeft >= 0);
Copy link
Member

@ahsonkhan ahsonkhan Mar 23, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this Debug.Assert is necessary, given the loop condition is already stronger. Maybe a better option would be to add a Debug.Assert after the loop exits, if we know that bytesLeft should never be negative and if that signals a coding bug:
Debug.Assert(bytesLeft == 0);

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The assert ensures bytesLeft is no negative

}
}
}
Expand Down
8 changes: 4 additions & 4 deletions samples/AzCopyCore/AzCopyCore/StorageRequests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ public struct PutRangeRequest : IStorageRequest

public PutRangeRequest(string filePath, Stream fileContent, long offset, int length)
{
if (offset < 0) throw new ArgumentOutOfRangeException(nameof(offset));
if (offset < 0 || offset > fileContent.Length - length) throw new ArgumentOutOfRangeException(nameof(offset));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm. What about the case when length > fileContent.Length?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AORE will be thrown. fileContent.Length - length will be negative and will be < offset (which has to be positive at the time of this check)

if (length < 1) throw new ArgumentOutOfRangeException(nameof(length));

_filePath = filePath;
Expand All @@ -139,7 +139,7 @@ class Writer : StorageRequestWriter<PutRangeRequest>

protected override async Task WriteBody(PipeWriter writer, PutRangeRequest arguments)
{
var stream = arguments._fileContent;
Stream stream = arguments._fileContent;
stream.Seek(arguments._offset, SeekOrigin.Begin);
await writer.WriteAsync(stream, arguments._length);
}
Expand All @@ -150,8 +150,8 @@ protected override void WriteXmsHeaders(ref BufferWriter writer, ref PutRangeReq
writer.WriteHeader("x-ms-date", Time, 'R');
// TODO (pri 3): this allocation should be eliminated

var start = arguments._offset;
var end = start + arguments._length - 1;
long start = arguments._offset;
long end = start + arguments._length - 1;
writer.WriteHeader("x-ms-range", $"bytes={start}-{end}");
writer.WriteHeader("x-ms-version", "2017-04-17");
writer.WriteHeader("x-ms-write", "update");
Expand Down