Skip to content

Automated Transfer Validation API #29778

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jaschrep-msft merged 47 commits into from
Jul 15, 2022
Merged

Automated Transfer Validation API #29778

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
90f9199
Reenabled validationOptions on 1:1 blob uploads
jaschrep-msft Mar 11, 2022
6be4609
restored block blob hashing tests
jaschrep-msft Mar 14, 2022
e2dfbc8
restored pageblob hashing tests
jaschrep-msft Mar 14, 2022
a04ea6e
Merge branch 'main' of https://github.com/Azure/azure-sdk-for-net int…
jaschrep-msft Mar 14, 2022
b1c8462
reenabled hashing on blob partitioned upload
jaschrep-msft Mar 14, 2022
fe0542c
Merge branch 'main' of https://github.com/Azure/azure-sdk-for-net int…
jaschrep-msft Mar 16, 2022
a7cb849
fixes for samples projects
jaschrep-msft Mar 16, 2022
37eae79
transfer validation in Azure.Storage.Blobs.Perf
jaschrep-msft Mar 16, 2022
5902e5e
added uploadfromfile to Azure.Storage.Blobs.Perf
jaschrep-msft Mar 16, 2022
9efdf74
file cleanup
jaschrep-msft Mar 17, 2022
e3b2641
put blob supports crc
jaschrep-msft Mar 17, 2022
d255fd3
perf fix
jaschrep-msft Mar 25, 2022
6ed99af
reenabled hashing on blob openwrite
jaschrep-msft Mar 29, 2022
fe735d5
Merge branch 'main' of https://github.com/Azure/azure-sdk-for-net int…
jaschrep-msft Mar 30, 2022
f4eddba
recording
jaschrep-msft Mar 30, 2022
ebd05bd
reenabled datalake hashing on upload
jaschrep-msft Apr 1, 2022
de982cf
checkpoint
jaschrep-msft Apr 25, 2022
ac2c4e1
Blob download transactional checksums
jaschrep-msft Apr 25, 2022
9368376
block blob upload hashing
jaschrep-msft Apr 25, 2022
e622c66
fixed tests
jaschrep-msft Apr 28, 2022
78c8708
datalake and shares reenabled
jaschrep-msft May 3, 2022
e3d17a2
tests & bug fixes
jaschrep-msft May 9, 2022
6dc19bc
new openwrite hashing tests
jaschrep-msft May 9, 2022
1fc1842
partitioned upload validation tests
jaschrep-msft May 9, 2022
1fccc42
parallel download validation tests
jaschrep-msft May 10, 2022
9b70d45
recording
jaschrep-msft May 10, 2022
84f3d8b
testing and bug fixes
jaschrep-msft May 10, 2022
a622231
recording
jaschrep-msft May 10, 2022
31b74ef
test fixes
jaschrep-msft May 11, 2022
3d6d8c3
Merge branch 'main' of https://github.com/Azure/azure-sdk-for-net int…
jaschrep-msft Jul 8, 2022
4f3811d
updates on initial review
jaschrep-msft Jul 12, 2022
12ad02b
standardize options names; exportapi
jaschrep-msft Jul 12, 2022
7452a38
cleanup
jaschrep-msft Jul 12, 2022
47eb5b3
minor PR feedback
jaschrep-msft Jul 13, 2022
e995f66
fix build
jaschrep-msft Jul 13, 2022
a1d6a08
default params
jaschrep-msft Jul 13, 2022
e6d37a6
fix overloads
jaschrep-msft Jul 14, 2022
93970c4
changes from shifting defaults
jaschrep-msft Jul 14, 2022
7ddcad8
update test mocks
jaschrep-msft Jul 14, 2022
8a5126b
remove default params
jaschrep-msft Jul 14, 2022
5a8a9c5
codegen source and exportapi
jaschrep-msft Jul 15, 2022
29ea07b
update snippets
jaschrep-msft Jul 15, 2022
daafeb3
codegen
jaschrep-msft Jul 15, 2022
15a5cde
Merge branch 'main' of https://github.com/Azure/azure-sdk-for-net int…
jaschrep-msft Jul 15, 2022
a3e2e26
generate and internalize model
jaschrep-msft Jul 15, 2022
be2a984
removed unnecessary constructor
jaschrep-msft Jul 15, 2022
5beebf1
changelog
jaschrep-msft Jul 15, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
1 change: 1 addition & 0 deletions sdk/storage/Azure.Storage.Blobs.Batch/samples/Azure.Storage.Blobs.Batch.Samples.Tests.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
<Compile Remove="$(AzureStorageSharedTestSources)\AzuriteNUnitFixture.cs" />
<Compile Remove="$(AzureStorageSharedTestSources)\ClientSideEncryptionTestExtensions.cs" />
<Compile Remove="$(AzureStorageSharedTestSources)\StorageTestBase.SasVersion.cs" />
<Compile Remove="$(AzureStorageSharedTestSources)\TransferValidationTestBase.cs" />
<None Include="$(AzureStorageSharedTestSources)\*.xml">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
Expand Down
2 changes: 2 additions & 0 deletions sdk/storage/Azure.Storage.Blobs.Batch/tests/Azure.Storage.Blobs.Batch.Tests.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
<Compile Include="$(AzureStorageSharedTestSources)\**\*.cs" LinkBase="Shared" />
<Compile Remove="$(AzureStorageSharedTestSources)\AzuriteFixture.cs" />
<Compile Remove="$(AzureStorageSharedTestSources)\AzuriteNUnitFixture.cs" />
<Compile Remove="$(AzureStorageSharedTestSources)\TransferValidationTestBase.cs" />
<Compile Remove="$(AzureStorageSharedTestSources)\ClientSideEncryptionTestExtensions.cs" />
<None Include="$(AzureStorageSharedTestSources)\*.xml">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
Expand All @@ -30,6 +31,7 @@
<Compile Include="$(MSBuildThisFileDirectory)..\..\Azure.Storage.Blobs\tests\DisposingContainer.cs" LinkBase="Shared" />
<Compile Include="$(AzureStorageSharedSources)StorageConnectionString.cs" LinkBase="Shared" />
<Compile Include="$(AzureStorageSharedSources)SharedAccessSignatureCredentials.cs" LinkBase="Shared" />
<Compile Include="$(AzureStorageSharedSources)TransferValidationOptionsExtensions.cs" LinkBase="Shared" />
<Compile Include="$(AzureStorageSharedSources)UriExtensions.cs" LinkBase="Shared" />
<Compile Include="$(AzureStorageSharedSources)UriQueryParamsCollection.cs" LinkBase="Shared" />
</ItemGroup>
Expand Down
2 changes: 2 additions & 0 deletions ...zure.Storage.Blobs.ChangeFeed/samples/Azure.Storage.Blobs.ChangeFeed.Samples.Tests.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
<Compile Remove="$(AzureStorageSharedTestSources)\AzuriteFixture.cs" />
<Compile Remove="$(AzureStorageSharedTestSources)\AzuriteNUnitFixture.cs" />
<Compile Remove="$(AzureStorageSharedTestSources)\StorageTestBase.SasVersion.cs" />
<Compile Remove="$(AzureStorageSharedTestSources)\TransferValidationTestBase.cs" />
<Compile Remove="$(AzureStorageSharedTestSources)\ClientSideEncryptionTestExtensions.cs" />
<None Include="$(AzureStorageSharedTestSources)\*.xml">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
Expand All @@ -30,5 +31,6 @@
<Compile Include="$(AzureStorageSharedSources)UriExtensions.cs" LinkBase="Shared" />
<Compile Include="$(AzureStorageSharedSources)UriQueryParamsCollection.cs" LinkBase="Shared" />
<Compile Include="$(AzureStorageSharedSources)StorageExceptionExtensions.cs" LinkBase="Shared" />
<Compile Remove="$(AzureStorageSharedTestSources)\TransferValidationTestBase.cs" />
</ItemGroup>
</Project>
14 changes: 12 additions & 2 deletions sdk/storage/Azure.Storage.Blobs.ChangeFeed/src/LazyLoadingBlobStream.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,8 +105,18 @@ private async Task DownloadBlock(bool async, CancellationToken cancellationToken
HttpRange range = new HttpRange(_offset, _blockSize);

response = async
? await _blobClient.DownloadStreamingAsync(range, cancellationToken: cancellationToken).ConfigureAwait(false)
: _blobClient.DownloadStreaming(range, cancellationToken: cancellationToken);
? await _blobClient.DownloadStreamingAsync(
new BlobDownloadOptions
{
Range = range
},
cancellationToken: cancellationToken).ConfigureAwait(false)
: _blobClient.DownloadStreaming(
new BlobDownloadOptions
{
Range = range
},
cancellationToken: cancellationToken);
_stream = response.Value.Content;
_offset += response.Value.Details.ContentLength;
_lastDownloadBytes = response.Value.Details.ContentLength;
Expand Down
1 change: 1 addition & 0 deletions sdk/storage/Azure.Storage.Blobs.ChangeFeed/tests/Azure.Storage.Blobs.ChangeFeed.Tests.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
<Compile Include="$(AzureStorageSharedTestSources)\**\*.cs" LinkBase="Shared" />
<Compile Remove="$(AzureStorageSharedTestSources)\AzuriteFixture.cs" />
<Compile Remove="$(AzureStorageSharedTestSources)\AzuriteNUnitFixture.cs" />
<Compile Remove="$(AzureStorageSharedTestSources)\TransferValidationTestBase.cs" />
<Compile Remove="$(AzureStorageSharedTestSources)\ClientSideEncryptionTestExtensions.cs" />
<None Include="$(AzureStorageSharedTestSources)\*.xml">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
Expand Down
12 changes: 0 additions & 12 deletions sdk/storage/Azure.Storage.Blobs.ChangeFeed/tests/ChangeFeedFactoryTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,18 +99,12 @@ public async Task ChangeFeedEnabledNoMetaSegmentsBlob()
if (IsAsync)
{
blobClient.Setup(r => r.DownloadStreamingAsync(
default,
default,
default,
default,
CancellationToken.None)).ThrowsAsync(ex);
}
else
{
blobClient.Setup(r => r.DownloadStreaming(
default,
default,
default,
default,
CancellationToken.None)).Throws(ex);
}
Expand Down Expand Up @@ -140,18 +134,12 @@ public async Task ChangeFeedEnabledNoMetaSegmentsBlob()
if (IsAsync)
{
blobClient.Verify(r => r.DownloadStreamingAsync(
default,
default,
default,
default,
CancellationToken.None));
}
else
{
blobClient.Verify(r => r.DownloadStreaming(
default,
default,
default,
default,
CancellationToken.None));
}
Expand Down
32 changes: 16 additions & 16 deletions sdk/storage/Azure.Storage.Blobs.ChangeFeed/tests/ChangeFeedTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,11 +61,11 @@ public async Task GetCursor()

if (IsAsync)
{
blobClient.Setup(r => r.DownloadStreamingAsync(default, default, default, default, default)).ReturnsAsync(downloadResponse);
blobClient.Setup(r => r.DownloadStreamingAsync(default, default)).ReturnsAsync(downloadResponse);
}
else
{
blobClient.Setup(r => r.DownloadStreaming(default, default, default, default, default)).Returns(downloadResponse);
blobClient.Setup(r => r.DownloadStreaming(default, default)).Returns(downloadResponse);
}

if (IsAsync)
Expand Down Expand Up @@ -191,11 +191,11 @@ public async Task GetCursor()

if (IsAsync)
{
blobClient.Verify(r => r.DownloadStreamingAsync(default, default, default, default, default));
blobClient.Verify(r => r.DownloadStreamingAsync(default, default));
}
else
{
blobClient.Verify(r => r.DownloadStreaming(default, default, default, default, default));
blobClient.Verify(r => r.DownloadStreaming(default, default));
}

if (IsAsync)
Expand Down Expand Up @@ -298,11 +298,11 @@ public async Task GetPage()

if (IsAsync)
{
blobClient.Setup(r => r.DownloadStreamingAsync(default, default, default, default, default)).ReturnsAsync(downloadResponse);
blobClient.Setup(r => r.DownloadStreamingAsync(default, default)).ReturnsAsync(downloadResponse);
}
else
{
blobClient.Setup(r => r.DownloadStreaming(default, default, default, default, default)).Returns(downloadResponse);
blobClient.Setup(r => r.DownloadStreaming(default, default)).Returns(downloadResponse);
}

if (IsAsync)
Expand Down Expand Up @@ -508,11 +508,11 @@ public async Task GetPage()

if (IsAsync)
{
blobClient.Verify(r => r.DownloadStreamingAsync(default, default, default, default, default));
blobClient.Verify(r => r.DownloadStreamingAsync(default, default));
}
else
{
blobClient.Verify(r => r.DownloadStreaming(default, default, default, default, default));
blobClient.Verify(r => r.DownloadStreaming(default, default));
}

if (IsAsync)
Expand Down Expand Up @@ -636,11 +636,11 @@ public async Task NoYearsAfterStartTime()

if (IsAsync)
{
blobClient.Setup(r => r.DownloadStreamingAsync(default, default, default, default, default)).ReturnsAsync(downloadResponse);
blobClient.Setup(r => r.DownloadStreamingAsync(default, default)).ReturnsAsync(downloadResponse);
}
else
{
blobClient.Setup(r => r.DownloadStreaming(default, default, default, default, default)).Returns(downloadResponse);
blobClient.Setup(r => r.DownloadStreaming(default, default)).Returns(downloadResponse);
}

if (IsAsync)
Expand Down Expand Up @@ -696,11 +696,11 @@ public async Task NoYearsAfterStartTime()

if (IsAsync)
{
blobClient.Verify(r => r.DownloadStreamingAsync(default, default, default, default, default));
blobClient.Verify(r => r.DownloadStreamingAsync(default, default));
}
else
{
blobClient.Verify(r => r.DownloadStreaming(default, default, default, default, default));
blobClient.Verify(r => r.DownloadStreaming(default, default));
}

if (IsAsync)
Expand Down Expand Up @@ -765,11 +765,11 @@ public async Task NoSegmentsRemainingInStartYear()

if (IsAsync)
{
blobClient.Setup(r => r.DownloadStreamingAsync(default, default, default, default, default)).ReturnsAsync(downloadResponse);
blobClient.Setup(r => r.DownloadStreamingAsync(default, default)).ReturnsAsync(downloadResponse);
}
else
{
blobClient.Setup(r => r.DownloadStreaming(default, default, default, default, default)).Returns(downloadResponse);
blobClient.Setup(r => r.DownloadStreaming(default, default)).Returns(downloadResponse);
}

if (IsAsync)
Expand Down Expand Up @@ -904,11 +904,11 @@ public async Task NoSegmentsRemainingInStartYear()

if (IsAsync)
{
blobClient.Verify(r => r.DownloadStreamingAsync(default, default, default, default, default));
blobClient.Verify(r => r.DownloadStreamingAsync(default, default));
}
else
{
blobClient.Verify(r => r.DownloadStreaming(default, default, default, default, default));
blobClient.Verify(r => r.DownloadStreaming(default, default));
}

if (IsAsync)
Expand Down
8 changes: 4 additions & 4 deletions sdk/storage/Azure.Storage.Blobs.ChangeFeed/tests/SegmentFactoryTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,11 +61,11 @@ public async Task BuildSegment_SegmentCursorNullCurrentShardPath()

if (IsAsync)
{
blobClient.Setup(r => r.DownloadStreamingAsync(default, default, default, default, default)).ReturnsAsync(downloadResponse);
blobClient.Setup(r => r.DownloadStreamingAsync(default, default)).ReturnsAsync(downloadResponse);
}
else
{
blobClient.Setup(r => r.DownloadStreaming(default, default, default, default, default)).Returns(downloadResponse);
blobClient.Setup(r => r.DownloadStreaming(default, default)).Returns(downloadResponse);
}

shardFactory.SetupSequence(r => r.BuildShard(
Expand Down Expand Up @@ -110,11 +110,11 @@ public async Task BuildSegment_SegmentCursorNullCurrentShardPath()

if (IsAsync)
{
blobClient.Verify(r => r.DownloadStreamingAsync(default, default, default, default, default));
blobClient.Verify(r => r.DownloadStreamingAsync(default, default));
}
else
{
blobClient.Verify(r => r.DownloadStreaming(default, default, default, default, default));
blobClient.Verify(r => r.DownloadStreaming(default, default));
}
}
}
Expand Down
32 changes: 16 additions & 16 deletions sdk/storage/Azure.Storage.Blobs.ChangeFeed/tests/SegmentTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,11 +66,11 @@ public async Task GetCursor()

if (IsAsync)
{
blobClient.Setup(r => r.DownloadStreamingAsync(default, default, default, default, default)).ReturnsAsync(downloadResponse);
blobClient.Setup(r => r.DownloadStreamingAsync(default, default)).ReturnsAsync(downloadResponse);
}
else
{
blobClient.Setup(r => r.DownloadStreaming(default, default, default, default, default)).Returns(downloadResponse);
blobClient.Setup(r => r.DownloadStreaming(default, default)).Returns(downloadResponse);
}

shardFactory.SetupSequence(r => r.BuildShard(
Expand Down Expand Up @@ -114,11 +114,11 @@ public async Task GetCursor()

if (IsAsync)
{
blobClient.Verify(r => r.DownloadStreamingAsync(default, default, default, default, default));
blobClient.Verify(r => r.DownloadStreamingAsync(default, default));
}
else
{
blobClient.Verify(r => r.DownloadStreaming(default, default, default, default, default));
blobClient.Verify(r => r.DownloadStreaming(default, default));
}

for (int i = 0; i < shards.Count; i++)
Expand Down Expand Up @@ -153,11 +153,11 @@ public async Task GetCursor_NoShards()

if (IsAsync)
{
blobClient.Setup(r => r.DownloadStreamingAsync(default, default, default, default, default)).ReturnsAsync(downloadResponse);
blobClient.Setup(r => r.DownloadStreamingAsync(default, default)).ReturnsAsync(downloadResponse);
}
else
{
blobClient.Setup(r => r.DownloadStreaming(default, default, default, default, default)).Returns(downloadResponse);
blobClient.Setup(r => r.DownloadStreaming(default, default)).Returns(downloadResponse);
}

SegmentFactory segmentFactory = new SegmentFactory(
Expand All @@ -179,11 +179,11 @@ public async Task GetCursor_NoShards()

if (IsAsync)
{
blobClient.Verify(r => r.DownloadStreamingAsync(default, default, default, default, default));
blobClient.Verify(r => r.DownloadStreamingAsync(default, default));
}
else
{
blobClient.Verify(r => r.DownloadStreaming(default, default, default, default, default));
blobClient.Verify(r => r.DownloadStreaming(default, default));
}
}

Expand Down Expand Up @@ -227,11 +227,11 @@ public async Task GetPage()

if (IsAsync)
{
blobClient.Setup(r => r.DownloadStreamingAsync(default, default, default, default, default)).ReturnsAsync(downloadResponse);
blobClient.Setup(r => r.DownloadStreamingAsync(default, default)).ReturnsAsync(downloadResponse);
}
else
{
blobClient.Setup(r => r.DownloadStreaming(default, default, default, default, default)).Returns(downloadResponse);
blobClient.Setup(r => r.DownloadStreaming(default, default)).Returns(downloadResponse);
}

shardFactory.SetupSequence(r => r.BuildShard(
Expand Down Expand Up @@ -303,11 +303,11 @@ public async Task GetPage()
containerClient.Verify(r => r.GetBlobClient(manifestPath));
if (IsAsync)
{
blobClient.Verify(r => r.DownloadStreamingAsync(default, default, default, default, default));
blobClient.Verify(r => r.DownloadStreamingAsync(default, default));
}
else
{
blobClient.Verify(r => r.DownloadStreaming(default, default, default, default, default));
blobClient.Verify(r => r.DownloadStreaming(default, default));
}

for (int i = 0; i < shards.Count; i++)
Expand Down Expand Up @@ -357,11 +357,11 @@ public async Task GetPage_NoMoreEvents()

if (IsAsync)
{
blobClient.Setup(r => r.DownloadStreamingAsync(default, default, default, default, default)).ReturnsAsync(downloadResponse);
blobClient.Setup(r => r.DownloadStreamingAsync(default, default)).ReturnsAsync(downloadResponse);
}
else
{
blobClient.Setup(r => r.DownloadStreaming(default, default, default, default, default)).Returns(downloadResponse);
blobClient.Setup(r => r.DownloadStreaming(default, default)).Returns(downloadResponse);
}

shardFactory.SetupSequence(r => r.BuildShard(
Expand Down Expand Up @@ -398,11 +398,11 @@ public async Task GetPage_NoMoreEvents()
containerClient.Verify(r => r.GetBlobClient(manifestPath));
if (IsAsync)
{
blobClient.Verify(r => r.DownloadStreamingAsync(default, default, default, default, default));
blobClient.Verify(r => r.DownloadStreamingAsync(default, default));
}
else
{
blobClient.Verify(r => r.DownloadStreaming(default, default, default, default, default));
blobClient.Verify(r => r.DownloadStreaming(default, default));
}

for (int i = 0; i < shards.Count; i++)
Expand Down
Loading