LargeBlobTest.uploadlargeblobinsingleuploadasync Causing OOM

[alzimmermsft]

Beginning on 2020-07-01 LargeBlobTest upload large blob in single upload async leads to an OutOfMemoryError to be thrown.

I'm not certain if this is just a test failure or an actually issue in the underlying code.

Jul 01, 2020 1:31:40 PM io.netty.channel.AbstractChannelHandlerContext invokeExceptionCaught
WARNING: An exception 'java.lang.OutOfMemoryError: Java heap space' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:
java.lang.OutOfMemoryError: Java heap space
	at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
	at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
	at com.azure.storage.common.implementation.PayloadSizeGate.write(PayloadSizeGate.java:44)
	at com.azure.storage.common.implementation.UploadUtils$$Lambda$289/1562764987.apply(Unknown Source)
	at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:367)
	at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:243)
	at reactor.core.publisher.FluxFilter$FilterSubscriber.onNext(FluxFilter.java:107)
	at reactor.core.publisher.FluxRepeat$RepeatSubscriber.onNext(FluxRepeat.java:83)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121)
	at reactor.core.publisher.FluxJust$WeakScalarSubscription.request(FluxJust.java:99)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:162)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2125)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:1999)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:90)
	at reactor.core.publisher.FluxJust.subscribe(FluxJust.java:70)
	at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62)
	at reactor.core.publisher.FluxRepeat$RepeatSubscriber.resubscribe(FluxRepeat.java:113)
	at reactor.core.publisher.FluxRepeat$RepeatSubscriber.onComplete(FluxRepeat.java:97)
	at reactor.core.publisher.FluxRepeat.subscribeOrReturn(FluxRepeat.java:54)
	at reactor.core.publisher.Flux.subscribe(Flux.java:8311)
	at reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onComplete(FluxConcatArray.java:207)
	at reactor.core.publisher.FluxConcatArray.subscribe(FluxConcatArray.java:80)
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:153)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.ignoreDone(MonoIgnoreThen.java:190)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreInner.onComplete(MonoIgnoreThen.java:240)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:141)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1755)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:144)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1755)
	at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:73)

[gapra-msft]

Unsure of why these errors are occurring. I have a feeling it is due to the VM not being configured to have enough heap space since this feature was added. I've disabled them for now #12728.
@kasobol-msft would you mind taking a look at these to see if there is some specific heap space setting we need to enable on the CI VMs?

[kasobol-msft]

We should set MemoryOptions to something bigger than 5GB here https://github.com/Azure/azure-sdk-for-java/blob/master/sdk/storage/tests.yml .

The default seems to be 3GB

azure-sdk-for-java/eng/pipelines/templates/variables/globals.yml

Line 5 in 02155b4

MemoryOptions: '-Xmx3072m'

[kasobol-msft]

@kurtzeborn @mikeharder This is one example where we're exhausting memory in a test we discussed today.

[kasobol-msft]

Attempting to run these in #23119

[kasobol-msft]

Consider using #19601 for some of disabled tests.

[kasobol-msft]

Todo: datalake.