Json Arithmetic operation resulted in an overflow

[iSazonov]

Experimenting with new Json API in PowerShell Core repo I get very large output and as result exception "Arithmetic operation resulted in an overflow".

The exception message looks non-user-friendly and non-useful.

Proposal:

• make the exception and message more informative in Json code. ("Result exceeded maximum buffer size (2 Gb?)")
• make the exception and message more informative in System.Buffers code too.

Additional information

The exception comes from

runtime/src/libraries/Common/src/System/Buffers/ArrayBufferWriter.cs

Line 177 in 00813df

int newSize = checked(_buffer.Length + growBy);

Stack trace from PowerShell:

PS > $previous | ConvertTo-Json -Depth 1000 | Out-File -Path c:\temp\q25.txt
ConvertTo-Json: Arithmetic operation resulted in an overflow.

PS > Get-Error

Exception             :
    Type       : System.OverflowException
    TargetSite :
        Name          : CheckAndResizeBuffer
        DeclaringType : System.Text.Json.PooledByteBufferWriter
        MemberType    : Method
        Module        : System.Text.Json.dll
    StackTrace :
   at System.Text.Json.PooledByteBufferWriter.CheckAndResizeBuffer(Int32 sizeHint)
   at System.Text.Json.PooledByteBufferWriter.GetMemory(Int32 sizeHint)
   at System.Text.Json.Utf8JsonWriter.Grow(Int32 requiredSize)
   at System.Text.Json.Utf8JsonWriter.WriteLiteralIndented(ReadOnlySpan`1 escapedPropertyName, ReadOnlySpan`1 value)
   at System.Text.Json.Utf8JsonWriter.WriteNull(JsonEncodedText propertyName)
   at System.Text.Json.WriteStackFrame.WriteObjectOrArrayStart(ClassType classType, JsonEncodedText propertyName, Utf8JsonWriter writer, Boolean writeNull)
   at System.Text.Json.WriteStackFrame.WriteObjectOrArrayStart(ClassType classType, Utf8JsonWriter writer, JsonSerializerOptions options, Boolean writeNull)
   at System.Text.Json.JsonSerializer.Write(Utf8JsonWriter writer, Int32 originalWriterDepth, Int32 flushThreshold, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.JsonSerializer.WriteCore(Utf8JsonWriter writer, Object value, Type type, JsonSerializerOptions options)
   at System.Text.Json.JsonSerializer.Serialize[TValue](Utf8JsonWriter writer, TValue value, JsonSerializerOptions options)
   at Microsoft.PowerShell.Commands.JsonObject.SerializePsProperties(Utf8JsonWriter writer, PSObject pso, Object obj, Boolean isCustomObj, JsonSerializerOptions options)
in C:\Users\sie\Documents\GitHub\iSazonov\PowerShell\src\Microsoft.PowerShell.Commands.Utility\commands\utility\WebCmdlet\JsonObject.cs:line 682
   at Microsoft.PowerShell.Commands.JsonObject.JsonConverterPSObject.Write(Utf8JsonWriter writer, PSObject pso, JsonSerializerOptions options) in
C:\Users\sie\Documents\GitHub\iSazonov\PowerShell\src\Microsoft.PowerShell.Commands.Utility\commands\utility\WebCmdlet\JsonObject.cs:line 641
   at System.Text.Json.JsonPropertyInfoNotNullable`4.OnWrite(WriteStackFrame& current, Utf8JsonWriter writer)
   at System.Text.Json.JsonPropertyInfo.Write(WriteStack& state, Utf8JsonWriter writer)
   at System.Text.Json.JsonSerializer.Write(Utf8JsonWriter writer, Int32 originalWriterDepth, Int32 flushThreshold, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.JsonSerializer.WriteCore(Utf8JsonWriter writer, Object value, Type type, JsonSerializerOptions options)
   at System.Text.Json.JsonSerializer.WriteCore(PooledByteBufferWriter output, Object value, Type type, JsonSerializerOptions options)
   at System.Text.Json.JsonSerializer.WriteCoreString(Object value, Type type, JsonSerializerOptions options)
   at System.Text.Json.JsonSerializer.Serialize(Object value, Type inputType, JsonSerializerOptions options)
   at Microsoft.PowerShell.Commands.JsonObject.ConvertToJson(Object objectToProcess, ConvertToJsonContext& context) in
C:\Users\sie\Documents\GitHub\iSazonov\PowerShell\src\Microsoft.PowerShell.Commands.Utility\commands\utility\WebCmdlet\JsonObject.cs:line 501
   at Microsoft.PowerShell.Commands.ConvertToJsonCommand.EndProcessing() in
C:\Users\sie\Documents\GitHub\iSazonov\PowerShell\src\Microsoft.PowerShell.Commands.Utility\commands\utility\WebCmdlet\ConvertToJsonCommand.cs:line 122
   at System.Management.Automation.Cmdlet.DoEndProcessing() in C:\Users\sie\Documents\GitHub\iSazonov\PowerShell\src\System.Management.Automation\engine\cmdlet.cs:line 187
   at System.Management.Automation.CommandProcessorBase.Complete() in
C:\Users\sie\Documents\GitHub\iSazonov\PowerShell\src\System.Management.Automation\engine\CommandProcessorBase.cs:line 590
    Message    : Arithmetic operation resulted in an overflow.
    Source     : System.Text.Json
    HResult    : -2146233066
CategoryInfo          : NotSpecified: (:) [ConvertTo-Json], OverflowException
FullyQualifiedErrorId : System.OverflowException,Microsoft.PowerShell.Commands.ConvertToJsonCommand
InvocationInfo        :
    MyCommand        : ConvertTo-Json
    ScriptLineNumber : 1
    OffsetInLine     : 13
    HistoryId        : 17
    Line             : $previous | ConvertTo-Json -Depth 1000 | Out-File -Path c:\temp\q25.txt
    PositionMessage  : At line:1 char:13
                       + $previous | ConvertTo-Json -Depth 1000 | Out-File -Path c:\temp\q25.t …
                       +             ~~~~~~~~~~~~~~~~~~~~~~~~~~
    InvocationName   : ConvertTo-Json
    CommandOrigin    : Internal
ScriptStackTrace      : at <ScriptBlock>, <No file>: line 1

PowerShell can be downloaded from the PR PowerShell/PowerShell#11198
Repo scripts:

            $start = 1
            $previous = @{
                Depth = 0
                Next1 = $null
                Next2 = $null
            }

            25..$start | ForEach-Object {
                $current = @{
                    Depth = $_
                    Next1 = $previous
                    Next2 = $previous
                }
                $previous = $current
            }

$previous | ConvertTo-Json -Depth 1000 | Out-File -Path c:\temp\q25.txt

[ahsonkhan]

The exception comes from

runtime/src/libraries/Common/src/System/Buffers/ArrayBufferWriter.cs

Line 177 in 00813df

int newSize = checked(_buffer.Length + growBy);

This particular exception (at least within the JSON serializer code) comes from the internal PooledByteBufferWriter:

runtime/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/PooledByteBufferWriter.cs

Line 141 in 57e5755

int newSize = checked(_rentedBuffer.Length + growBy);

Proposal:

• make the exception and message more informative in Json code. ("Result exceeded maximum buffer size (2 Gb?)")
• make the exception and message more informative in System.Buffers code too.

Both of those seem reasonable to me. Adding an if-check on the size that throws makes sense (though I don't know if, in some contexts, checking/throwing might introduce noticeable overhead). That said, it is possible that we have other edge cases where checked arithmetic is used to guard against integer overflow or OOM when asking for buffers that won't fit in a T[] or sizes that the arraypool can't honor.

Anyone interested in a PR with tests?

[felipepessoto]

@ahsonkhan I'm interested, could you assign me?

[ahsonkhan]

@felipepessoto done.

[felipepessoto]

@ahsonkhan currently the checked keyword only validates the integer overflow (if the value would exceed int.MaxValue).
But ArrayPool.Shared.Rent won't work for int.MaxValue anyway, because we can't allocate new byte[] of int.MaxValue size (I believe that is because of array headers overhead). Throwing another non-user-friendly exception in rare occasions where the newSize is equal to int.MaxSize or very close to it.

In this case, should we validate if the newSize is smaller than (int.MaxValue - 56) (for 64 bit process)?
For 32bits processes, I guess the header is a bit smaller. But it will generate an OutOfMemoryException anyway, unless we set the LARGEADDRESSAWARE flag (I haven't tested this scenario)

@sywhang

[mikedn]

because we can't allocate new byte[] of int.MaxValue size (I believe that is because of array headers overhead)

It's a bit more complicated than headers overhead.

https://docs.microsoft.com/en-us/dotnet/api/system.array?view=netcore-3.1

The array size is limited to a total of 4 billion elements, and to a maximum index of 0X7FEFFFFF in any given dimension (0X7FFFFFC7 for byte arrays and arrays of single-byte structures).

[felipepessoto]

Nice, I didn't know that. So we can compare it to Array.MaxByteArrayLength or 0X7FFFFFC7

[felipepessoto]

Another question, throwing JsonException with a custom message is ok for this scenario? Or should I create a specific Exception?

Thanks

[iSazonov]

@felipepessoto The question about Json exceptions is tracked in #610

[mikedn]

So we can compare it to Array.MaxByteArrayLength or 0X7FFFFFC7

Using MaxByteArrayLength would be better but AFAIK it's internal to corelib and this code seems to be in another assembly.

Actually, just clamping the result to int.MaxValue might be enough - if you attempt to allocate an int.MaxValue length array the runtime will generate:

System.OutOfMemoryException: Array dimensions exceeded supported range.

which seems like a reasonable error to me.

[felipepessoto]

@mikedn, if we let the array constructor throw the OutOfMemoryException, we'll be back to the original problem where the Exception message is not clear. I could validate the size and throw an OutOfMemoryException with a custom message.

Stephen also gave a suggestion here: #1308 (comment)

[mikedn]

we'll be back to the original problem where the Exception message is not clear

I don't think so, the original message is "Arithmetic operation resulted in an overflow." which is indeed not very clear, it's not obvious to anyone what the cause of the problem is. The "Array dimensions exceeded supported range" message should be obvious enough. Unless you really want to tell the user - "Hey, your JSON is yugeeee!" - so there's definitely no confusion :)

Besides, this looks very much like a corner case. The fact that a 2GB JSON is first generated in memory and then written to a file seems pretty dubious to me.

[felipepessoto]

we'll be back to the original problem where the Exception message is not clear

I don't think so, the original message is "Arithmetic operation resulted in an overflow." which is indeed not very clear, it's not obvious to anyone what the cause of the problem is. The "Array dimensions exceeded supported range" message should be obvious enough. Unless you really want to tell the user - "Hey, your JSON is yugeeee!" - so there's definitely no confusion :)

Besides, this looks very much like a corner case. The fact that a 2GB JSON is first generated in memory and then written to a file seems pretty dubious to me.

You are right. Agreed

[ahsonkhan]

Re-opening until #32587 is done.

[ahsonkhan]

Re-opening until #34040 is done.