Implement Flexible Handling of 429 Errors in Invoke-CosmosDBRequest

[PlagueHO]

Error 429 is returned by Cosmos DB when a collection has exceeded it's assigned RU's. The typical way of managing this is to back off a short amount of time and repeat the request. This should be configurable within the context object or via some global parameter.

[jasonchester]

@PlagueHO I'm considering taking a stab at this, would appreciate any thoughts on approach.

For determining the backoff, cosmos returns x-ms-retry-after-ms so we may only need to keep track of the number of retries. I was thinking of starting by adding a [int][ref] $Retries parameter to Invoke-CosmosDbRequest in utils.ps1.

Passing it by [ref] will allow me to retry / decrement $Retries when $Retries -gt 0 leaving the caller to decide when to reset the retry counter.

Once it is added to the internal function, it can get incorporated to the user facing functions as needed. Since there needs to be access to the header results, it will likely require switching over to Invoke-WebRequest globally.

Relevant documentation...

• https://docs.microsoft.com/en-us/azure/cosmos-db/request-units#RequestRateTooLarge
• https://blogs.msdn.microsoft.com/bigdatasupport/2015/09/02/dealing-with-requestratetoolarge-errors-in-azure-documentdb-and-testing-performance/

Research

Execute-With-Retry script
https://gist.github.com/alexbevi/34b700ff7c7c53c7780b

[jasonchester]

Here's a quick POC I have working on powershell core... there's got to be a better way to extract the header

# utils.ps1#L505
    do {
        try
        {
            if ($UseWebRequest)
            {
                $restResult = Invoke-WebRequest -UseBasicParsing @invokeRestMethodParameters
            }
            else
            {
                $restResult = Invoke-RestMethod @invokeRestMethodParameters
            }
            $done = $true;
        }
        catch [Microsoft.PowerShell.Commands.HttpResponseException]
        {
            if($_.Exception.Response.StatusCode -eq 429)
            {
                [int] $delay = [int](($_.Exception.Response.Headers | Where-Object Key -eq 'x-ms-retry-after-ms').Value[0])
                Write-Verbose -Message "Retry Caught, delaying $delay ms"
                Start-Sleep -Milliseconds $delay
            }
            else
            {
                Throw $_
            }
        }
        catch [System.Net.WebException]
        {
            <#
                Write out additional exception information into the verbose stream
                In a future version a custom exception type for CosmosDB that
                contains this additional information.
            #>
            if ($_.Exception.Response)
            {
                $exceptionStream = $_.Exception.Response.GetResponseStream()
                $streamReader = New-Object -TypeName System.IO.StreamReader -ArgumentList $exceptionStream
                $exceptionResponse = $streamReader.ReadToEnd()
                if ($exceptionResponse)
                {
                    Write-Verbose -Message $exceptionResponse
                }
            }

            Throw $_
        }
        catch
        {
            Throw $_
        }
    } while ($done -ne $true)

    return $restResult