Rate throttle git push heroku master

[schneems]

When an app is attempted to be deployed it can fail due to an API rate limit like this:

     Hatchet::App::FailedDeployError:
       Could not deploy 'hatchet-t-3ae0c484c6' (rails5_ruby_schema_format) using 'Hatchet::GitApp' at path: 'repo_fixtures/repos/ci/rails5_ruby_schema_format'
       if this was expected add `allow_failure: true` to your deploy hash.
       output:
       Buildpack: nil
       Repo: https://git.heroku.com/hatchet-t-3ae0c484c6.git
       remote: Compressing source files... done.        
       remote: Building source:        
       remote: 
       remote: !    Your account reached the API rate limit Please wait a few minutes before making new requests        
       remote: 
       To https://git.heroku.com/hatchet-t-3ae0c484c6.git
        ! [remote rejected] master -> master (pre-receive hook declined)
       error: failed to push some refs to 'https://git.heroku.com/hatchet-t-3ae0c484c6.git'

This can happen at both the initial deploy and at the release phase. Currently this rate limit case is unhandled. We are using the general/generic retry mechanism of hatchet to account for this case.

This is not great as the retry behavior has no sleep/backoff behavior and will quickly hit the limit again. Also it only retries if the app failed to deploy, if the allow_failure flag is set it will not be retried but any assertions on the output will fail since the app never actually deployed.

This PR proposes that we hook into the existing PlatformAPI rate limits. This is also done manually at

hatchet/lib/hatchet/test_run.rb

Lines 184 to 189 in a8d2c2f

	PlatformAPI.rate_throttle.call do
	Excon.put(source_put_url,
	expects: [200],
	body: File.read('slug.tgz'))
	end
	end

.

The only difference is that the PlatfromAPI.rate_throttle interface expects a response object that includes a status that returns an integer and headers that returns a hash. We can re-use this logic by generating our own FakeResponse object. In the case where the output matches "reached the API rate limit" then we will tell the rate throttle to sleep and try again (by using 429 response code). When it succeeds, we will tell it we got a successful response (200 status code).

This logic will prevent a failure from rate limiting while git push heroku master is being called. Since the rate throttling rate is tuned as the process is running (the sleep value is remembered and adjusted) by using the same rate throttle mechanism here a rate throttle will inform the rest of our system.