A little more rate limit tweaking for v0.4.4 #142
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds a bunch of little tweaks related to the recent rate limit work. It should be more-or-less ready for release after this.
The main thing is adding a 60-second delay for 429 (rate limit) responses. Looking back at the docstring for the
RateLimitError
class, I think I might have originally intended to always raise instead of retry these, but git history shows it never actually worked that way. I want to get a patch release out with the new retry timings and logs, but patch release doesn’t seem like a good time change whether a response gets retried, so I’ve just made a longer delay here for now.Then some other assorted bits and bobs:
Downgrades the log level when we catch and retry exceptions (used to be WARN, now just INFO). The whole point of retry is to make this situations not special, so I don’t think it’s actually worth a warning.
Actually handle all forms of
Retry-After
header, and move logic to_utils
. We didn’t previously handle the date form of the header, just the integer form that indicated a number of seconds. Now we handle both. I also moved the logic out of theexceptions
module since it was too much special logic (arguably this was already the case even before I improved the parsing).Fix the
total_time
calculation forWaybackRetryError
. I’m not sure what I was thinking with the previous code, which was… not even sort-of correct.Add
read_and_close()
everywhere that we raise an exception. We did it in some spots, but had missed others.