storage/swift: tweak retry config

[buchanae]

We're getting frequent failures from our swift cluster (503 unavailable) and the retrier isn't doing enough. In it's current default configuration, it will only retry for a couple seconds.

Just how long to retry is an open question, but I'd say we should err on the side of retrying for too long, rather than too briefly. These are large files being uploaded, created as the result of processes that potentially take multiple days, so it's worth retrying for hours even.

More technically, we could drop the MaxRetries in favor of MaxElapsedTime, which I think is a better configuration. It's more difficult to calculate the max amount of time you're retrying from a count number with exponential backoff involved.

[buchanae]

Somewhat related: it'd be nice to know exactly how the underlying ncw/swift library handles 503 errors. Does one 503 error mean the entire file needs to be retried? Can it have better handling of segment uploads to the whole upload progress isn't lost?