storage/swift: better retries and bug fixes

[buchanae]

This PR is not ready, and depends on #551

This upgrades a number of features of our Swift storage retrier:

• logging messages about when retries are happening
• fixing a bugs in error matching in ShouldRetry
• getting rid of MaxRetries in favor of max elapsed time (closes storage/swift: tweak retry config #475)

[buchanae]

Also closes #468

Message is now:

local                upload failed
attempt              0
error                swift: opening host file for URL "swift://buchanan/funnel-test/object-corrupted": open /home/ubuntu/funnel-work-dir/bc087t6nu5c1770h17t0/outputs/buchanan/funnel-test/object-corrupted: no such file or directory
index                0
taskID               bc087t6nu5c1770h17t0
time                 2018-05-19T20:07:19Z
timestamp            2018-05-19T20:07:19.438671954Z
url                  swift://buchanan/funnel-test/object-corrupted

[buchanae]

I've added an environment variable for testing swift with an alternative storage service URL. This allows me to set up a swift proxy that deliberately injects failures into the HTTP request/response, for testing retries and failure handling in Funnel.

example:

export FUNNEL_SWIFT_STORAGE_URL=http://10.50.50.130:8080/v1/AUTH_abced<auth token here>/

TODO:

• figure out how best to get the /AUTH_<token goes here> part of the URL and document that.

[adamstruck]

I strongly prefer MaxRetries to MaxRetryDuration. Making retries duration based makes it pretty much impossible to know how many times Funnel will try to download/upload a particular file.

[buchanae]

I strongly prefer MaxRetries to MaxRetryDuration. Making retries duration based makes it pretty much impossible to know how many times Funnel will try to download/upload a particular file.

You're right. I've reverted that and updated the default Swift config instead.

figure out how best to get the /AUTH_ part of the URL and document that.

The Swift storage URL can be found by going to the OpenStack dashboard > Access & Security page > API Access tab. I've noted this in the code comments.

[buchanae]

I might need to separate the Swift storage URL override. Calling Authenticate() in the constructor seems to have downstream effects on our storage code.

[buchanae]

I've removed the CLI conflict with #551 and the Swift storage URL override code; it's not critical here. I'll revisit it in the future.