Uses urllib handlers to retrieve files from all source types #507
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch uses urllib's built-in functionality to retrieve files from remote and local sources. Syntax for any handler supported by urllib.request will work.
The prior custom S3 file download methods are replaced by a custom urllib S3Handler. If boto3 is available, the handler is installed as a urllib opener so it can be used through urlopen. As a result, the
s3_source
argument is no longer needed and has been removed.Note: Additional handlers (e.g. Azure) can be supported similarly, just by adding a class with an appropriate method to
watchmaker.utils.urllib
, and installing the opener.Also included is a helper function that resolves local file paths to the URI syntax expected by the urllib handlers. Since urllib requires the absolute path, to support relative paths passed by the user, this helper also qualifies relative=>absolute paths. For example:
/foo/bar/baz.yaml
=>file:///foo/bar/baz/.yaml
baz.yaml
=>file:///foo/bar/baz/.yaml
file://baz.yaml
=>file:///foo/bar/baz/.yaml
file://
indicates a relative path, which needs to be qualifiedC:\foo\bar\baz.yaml
=>file:///C:/foo/bar/baz/.yaml
Example usages:
https://host/path/to/config.yaml
s3://bucket/key/config.yaml
file:///abspath/to/config.yaml
file://C:/abspath/to/config.yaml
file://relpath/to/config.yaml
relpath/to/config.yaml
relpath\to\config.yaml
/abspath/to/config.yaml
C:\abspath\to\config.yaml
Closes #79
Closes #143
Closes #501