Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Uses urllib handlers to retrieve files from all source types #507

Merged
merged 5 commits into from
Feb 6, 2018
Merged

Uses urllib handlers to retrieve files from all source types #507

merged 5 commits into from
Feb 6, 2018

Conversation

lorengordon
Copy link
Member

@lorengordon lorengordon commented Feb 5, 2018

This patch uses urllib's built-in functionality to retrieve files from remote and local sources. Syntax for any handler supported by urllib.request will work.

The prior custom S3 file download methods are replaced by a custom urllib S3Handler. If boto3 is available, the handler is installed as a urllib opener so it can be used through urlopen. As a result, the s3_source argument is no longer needed and has been removed.

Note: Additional handlers (e.g. Azure) can be supported similarly, just by adding a class with an appropriate method to watchmaker.utils.urllib, and installing the opener.

Also included is a helper function that resolves local file paths to the URI syntax expected by the urllib handlers. Since urllib requires the absolute path, to support relative paths passed by the user, this helper also qualifies relative=>absolute paths. For example:

  • /foo/bar/baz.yaml => file:///foo/bar/baz/.yaml
  • baz.yaml => file:///foo/bar/baz/.yaml
  • file://baz.yaml => file:///foo/bar/baz/.yaml
    • Note: lack of a third-slash after file:// indicates a relative path, which needs to be qualified
  • C:\foo\bar\baz.yaml => file:///C:/foo/bar/baz/.yaml

Example usages:

  • https://host/path/to/config.yaml
  • s3://bucket/key/config.yaml
  • file:///abspath/to/config.yaml
  • file://C:/abspath/to/config.yaml
  • file://relpath/to/config.yaml
  • relpath/to/config.yaml
  • relpath\to\config.yaml
  • /abspath/to/config.yaml
  • C:\abspath\to\config.yaml

Closes #79
Closes #143
Closes #501

Copy link
Contributor

@MarionTheBull MarionTheBull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussion has happened in external channels around this PR. This is the best approach to implement urllib for py2/py3 integration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants