Use fsspec in load to resolve more paths/URLs from storage backends

[ananthsub]

What does this PR do?

Fixes #3691

Before submitting

• Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together? Otherwise, we ask you to create a separate PR for every change.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?
• Did you verify new and existing tests pass locally with your changes?
• If you made a notable change (that affects users), did you update the CHANGELOG?

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[Borda]

it shall be fine

…

@Borda <https://github.com/Borda> for these tests to pass, we need to pip install fsspec[http] extra in CI. Can we do that? Shouldn't be a problem right?

[ananthsub]

@Borda @awaelchli aiohttps needs to be included in the dependencies for the tests to run

[williamFalcon]

@ananthsub so this adds a dep to PL?

[ananthsub]

@williamFalcon it should already be there from relying on fsspec. https://github.com/intake/filesystem_spec/blob/master/setup.py#L45

from @awaelchli

oh so it needs be installed with [http] as brackets
before we didn't get that because http was redirected to torch.hub.load
yep, so if I do pip install fsspec[http], i get the aiohttp
I guess we need at least http for the CI test suite

[Borda]

@ananthsub so this adds a dep to PL?

if it is just for testing we are fine...

[ananthsub]

@Borda any pointers on how to include fsspec's http dependencies for the tests?

[Borda]

@Borda any pointers on how to include fsspec's http dependencies for the tests?

Let me see, the case is that as we install requirements/devel.txt which chain all and pip do not like listed one package twice...

[Borda]

@Borda any pointers on how to include fsspec's http dependencies for the tests?

it seems that you already added them manually:
https://github.com/intake/filesystem_spec/blob/974e9c9b172f39816d929fe51e604ff7829ce1d3/setup.py#L45

[Borda]

@awaelchli I have tried #3692 (comment) and it was the same, then I reverted the change...

[codecov]

Codecov Report

Merging #3692 into master will increase coverage by 4%.
The diff coverage is 100%.

@@           Coverage Diff           @@
##           master   #3692    +/-   ##
=======================================
+ Coverage      84%     88%    +4%     
=======================================
  Files         111     112     +1     
  Lines        8756    8915   +159     
=======================================
+ Hits         7354    7809   +455     
+ Misses       1402    1106   -296