infer `skip_signature: True` when using`from_url`

[vincentsarago]

I see that's it's expected that when using obstore.store.from_url with an amazon s3 https url we get an S3 store but to me I would also expect that skip_signature: True would be set. Maybe I'm misunderstanding but https url should be considered as public 🤷

import obstore as obs
store = obs.store.from_url("https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif")

store
>>> S3Store(bucket="sentinel-cogs", prefix="sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif")

obs.get_range(store, "", start=0, end=10)
>>> GenericError: Generic {
    store: "S3",
    source: Reqwest {
        retries: 10,
        max_retries: 10,
        elapsed: 5.794443875s,
        retry_timeout: 180s,
        source: reqwest::Error {
            kind: Request,
            url: "http://169.254.169.254/latest/api/token",
            source: hyper_util::client::legacy::Error(
                Connect,
                ConnectError(
                    "tcp connect error",
                    Os {
                        code: 64,
                        kind: Uncategorized,
                        message: "Host is down",
                    },
                ),
            ),
        },
    },
}

# with skip_signature 
store = obs.store.from_url("https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif", config={"skip_signature": True})
obs.get_range(store, "", start=0, end=10)
>>> Bytes(b"II*\x00\x08\x00\x00\x00\x13\x00")

[kylebarron]

I don't think there's anything we can do about this, because reading the upstream code, skip_signature=True overrides any passed-in authentication.

So we couldn't default to skip_signature=True if there were ever a case where someone wanted to use an https://bucket.s3.us-west-2.amazonaws.com URL with authentication. I think this is too likely to opt-in to skip_signature ourselves.

[kylebarron]

I don't think there's anything we can realistically do here, so I'll close this issue.

[cas--]

I was curious about this issue, perhaps a misunderstanding that https URLs are always public. An S3 bucket you will get two option for S3 URI and https URL regardless of whether they are public or private. As Kyle mentions using skip_signature=True with a private URL will fail authentication.

I guess users can use the same technique as inferring the bucket region¹ to infer the skip_signature:

import requests
from obstore.store import from_url

def is_url_public(url, timeout=10):
    """Determine if a URL public and accessible."""
    try:
        response = requests.head(url, timeout=timeout, allow_redirects=True)
        response.raise_for_status()
        return True
    except requests.exceptions.RequestException:
        return False

url = "https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif"
store = from_url(url, skip_signature=is_url_public(url))
store.get_range("", start=0, end=10)

Alternatively I found that arrow-rs-obstore will read env vars with AWS_ prefix that match config keys so can set AWS_SKIP_SIGNATURE which might help if you know all URLs will be public.

os.environ["AWS_SKIP_SIGNATURE"] = "true"
store = from_url(url)
store.get_range("", start=0, end=10)

Footnotes

1. Interestingly S3 https URLs seems better suited with from_url since it will extract the S3 bucket region from the URL. ↩