load.read_file: do not use blobfile for local files

[dtrifiro]

If files are local, there's no need to go through the (optional) blobfile dependency, we can just read them using the standard open/read API.

This solves some issues when trying to load files that have already been downloaded to the HF cache.

See for example vllm-project/vllm#21750

[tiran]

@dtrifiro here is a better version that also handles file:// URI correctly:

import urllib.parse

def read_file(blobpath: str) -> bytes:
    # convert file:// URI to local file path
    if blobpath.startswith("file://"):
        parsed = urllib.parse.urlparse(blobpath)
        if parsed.hostname and parsed.hostname != "localhost":
            # must be None, empty, or localhost
            raise ValueError(f"invalid file URI {blobpath}")
        blobpath = urllib.parse.unquote(parsed.path)

    # a path without :// is a local path
    if "://" not in blobpath:
        with open(blobpath, "rb") as f:
            return f.read()

    if blobpath.startswith(("http://", "https://")):
        # avoiding blobfile for public files helps avoid auth issues, like MFA prompts.
        import requests

        resp = requests.get(blobpath)
        resp.raise_for_status()
        return resp.content

    try:
        import blobfile
    except ImportError as e:
        raise ImportError(
            "blobfile is not installed. Please install it by running `pip install blobfile`."
        ) from e
    with blobfile.BlobFile(blobpath, "rb") as f:
        return f.read()

[dtrifiro]

Updated as suggested

[dtrifiro]

Hey @hauntsaninja, mind taking a look if you have some bandwidth?

Thanks!

[dtrifiro]

A similar solution has been merged in https://github.com/openai/tiktoken/pull/451/files#diff-c2902f8168a36c136ab289157c834cd39fec3758c5269607cd4c7a05d6406058R9