Standardize content_type handling when writing files

[TomAugspurger]

Would it be helpful to standardize the ability to set the content type of files written by fsspec? Several backends have / are requesting the ability to set this (https://github.com/fsspec/s3fs/blob/4f289eaa34dfe8337a72f5a0148c41a44793fde0/s3fs/core.py#L971-L974, fsspec/adlfs#294). Each backend will typically have a different "native" was for setting things. With S3 it's a ContentType keyword. With azure, it's content_settings=azure.storage.blob.ContentSettings(content_type=...).

This proposal would make content_type a proper keyword of pipe, put, (and any others writing data). The backend would be responsible for setting it appropriately.

We might also need to standardize a default behavior. s3fs uses a library to guess the content type. I'm not sure if that's appropriate (if it were, I'd think that boto / azure-storage-blob would do it?).

[martindurant]

You are right that this MIME-like thing applies to a few backends, but by no means all. I would be happy for the backends that do have this concept to support a standerdised kwargs which can be translated into whatever specific thing they each require - but of course the direct route to doing the same thing needs to remain available.

About guessing the content type... it's a little controversial, to be sure.

[ollie-bell]

I am interested in this feature if it's still being considered? In particular my use case is writing netcdf to a cloud storage bucket via the simplecache protocol. I can't seem to inject content_type="application/netcdf" for example into the fsspec.open call.

with fsspec.open(f"simplecache::{path.as_uri()}", mode="wb") as f:
    ds.to_netcdf(f.name, engine="h5netcdf")

Seems to me like this line needs to be modified to get kwargs from **self.fs.storage_options?

filesystem_spec/fsspec/core.py

Line 105 in f2c7717

f = self.fs.open(self.path, mode=mode)

[martindurant]

I am not sure how indeed you can inject extra arguments into that place - there are so many different places already that kwargs can get picked up. storage_options are normally about configuring the filesystem instances (simplecache and gcsfs in this case), although there is precedent for changing the read cache and blocksize for files. I'd be happy to see a solution...

[ollie-bell]

my very hacky solution to test out was to modify that line 105 to

f = self.fs.open(self.path, mode=mode, **self.fs.storage_options.get("test_kwargs", {})

and then instantiated like

fsspec.open(..., test_kwargs={"content_type": ...})

I haven't used fsspec enough to know if this could be made into something general (obviously renaming "test_kwargs" to something different) or if it just happened to work for my use case.

... but I have an acceptable workaround atm whereby I call fs.setxattrs(path, content_type=...) afterwards (works for GCS)

[martindurant]

I fear that would result in "unexpected argument" for at least some backends.

I have an acceptable workaround atm whereby I call fs.setxattrs(path, content_type=...) afterwards

This is also backend-specific :|. There is probably more that we can do to standardise these extra pieces of functionality in fsspec.

[Cowen]

Consider this another vote in favour of a solution like this. Or alternatively, consider this someone else experiencing this pain with nowhere else to vent. 😄

I have an internal library that I'm trying to replace with fsspec, since it almost tries to solve almost exactly the same problem. But lacking a consistent way to set a MIME type on put is causing me extra frustrations.