Unicode Decode Error

[dzollinger]

When running s3cmd to add headers, if the object has an existing header that has unicode characters it will fail.

We store user uploads with a hashed object name in a bucket and during upload we add a header for Content-Disposition: filename="originalFileName.jpg". So if the user downloads the image later it gives them the original file name and not the ugly hashed object name.

I am attempting to add Cache-Control to all objects in the bucket and discovered that if you have a unicode character already in the headers (in our case in the originalFileName.jpg) it will fail with the decode error.

Is there a workaround or is this a known issue?

[fviard]

@dzollinger Can you test the latest master? I just pushed some fixes that I think will solve your issue.
With that, having unicode in custom headers will not trigger anymore errors.

But, to be noted, after a few investigation on internet, doing that is a "grey" area.
Mostly it is for "content-disposition", but the original rfc stated that it was supposed to be just ascii.
So, for users, that can work on some internet browsers and not others.
(To be noted that aws s3 itself doesn't care about having unicode there).
See:
https://stackoverflow.com/questions/93551/how-to-encode-the-filename-parameter-of-content-disposition-header-in-http

btw, i did notice that s3cmd has a "content-disposition" argument that is not really used except for the command to generate a "signed url". As an enhancement, it could be good to improve this option to be able to create the good content-disposition value with "filename*=". but that is not the case currently.