UnicodeDecodeError #847

baiyanchun · 2017-03-22T09:27:43Z

I used s3cmd command (the latest s3cmd code I use) to put a file named in Chinese, when I use s3cmd info bucket, An unexpected error has occurred. The following is my operation process and result.

1 Create a bucket
s3cmd mb s3://bucket
Bucket 's3://bucket/' created

2 Create a file named in Chinese
touch 中文.txt

3 Put file to s3://bucket
s3cmd put 中文.txt s3://bucket
upload: '中文.txt' -> 's3://bucket/中文.txt' [1 of 1]
0 of 0 0% in 0s 0.00 B/s done

4 Ls this file
s3cmd ls s3://bucket
2017-03-21 02:55 0 s3://bucket/中文.txt

5 Get bucket info
s3cmd info s3://bucket

The error message:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Invoked as: /usr/local/bin/s3cmd info s3://bucket
Problem: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 496: ordinal not in range(128)
S3cmd: 1.6.1
python: 2.7.6 (default, Oct 26 2016, 20:30:19)
[GCC 4.8.4]
environment LANG=en_US.UTF-8

Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/s3cmd-1.6.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 2996, in
File "/usr/local/lib/python2.7/dist-packages/s3cmd-1.6.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 2915, in main
File "/usr/local/lib/python2.7/dist-packages/s3cmd-1.6.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 918, in cmd_info
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 496: ordinal not in range(128)

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

baiyanchun · 2017-03-22T09:30:12Z

the detail error log.
s3cmd.txt

mdomsch · 2017-03-23T21:25:19Z

The next output line should be a policy. By chance does your policy have unicode characters in it? That's where it's failing.

…

On Wed, Mar 22, 2017 at 4:30 AM, baiyanchun ***@***.***> wrote: the detail error log. s3cmd.txt <https://github.com/s3tools/s3cmd/files/860924/s3cmd.txt> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#847 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAqDqhkBz84sgwk_4UA62Ml3OeqjxtDyks5roOolgaJpZM4Mk7PB> .

fviard · 2017-04-02T12:47:50Z

@baiyanchun Any feedback regarding the idea of mdomsch?

fviard · 2017-06-25T21:45:54Z

@baiyanchun Is it possible for your to confirm that the issue come from having chinese characters somewhere in your policy? and if you paste us an extract here? so I can reproduce and fix?
Thank you

jdurand · 2018-05-08T17:18:44Z

I get this issue when trying to transfer file with accents in the filename (i.e. Capture_d’écran_2015-10-23_à_09.20.45.png)

jdurand · 2018-05-08T17:36:22Z

Commenting out the console output fixes the issue:

s3cmd/s3cmd

Line 1103 in 65b34b7

    
           output("remote copy: '%(src)s' -> '%(dst)s'" % { "src" : src_uri, "dst" : dst_uri })

Looks like an issue with python incorrectly detecting the filename encoding or something.

fviard · 2018-05-08T19:50:36Z

@jdurand : Can you tell me what is the version of Python and s3cmd that you are using? Also the command that you are running (hiding private parts of paths).
Something interesting would be for you to run the command with the '-d' flag to have full debug output.

Also in your configuration you can check if you have any 'encoding' set.

Because what surprise me is that normally in this part of the code, path are already converted to Unicode to not cause such an issue.

jdurand · 2018-05-08T20:45:53Z

Python 2.7.15
s3cmd 2.0.1

I'm not familiar with Python, what command should I run or which config file should I check?
I'm running a Homebrew Python on macOS.

fviard · 2018-05-08T22:42:56Z

What is the "s3cmd" command line with arguments that you run for your transfer that is failing?
you just have to had " -d" at the end of the command line that you run to display the full debug log.

jdurand · 2018-05-08T23:07:51Z

I was running s3cmd sync s3://SRC s3://DEST but got into some other issues... I large number of files were missing so I ended up syncing my buckets with aws-cli.

fviard · 2018-05-08T23:43:39Z

Would it still be possible for you to run that command with the -d and look what is the root cause of the failure?
There could be interesting issues coming from the fact that you are using s3cmd on macOS.
As I'm don't have a macOS device to test.

jdurand · 2018-05-09T00:24:40Z

s3cmd sync s3://[...]/test s3://[...]/ --access_key=[...] --secret_key=[...] --acl-public -d

DEBUG: s3cmd version 2.0.1
DEBUG: ConfigParser: Reading file '/Users/[...]/.s3cfg'
DEBUG: ConfigParser: access_key->AK...17_chars...Q
DEBUG: ConfigParser: bucket_location->US
DEBUG: ConfigParser: cloudfront_host->cloudfront.amazonaws.com
DEBUG: ConfigParser: cloudfront_resource->/2010-07-15/distribution
DEBUG: ConfigParser: default_mime_type->binary/octet-stream
DEBUG: ConfigParser: delete_removed->False
DEBUG: ConfigParser: dry_run->False
DEBUG: ConfigParser: encoding->UTF-8
DEBUG: ConfigParser: encrypt->False
DEBUG: ConfigParser: follow_symlinks->False
DEBUG: ConfigParser: force->False
DEBUG: ConfigParser: get_continue->False
DEBUG: ConfigParser: gpg_command->None
DEBUG: ConfigParser: gpg_decrypt->%(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_encrypt->%(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_passphrase->mk...4_chars...o
DEBUG: ConfigParser: guess_mime_type->True
DEBUG: ConfigParser: host_base->s3.amazonaws.com
DEBUG: ConfigParser: host_bucket->%(bucket)s.s3.amazonaws.com
DEBUG: ConfigParser: human_readable_sizes->False
DEBUG: ConfigParser: list_md5->False
DEBUG: ConfigParser: log_target_prefix->
DEBUG: ConfigParser: preserve_attrs->True
DEBUG: ConfigParser: progress_meter->True
DEBUG: ConfigParser: proxy_host->
DEBUG: ConfigParser: proxy_port->0
DEBUG: ConfigParser: recursive->False
DEBUG: ConfigParser: recv_chunk->4096
DEBUG: ConfigParser: reduced_redundancy->False
DEBUG: ConfigParser: secret_key->5M...37_chars...h
DEBUG: ConfigParser: send_chunk->4096
DEBUG: ConfigParser: simpledb_host->sdb.amazonaws.com
DEBUG: ConfigParser: skip_existing->False
DEBUG: ConfigParser: socket_timeout->300
DEBUG: ConfigParser: urlencoding_mode->normal
DEBUG: ConfigParser: use_https->False
DEBUG: ConfigParser: verbosity->WARNING
DEBUG: Updating Config.Config access_key -> [...]
DEBUG: Updating Config.Config cache_file ->
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config secret_key -> [...]
DEBUG: Updating Config.Config verbosity -> 10
DEBUG: Unicodising 'sync' using UTF-8
DEBUG: Unicodising 's3://[SRC]/test' using UTF-8
DEBUG: Unicodising 's3://[DEST]/' using UTF-8
DEBUG: Command: sync
INFO: Retrieving list of remote files for s3://[SRC]/test ...
DEBUG: CreateRequest: resource[uri]=/
DEBUG: ===== Send_request inner request to determine the bucket region =====
DEBUG: CreateRequest: resource[uri]=/
DEBUG: Using signature v4
DEBUG: get_hostname([SRC]): s3.amazonaws.com
DEBUG: canonical_headers = host:s3.amazonaws.com
x-amz-content-sha256:[...]
x-amz-date:[...]

DEBUG: Canonical Request:
GET
/[SRC]/
location=
host:s3.amazonaws.com
x-amz-content-sha256:[...]
x-amz-date:[...]

host;x-amz-content-sha256;x-amz-date
[...]
----------------------
DEBUG: signature-v4 headers: {'x-amz-content-sha256': u'[...]', 'Authorization': u'AWS4-HMAC-SHA256 Credential=[...]/20180509/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=[...]', 'x-amz-date': '[...]'}
DEBUG: Processing request, please wait...
DEBUG: get_hostname([SRC]): s3.amazonaws.com
DEBUG: ConnMan.get(): creating new connection: http://s3.amazonaws.com
DEBUG: non-proxied HTTPConnection(s3.amazonaws.com, None)
DEBUG: format_uri(): /[SRC]/?location
DEBUG: Sending request method_string='GET', uri=u'/[SRC]/?location', headers={'x-amz-content-sha256': u'[...]', 'Authorization': u'AWS4-HMAC-SHA256 Credential=[...]/20180509/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=[...]', 'x-amz-date': '[...]'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (http://s3.amazonaws.com#1)
DEBUG: Response:
{'data': '<?xml version="1.0" encoding="UTF-8"?>\n<LocationConstraint xmlns="http://s3.amazonaws.com/doc/2006-03-01/"/>',
 'headers': {'content-type': 'application/xml',
             'date': 'Wed, 09 May 2018 00:15:13 GMT',
             'server': 'AmazonS3',
             'transfer-encoding': 'chunked',
             'x-amz-id-2': '[...]',
             'x-amz-request-id': '[...]'},
 'reason': 'OK',
 'status': 200}
DEBUG: ===== END send_request inner request to determine the bucket region ('us-east-1') =====
DEBUG: Using signature v4
DEBUG: get_hostname([SRC]): [SRC].s3.amazonaws.com
DEBUG: canonical_headers = host:[SRC].s3.amazonaws.com
x-amz-content-sha256:[...]
x-amz-date:[...]

DEBUG: Canonical Request:
GET
/
prefix=test
host:[SRC].s3.amazonaws.com
x-amz-content-sha256:[...]
x-amz-date:[...]

host;x-amz-content-sha256;x-amz-date
[...]
----------------------
DEBUG: signature-v4 headers: {'x-amz-content-sha256': u'[...]', 'Authorization': u'AWS4-HMAC-SHA256 Credential=[...]/20180509/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=[...]', 'x-amz-date': '[...]'}
DEBUG: Processing request, please wait...
DEBUG: get_hostname([SRC]): [SRC].s3.amazonaws.com
DEBUG: ConnMan.get(): creating new connection: http://[SRC].s3.amazonaws.com
DEBUG: non-proxied HTTPConnection([SRC].s3.amazonaws.com, None)
DEBUG: format_uri(): /?prefix=test
DEBUG: Sending request method_string='GET', uri=u'/?prefix=test', headers={'x-amz-content-sha256': u'[...]', 'Authorization': u'AWS4-HMAC-SHA256 Credential=[...]/20180509/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=[...]', 'x-amz-date': '[...]'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (http://[SRC].s3.amazonaws.com#1)
DEBUG: Response:
{'data': '<?xml version="1.0" encoding="UTF-8"?>\n<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>[SRC]</Name><Prefix>test</Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>test/</Key><LastModified>2018-05-09T00:13:22.000Z</LastModified><ETag>&quot;d41d8cd98f00b204e9800998ecf8427e&quot;</ETag><Size>0</Size><Owner><ID>94f2229fabc67d1095f5ae2bdaf911de1d49bffb5f14a945b88018648d479f35</ID><DisplayName>[...]77</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>test/test l\xe2\x80\x99e\xcc\x81e\xcc\x81e\xcc\x81e\xcc\x81e\xcc\x81a\xcc\x80a\xcc\x80a\xcc\x80a\xcc\x80a\xcc\x80.txt</Key><LastModified>2018-05-09T00:13:44.000Z</LastModified><ETag>&quot;d8e8fca2dc0f896fd7cb4cb0031ba249&quot;</ETag><Size>5</Size><Owner><ID>94f2229fabc67d1095f5ae2bdaf911de1d49bffb5f14a945b88018648d479f35</ID><DisplayName>[...]77</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>',
 'headers': {'content-type': 'application/xml',
             'date': 'Wed, 09 May 2018 00:15:13 GMT',
             'server': 'AmazonS3',
             'transfer-encoding': 'chunked',
             'x-amz-bucket-region': 'us-east-1',
             'x-amz-id-2': '[...]',
             'x-amz-request-id': '342B9092890629BF'},
 'reason': 'OK',
 'status': 200}
DEBUG: Applying --exclude/--include
DEBUG: CHECK: test/
DEBUG: PASS: u'test/'
DEBUG: CHECK: test/test l’éééééààààà.txt
DEBUG: PASS: u'test/test l\u2019e\u0301e\u0301e\u0301e\u0301e\u0301a\u0300a\u0300a\u0300a\u0300a\u0300.txt'
INFO: Retrieving list of remote files for s3://[DEST]/test ...
DEBUG: CreateRequest: resource[uri]=/
DEBUG: ===== Send_request inner request to determine the bucket region =====
DEBUG: CreateRequest: resource[uri]=/
DEBUG: Using signature v4
DEBUG: get_hostname([DEST]): s3.amazonaws.com
DEBUG: canonical_headers = host:s3.amazonaws.com
x-amz-content-sha256:[...]
x-amz-date:20180509T001513Z

DEBUG: Canonical Request:
GET
/[DEST]/
location=
host:s3.amazonaws.com
x-amz-content-sha256:[...]
x-amz-date:20180509T001513Z

host;x-amz-content-sha256;x-amz-date
[...]
----------------------
DEBUG: signature-v4 headers: {'x-amz-content-sha256': u'[...]', 'Authorization': u'AWS4-HMAC-SHA256 Credential=[...]/20180509/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=[...]', 'x-amz-date': '[...]'}
DEBUG: Processing request, please wait...
DEBUG: get_hostname([DEST]): s3.amazonaws.com
DEBUG: ConnMan.get(): re-using connection: http://s3.amazonaws.com#1
DEBUG: format_uri(): /[DEST]/?location
DEBUG: Sending request method_string='GET', uri=u'/[DEST]/?location', headers={'x-amz-content-sha256': u'[...]', 'Authorization': u'AWS4-HMAC-SHA256 Credential=[...]/20180509/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=[...]', 'x-amz-date': '[...]'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (http://s3.amazonaws.com#2)
DEBUG: Response:
{'data': '<?xml version="1.0" encoding="UTF-8"?>\n<LocationConstraint xmlns="http://s3.amazonaws.com/doc/2006-03-01/">us-east-2</LocationConstraint>',
 'headers': {'content-type': 'application/xml',
             'date': 'Wed, 09 May 2018 00:15:14 GMT',
             'server': 'AmazonS3',
             'transfer-encoding': 'chunked',
             'x-amz-id-2': '[...]',
             'x-amz-request-id': '[...]'},
 'reason': 'OK',
 'status': 200}
DEBUG: ===== END send_request inner request to determine the bucket region (u'us-east-2') =====
DEBUG: Using signature v4
DEBUG: get_hostname([DEST]): [DEST].s3.amazonaws.com
DEBUG: canonical_headers = host:[DEST].s3.amazonaws.com
x-amz-content-sha256:[...]
x-amz-date:20180509T001513Z

DEBUG: Canonical Request:
GET
/
prefix=test
host:[DEST].s3.amazonaws.com
x-amz-content-sha256:[...]
x-amz-date:20180509T001513Z

host;x-amz-content-sha256;x-amz-date
[...]
----------------------
DEBUG: signature-v4 headers: {'x-amz-content-sha256': u'[...]', 'Authorization': u'AWS4-HMAC-SHA256 Credential=[...]/20180509/us-east-2/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=[...]', 'x-amz-date': '[...]'}
DEBUG: Processing request, please wait...
DEBUG: get_hostname([DEST]): [DEST].s3.amazonaws.com
DEBUG: ConnMan.get(): creating new connection: http://[DEST].s3.amazonaws.com
DEBUG: non-proxied HTTPConnection([DEST].s3.amazonaws.com, None)
DEBUG: format_uri(): /?prefix=test
DEBUG: Sending request method_string='GET', uri=u'/?prefix=test', headers={'x-amz-content-sha256': u'[...]', 'Authorization': u'AWS4-HMAC-SHA256 Credential=[...]/20180509/us-east-2/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=[...]', 'x-amz-date': '[...]'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (http://[DEST].s3.amazonaws.com#1)
DEBUG: Response:
{'data': '<?xml version="1.0" encoding="UTF-8"?>\n<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>[DEST]</Name><Prefix>test</Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated></ListBucketResult>',
 'headers': {'content-type': 'application/xml',
             'date': 'Wed, 09 May 2018 00:15:14 GMT',
             'server': 'AmazonS3',
             'transfer-encoding': 'chunked',
             'x-amz-bucket-region': 'us-east-2',
             'x-amz-id-2': '[...]',
             'x-amz-request-id': '[...]'},
 'reason': 'OK',
 'status': 200}
DEBUG: Applying --exclude/--include
INFO: Found 2 source files, 0 destination files
INFO: Verifying attributes...
DEBUG: Comparing filelists (direction: remote -> remote)
DEBUG: CHECK: test/
DEBUG: CHECK: test/test l’éééééààààà.txt
Summary: 2 source files to copy, 0 files at destination to delete
DEBUG: String 'test/' encoded to 'test/'
DEBUG: CreateRequest: resource[uri]=/test/
DEBUG: Using signature v4
DEBUG: get_hostname([DEST]): [DEST].s3.amazonaws.com
DEBUG: canonical_headers = host:[DEST].s3.amazonaws.com
x-amz-acl:public-read
x-amz-content-sha256:[...]
x-amz-copy-source:/[SRC]/test/
x-amz-date:20180509T001513Z
x-amz-metadata-directive:COPY
x-amz-storage-class:STANDARD

DEBUG: Canonical Request:
PUT
/test/

host:[DEST].s3.amazonaws.com
x-amz-acl:public-read
x-amz-content-sha256:[...]
x-amz-copy-source:/[SRC]/test/
x-amz-date:20180509T001513Z
x-amz-metadata-directive:COPY
x-amz-storage-class:STANDARD

host;x-amz-acl;x-amz-content-sha256;x-amz-copy-source;x-amz-date;x-amz-metadata-directive;x-amz-storage-class
[...]
----------------------
DEBUG: signature-v4 headers: {'x-amz-content-sha256': u'[...]', 'x-amz-acl': 'public-read', 'x-amz-storage-class': 'STANDARD', 'x-amz-date': '20180509T001513Z', 'x-amz-copy-source': u'/[SRC]/test/', 'Authorization': u'AWS4-HMAC-SHA256 Credential=[...]/20180509/us-east-2/s3/aws4_request,SignedHeaders=host;x-amz-acl;x-amz-content-sha256;x-amz-copy-source;x-amz-date;x-amz-metadata-directive;x-amz-storage-class,Signature=[...]', 'x-amz-metadata-directive': 'COPY'}
DEBUG: Processing request, please wait...
DEBUG: get_hostname([DEST]): [DEST].s3.amazonaws.com
DEBUG: ConnMan.get(): re-using connection: http://[DEST].s3.amazonaws.com#1
DEBUG: format_uri(): /test/
DEBUG: Sending request method_string='PUT', uri=u'/test/', headers={'x-amz-content-sha256': u'[...]', 'x-amz-acl': 'public-read', 'x-amz-storage-class': 'STANDARD', 'x-amz-date': '20180509T001513Z', 'x-amz-copy-source': u'/[SRC]/test/', 'Authorization': u'AWS4-HMAC-SHA256 Credential=[...]/20180509/us-east-2/s3/aws4_request,SignedHeaders=host;x-amz-acl;x-amz-content-sha256;x-amz-copy-source;x-amz-date;x-amz-metadata-directive;x-amz-storage-class,Signature=[...]', 'x-amz-metadata-directive': 'COPY'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (http://[DEST].s3.amazonaws.com#2)
DEBUG: Response:
{'data': '<?xml version="1.0" encoding="UTF-8"?>\n<CopyObjectResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><LastModified>2018-05-09T00:15:14.000Z</LastModified><ETag>&quot;d41d8cd98f00b204e9800998ecf8427e&quot;</ETag></CopyObjectResult>',
 'headers': {'content-length': '234',
             'content-type': 'application/xml',
             'date': 'Wed, 09 May 2018 00:15:14 GMT',
             'server': 'AmazonS3',
             'x-amz-id-2': 'zCwH37aiFAJ1uLh3rjlj3Q4sNIcK8jY8TLw4S00l2WHfypk5sxd7C/KgHe1AaLCaTpIEK5uiczI=',
             'x-amz-request-id': '3D886AEAEF0ACACF'},
 'reason': 'OK',
 'status': 200}
DEBUG: DeUnicodising u's3://[SRC]/test/' using UTF-8
DEBUG: DeUnicodising u's3://[DEST]/test/' using UTF-8
remote copy: 's3://[SRC]/test/' -> 's3://[DEST]/test/'
remote copy: 'xxx' -> 'xxx'
DEBUG: String 'test/test l’éééééààààà.txt' encoded to 'test/test%20l%E2%80%99e%CC%81e%CC%81e%CC%81e%CC%81e%CC%81a%CC%80a%CC%80a%CC%80a%CC%80a%CC%80.txt'
DEBUG: CreateRequest: resource[uri]=/test/test l’éééééààààà.txt
DEBUG: Using signature v4
DEBUG: get_hostname([DEST]): [DEST].s3.amazonaws.com
DEBUG: canonical_headers = host:[DEST].s3.amazonaws.com
x-amz-acl:public-read
x-amz-content-sha256:[...]
x-amz-copy-source:/[SRC]/test/test%20l%E2%80%99e%CC%81e%CC%81e%CC%81e%CC%81e%CC%81a%CC%80a%CC%80a%CC%80a%CC%80a%CC%80.txt
x-amz-date:20180509T001513Z
x-amz-metadata-directive:COPY
x-amz-storage-class:STANDARD

DEBUG: Canonical Request:
PUT
/test/test%20l%E2%80%99e%CC%81e%CC%81e%CC%81e%CC%81e%CC%81a%CC%80a%CC%80a%CC%80a%CC%80a%CC%80.txt

host:[DEST].s3.amazonaws.com
x-amz-acl:public-read
x-amz-content-sha256:[...]
x-amz-copy-source:/[SRC]/test/test%20l%E2%80%99e%CC%81e%CC%81e%CC%81e%CC%81e%CC%81a%CC%80a%CC%80a%CC%80a%CC%80a%CC%80.txt
x-amz-date:20180509T001513Z
x-amz-metadata-directive:COPY
x-amz-storage-class:STANDARD

host;x-amz-acl;x-amz-content-sha256;x-amz-copy-source;x-amz-date;x-amz-metadata-directive;x-amz-storage-class
[...]
----------------------
DEBUG: signature-v4 headers: {'x-amz-content-sha256': u'[...]', 'x-amz-acl': 'public-read', 'x-amz-storage-class': 'STANDARD', 'x-amz-date': '20180509T001513Z', 'x-amz-copy-source': u'/[SRC]/test/test%20l%E2%80%99e%CC%81e%CC%81e%CC%81e%CC%81e%CC%81a%CC%80a%CC%80a%CC%80a%CC%80a%CC%80.txt', 'Authorization': u'AWS4-HMAC-SHA256 Credential=[...]/20180509/us-east-2/s3/aws4_request,SignedHeaders=host;x-amz-acl;x-amz-content-sha256;x-amz-copy-source;x-amz-date;x-amz-metadata-directive;x-amz-storage-class,Signature=[...]', 'x-amz-metadata-directive': 'COPY'}
DEBUG: Processing request, please wait...
DEBUG: get_hostname([DEST]): [DEST].s3.amazonaws.com
DEBUG: ConnMan.get(): re-using connection: http://[DEST].s3.amazonaws.com#2
DEBUG: format_uri(): /test/test%20l%E2%80%99e%CC%81e%CC%81e%CC%81e%CC%81e%CC%81a%CC%80a%CC%80a%CC%80a%CC%80a%CC%80.txt
DEBUG: Sending request method_string='PUT', uri=u'/test/test%20l%E2%80%99e%CC%81e%CC%81e%CC%81e%CC%81e%CC%81a%CC%80a%CC%80a%CC%80a%CC%80a%CC%80.txt', headers={'x-amz-content-sha256': u'[...]', 'x-amz-acl': 'public-read', 'x-amz-storage-class': 'STANDARD', 'x-amz-date': '20180509T001513Z', 'x-amz-copy-source': u'/[SRC]/test/test%20l%E2%80%99e%CC%81e%CC%81e%CC%81e%CC%81e%CC%81a%CC%80a%CC%80a%CC%80a%CC%80a%CC%80.txt', 'Authorization': u'AWS4-HMAC-SHA256 Credential=[...]/20180509/us-east-2/s3/aws4_request,SignedHeaders=host;x-amz-acl;x-amz-content-sha256;x-amz-copy-source;x-amz-date;x-amz-metadata-directive;x-amz-storage-class,Signature=[...]', 'x-amz-metadata-directive': 'COPY'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (http://[DEST].s3.amazonaws.com#3)
DEBUG: Response:
{'data': '<?xml version="1.0" encoding="UTF-8"?>\n<CopyObjectResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><LastModified>2018-05-09T00:15:14.000Z</LastModified><ETag>&quot;d8e8fca2dc0f896fd7cb4cb0031ba249&quot;</ETag></CopyObjectResult>',
 'headers': {'content-length': '234',
             'content-type': 'application/xml',
             'date': 'Wed, 09 May 2018 00:15:14 GMT',
             'server': 'AmazonS3',
             'x-amz-id-2': '[...]',
             'x-amz-request-id': '[...]'},
 'reason': 'OK',
 'status': 200}
DEBUG: DeUnicodising u's3://[SRC]/test/test l\u2019e\u0301e\u0301e\u0301e\u0301e\u0301a\u0300a\u0300a\u0300a\u0300a\u0300.txt' using UTF-8
DEBUG: DeUnicodising u's3://[DEST]/test/test l\u2019e\u0301e\u0301e\u0301e\u0301e\u0301a\u0300a\u0300a\u0300a\u0300a\u0300.txt' using UTF-8

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
  Please try reproducing the error using
  the latest s3cmd code from the git master
  branch found at:
    https://github.com/s3tools/s3cmd
  and have a look at the known issues list:
    https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions
  If the error persists, please report the
  following lines (removing any private
  info as necessary) to:
   [email protected]


!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

DEBUG: Unicodising '/usr/local/Cellar/s3cmd/2.0.1/libexec/bin/s3cmd' using UTF-8
DEBUG: Unicodising 'sync' using UTF-8
DEBUG: Unicodising 's3://[SRC]/test' using UTF-8
DEBUG: Unicodising 's3://[DEST]/' using UTF-8
DEBUG: Unicodising '--access_key=[...]' using UTF-8
DEBUG: Unicodising '--secret_key=[...]' using UTF-8
DEBUG: Unicodising '--acl-public' using UTF-8
DEBUG: Unicodising '-d' using UTF-8
Invoked as: /usr/local/Cellar/s3cmd/2.0.1/libexec/bin/s3cmd sync s3://[SRC]/test s3://[DEST]/ --access_key=[...] --secret_key=[...] --acl-public -d
Problem: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 58: ordinal not in range(128)
S3cmd:   2.0.1
python:   2.7.10 (default, Oct  6 2017, 22:29:07)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.31)]
environment LANG=fr_CA.UTF-8

Traceback (most recent call last):
  File "/usr/local/Cellar/s3cmd/2.0.1/libexec/bin/s3cmd", line 3074, in <module>
    rc = main()
  File "/usr/local/Cellar/s3cmd/2.0.1/libexec/bin/s3cmd", line 2990, in main
    rc = cmd_func(args)
  File "/usr/local/Cellar/s3cmd/2.0.1/libexec/bin/s3cmd", line 1878, in cmd_sync
    return cmd_sync_remote2remote(args)
  File "/usr/local/Cellar/s3cmd/2.0.1/libexec/bin/s3cmd", line 1112, in cmd_sync_remote2remote
    ret, seq, nb_files, size = _upload(src_list, seq, src_count + update_count)
  File "/usr/local/Cellar/s3cmd/2.0.1/libexec/bin/s3cmd", line 1098, in _upload
    output("remote copy: '%(src)s' -> '%(dst)s'" % { "src" : src_uri, "dst" : dst_uri })
  File "/usr/local/Cellar/s3cmd/2.0.1/libexec/bin/s3cmd", line 66, in output
    sys.stdout.write(message + "\n")
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/codecs.py", line 357, in write
    data, consumed = self.encode(object, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 58: ordinal not in range(128)

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
  Please try reproducing the error using
  the latest s3cmd code from the git master
  branch found at:
    https://github.com/s3tools/s3cmd
  and have a look at the known issues list:
    https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions
  If the error persists, please report the
  above lines (removing any private
  info as necessary) to:
   [email protected]
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

This patch allows cmd_info to work properly in case of getting any unicode chars in reply to bucket Policy/CORS requests. Fixes at least s3tools#847

offshore · 2018-12-06T12:31:52Z

I've submitted a patch which should fix this.

vlada-dudr · 2019-04-02T12:53:41Z

The issue is somehow still around...

Invoked as: s3cmd-master/s3cmd sync -P s3://bucket1/ s3://bucket2/somedir/                                                                                                                                               [9/1981]
Problem: UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 72: ordinal not in range(128)
S3cmd:   2.0.2                                          
python:   2.7.13 (default, Sep 26 2018, 18:42:22)                                
[GCC 6.3.0 20170516]                      
environment LANG=en_US.UTF-8       
                        
Traceback (most recent call last):   
  File "s3cmd-master/s3cmd", line 3104, in <module>                   
    rc = main()                                                                            
  File "s3cmd-master/s3cmd", line 3013, in main                       
    rc = cmd_func(args)
  File "s3cmd-master/s3cmd", line 1901, in cmd_sync
    return cmd_sync_remote2remote(args)
  File "s3cmd-master/s3cmd", line 1128, in cmd_sync_remote2remote
    ret, seq, nb_files, size = _upload(src_list, seq, src_count + update_count)
  File "s3cmd-master/s3cmd", line 1115, in _upload
    output("remote copy: '%(src)s' -> '%(dst)s'" % { "src" : src_uri, "dst" : dst_uri })
  File "s3cmd-master/s3cmd", line 71, in output
    sys.stdout.write(message + "\n")
  File "/usr/lib/python2.7/codecs.py", line 369, in write
    data, consumed = self.encode(object, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 72: ordinal not in range(128)
`

fviard · 2020-03-29T01:47:26Z

I think that @offshore fixed the original poster error,
and that @jdurand and @vlada-dudr had a different issue that I just pushed a fix for.
Thank you all for your reports and your help.

fviard added the waiting-feedback label Jun 25, 2017

offshore added a commit to offshore/s3cmd that referenced this issue Dec 6, 2018

Return decoded response in get_policy and get_cors

3048fdc

This patch allows cmd_info to work properly in case of getting any unicode chars in reply to bucket Policy/CORS requests. Fixes at least s3tools#847

offshore mentioned this issue Dec 6, 2018

Return decoded response in get_policy and get_cors #1021

Merged

fviard added to-be-fixed and removed waiting-feedback labels Mar 29, 2020

fviard closed this as completed in a3dcb51 Mar 29, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UnicodeDecodeError #847

UnicodeDecodeError #847

baiyanchun commented Mar 22, 2017

baiyanchun commented Mar 22, 2017

mdomsch commented Mar 23, 2017 via email

fviard commented Apr 2, 2017

fviard commented Jun 25, 2017

jdurand commented May 8, 2018

jdurand commented May 8, 2018

fviard commented May 8, 2018

jdurand commented May 8, 2018

fviard commented May 8, 2018

jdurand commented May 8, 2018

fviard commented May 8, 2018

jdurand commented May 9, 2018

offshore commented Dec 6, 2018

vlada-dudr commented Apr 2, 2019

fviard commented Mar 29, 2020

UnicodeDecodeError #847

UnicodeDecodeError #847

Comments

baiyanchun commented Mar 22, 2017

baiyanchun commented Mar 22, 2017

mdomsch commented Mar 23, 2017 via email

fviard commented Apr 2, 2017

fviard commented Jun 25, 2017

jdurand commented May 8, 2018

jdurand commented May 8, 2018

fviard commented May 8, 2018

jdurand commented May 8, 2018

fviard commented May 8, 2018

jdurand commented May 8, 2018

fviard commented May 8, 2018

jdurand commented May 9, 2018

offshore commented Dec 6, 2018

vlada-dudr commented Apr 2, 2019

fviard commented Mar 29, 2020