TypeError: Unicode-objects must be encoded before hashing in s3cmd sync --cf-invalidate

[benkuhn]

Thanks for making s3cmd, it's super useful!

I recently tried to use it with python3 and got the following traceback when using the --cf-invalidate flag:

Traceback (most recent call last):
  File "/Users/ben/.virtualenvs/bknet/bin/s3cmd", line 3092, in <module>
    rc = main()
  File "/Users/ben/.virtualenvs/bknet/bin/s3cmd", line 3001, in main
    rc = cmd_func(args)
  File "/Users/ben/.virtualenvs/bknet/bin/s3cmd", line 1885, in cmd_sync
    return cmd_sync_local2remote(args)
  File "/Users/ben/.virtualenvs/bknet/bin/s3cmd", line 1864, in cmd_sync_local2remote
    _invalidate_on_cf(destination_base_uri)
  File "/Users/ben/.virtualenvs/bknet/bin/s3cmd", line 1834, in _invalidate_on_cf
    results = cf.InvalidateObjects(destination_base_uri, uploaded_objects_list, default_index_file, cfg.invalidate_default_index_on_cf, cfg.invalidate_default_index_root_on_cf)
  File "/Users/ben/.virtualenvs/bknet/lib/python3.5/site-packages/S3/CloudFront.py", line 452, in InvalidateObjects
    cfuris = self.get_dist_name_for_bucket(uri)
  File "/Users/ben/.virtualenvs/bknet/lib/python3.5/site-packages/S3/CloudFront.py", line 587, in get_dist_name_for_bucket
    response = self.GetList()
  File "/Users/ben/.virtualenvs/bknet/lib/python3.5/site-packages/S3/CloudFront.py", line 323, in GetList
    response = self.send_request("GetList")
  File "/Users/ben/.virtualenvs/bknet/lib/python3.5/site-packages/S3/CloudFront.py", line 506, in send_request
    request = self.create_request(operation, dist_id, request_id, headers)
  File "/Users/ben/.virtualenvs/bknet/lib/python3.5/site-packages/S3/CloudFront.py", line 555, in create_request
    signature = self.sign_request(headers)
  File "/Users/ben/.virtualenvs/bknet/lib/python3.5/site-packages/S3/CloudFront.py", line 567, in sign_request
    signature = sign_string_v2(string_to_sign)
  File "/Users/ben/.virtualenvs/bknet/lib/python3.5/site-packages/S3/Crypto.py", line 68, in sign_string_v2
    signature = base64.encodestring(hmac.new(encode_to_s3(secret_key), string_to_sign, sha1).digest()).strip()
  File "/Users/ben/.virtualenvs/bknet/lib/python3.5/hmac.py", line 144, in new
    return HMAC(key, msg, digestmod)
  File "/Users/ben/.virtualenvs/bknet/lib/python3.5/hmac.py", line 84, in __init__
    self.update(msg)
  File "/Users/ben/.virtualenvs/bknet/lib/python3.5/hmac.py", line 93, in update
    self.inner.update(msg)
TypeError: Unicode-objects must be encoded before hashing

Tried on both python3.5.1 and python3.7.0.

[rgant]

Having the same issue. I tried encoding string_to_sign in Crypto.sign_string_v2 but then many other bytes vs strings errors appeared.

Problem: <class 'TypeError: can only concatenate str (not "bytes") to str
S3cmd:   2.0.2
python:   3.7.0 (default, Jul 23 2018, 20:22:55) 
[Clang 9.1.0 (clang-902.0.39.2)]
environment LANG=en_US.UTF-8

Traceback (most recent call last):
  File "/usr/local/Cellar/s3cmd/2.0.2_1/libexec/bin/s3cmd", line 3092, in <module>
    rc = main()
  File "/usr/local/Cellar/s3cmd/2.0.2_1/libexec/bin/s3cmd", line 3001, in main
    rc = cmd_func(args)
  File "/usr/local/Cellar/s3cmd/2.0.2_1/libexec/bin/s3cmd", line 1885, in cmd_sync
    return cmd_sync_local2remote(args)
  File "/usr/local/Cellar/s3cmd/2.0.2_1/libexec/bin/s3cmd", line 1864, in cmd_sync_local2remote
    _invalidate_on_cf(destination_base_uri)
  File "/usr/local/Cellar/s3cmd/2.0.2_1/libexec/bin/s3cmd", line 1834, in _invalidate_on_cf
    results = cf.InvalidateObjects(destination_base_uri, uploaded_objects_list, default_index_file, cfg.invalidate_default_index_on_cf, cfg.invalidate_default_index_root_on_cf)
  File "/usr/local/Cellar/s3cmd/2.0.2_1/libexec/lib/python3.7/site-packages/S3/CloudFront.py", line 452, in InvalidateObjects
    cfuris = self.get_dist_name_for_bucket(uri)
  File "/usr/local/Cellar/s3cmd/2.0.2_1/libexec/lib/python3.7/site-packages/S3/CloudFront.py", line 587, in get_dist_name_for_bucket
    response = self.GetList()
  File "/usr/local/Cellar/s3cmd/2.0.2_1/libexec/lib/python3.7/site-packages/S3/CloudFront.py", line 323, in GetList
    response = self.send_request("GetList")
  File "/usr/local/Cellar/s3cmd/2.0.2_1/libexec/lib/python3.7/site-packages/S3/CloudFront.py", line 506, in send_request
    request = self.create_request(operation, dist_id, request_id, headers)
  File "/usr/local/Cellar/s3cmd/2.0.2_1/libexec/lib/python3.7/site-packages/S3/CloudFront.py", line 556, in create_request
    headers["Authorization"] = "AWS "+self.config.access_key+":"+signature
TypeError: can only concatenate str (not "bytes") to str

[blimmer]

I also just experienced this problem.

[fviard]

@benkuhn @rgant @blimmer I have pushed some fixes that should solve your issues with cloudfront and py3 issues.
It would be great if you could give them a try to confirm that your issues are indeed solved. Thanks.

[dcaravana]

I can confirm that with the current version (2.0.2) the problem is there, while with the master branch version it's been fixed.

[rgant]

It's going to take me a bit to figure out how to setup the s3cmd development environment again. But reading through the changes it appears like it should work.

[blimmer]

I got a new error when running this from master

Invoked as: /tmp/s3cmd/s3cmd --cf-invalidate -c /some-path-to-my-config -r sync _site/ s3://mybucket -v
Problem: AttributeError: 'NoneType' object has no attribute 'getiterator'
S3cmd:   2.0.2
python:   2.7.15 (default, Feb 12 2019, 11:00:12)
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)]
environment LANG=en_US.UTF-8

Traceback (most recent call last):
  File "/tmp/s3cmd/s3cmd", line 3104, in <module>
    rc = main()
  File "/tmp/s3cmd/s3cmd", line 3013, in main
    rc = cmd_func(args)
  File "/tmp/s3cmd/s3cmd", line 1897, in cmd_sync
    return cmd_sync_local2remote(args)
  File "/tmp/s3cmd/s3cmd", line 1876, in cmd_sync_local2remote
    _invalidate_on_cf(destination_base_uri)
  File "/tmp/s3cmd/s3cmd", line 1846, in _invalidate_on_cf
    results = cf.InvalidateObjects(destination_base_uri, uploaded_objects_list, default_index_file, cfg.invalidate_default_index_on_cf, cfg.invalidate_default_index_root_on_cf)
  File "/private/tmp/s3cmd/S3/CloudFront.py", line 492, in InvalidateObjects
    debug("InvalidateObjects(): request_body: %s" % invalbatch)
  File "/private/tmp/s3cmd/S3/CloudFront.py", line 321, in __str__
    return ET.tostring(self.get_printable_tree())
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1126, in tostring
    ElementTree(element).write(file, encoding, method=method)
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 817, in write
    self._root, encoding, default_namespace
  File "/usr/local/Cellar/python@2/2.7.15_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 876, in _namespaces
    iterate = elem.getiterator # cET compatibility
AttributeError: 'NoneType' object has no attribute 'getiterator'

[fviard]

@blimmer thank you very much for your test. Shame on me, there was a very little bit of code that was forgotten from the previous commit.
It is pushed and it should now really be fixed in master. (Crossing fingers :-) )

[blimmer]

No shame! It worked great this time. Thank you 💯

[sdanbury]

Any idea when this will make it into PyPI please?

[Mickael-van-der-Beek]

Please deploy this fix. Currently s3cmd is completely broken and unusable due to it.

[sam0x17]

This is still broken on fresh installs of OSX using brew to install s3cmd

[sam0x17]

here is a drop-in docker-based workaround I set up in the mean time: https://github.com/sam0x17/s3cmd-shim

[sam0x17]

also broken installing directly from pip

[bobhacks]

Invoked as: /usr/bin/s3cmd ls
Problem: <class 'TypeError: can only concatenate str (not "bytes") to str
S3cmd: 2.0.2
python: 3.7.5 (default, Nov 20 2019, 09:21:52)
[GCC 9.2.1 20191008]
environment LANG=C.UTF-8

Traceback (most recent call last):
File "/usr/bin/s3cmd", line 3092, in
rc = main()
File "/usr/bin/s3cmd", line 3001, in main
rc = cmd_func(args)
File "/usr/bin/s3cmd", line 149, in cmd_ls
subcmd_all_buckets_list(s3)
File "/usr/bin/s3cmd", line 154, in subcmd_all_buckets_list
response = s3.list_all_buckets()
File "/usr/lib/python3/dist-packages/S3/S3.py", line 302, in list_all_buckets
response = self.send_request(request)
File "/usr/lib/python3/dist-packages/S3/S3.py", line 1252, in send_request
method_string, resource, headers = request.get_triplet()
File "/usr/lib/python3/dist-packages/S3/S3.py", line 201, in get_triplet
self.sign()
File "/usr/lib/python3/dist-packages/S3/S3.py", line 176, in sign
self.headers = sign_request_v2(self.method_string, resource_uri, self.params, self.headers)
File "/usr/lib/python3/dist-packages/S3/Crypto.py", line 101, in sign_request_v2
string_to_sign += header + ":" + cur_headers[header] + "\n"
TypeError: can only concatenate str (not "bytes") to str