New release is broken, had to revert

[paulcdejean]

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
TypeError: the JSON object must be str, not 'bytes'
    s.__class__.__name__))
  File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/usr/lib/python3.5/json/__init__.py", line 268, in load
    creds=json.load(resp)
  File "/usr/local/lib/python3.5/dist-packages/S3/Config.py", line 266, in role_config
    self.role_config()
  File "/usr/local/lib/python3.5/dist-packages/S3/Config.py", line 245, in __init__
    cfg = Config(options.config, options.access_key, options.secret_key, options.access_token)
  File "/usr/local/bin/s3cmd", line 2792, in main
    rc = main()
  File "/usr/local/bin/s3cmd", line 3092, in <module>
Traceback (most recent call last):
environment LANG=en_US.UTF-8
[GCC 5.4.0 20160609]
python:   3.5.2 (default, Nov 23 2017, 16:37:01)
S3cmd:   2.0.2
Problem: <class 'TypeError: the JSON object must be str, not 'bytes'
Invoked as: /usr/local/bin/s3cmd sync --quiet -F --delete-removed s3://dashboardcopy/master/ /sites/
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

2.0.1 works fine so I reverted to that thanks.

[fviard]

Thank you very much for the notice.
Are you using IAM Credentials?

[paulcdejean]

Oh yeah that was the issue. But it crashes instead of giving an informative error like the last version.

I was trying to use the server's IAM role, but s3cmd doesn't get it automatically so I need to query server metadata and parse it myself, working on setting that up now.

[fviard]

Yes, it is a recent change that was probably not safe enough for encoding issues.
But before it was not working, displaying an error?
Can you indicate me what error it was?

[paulcdejean]

The error before was:

ERROR: /home/admin/.s3cfg: None
ERROR: Configuration file not available.
ERROR: Consider using --configure parameter to create one.

It seems in the new version you're trying to automagically get the role credentials. However it doesn't seem to be working. Maybe you only tested on python2?

[paulcdejean]

Confirmed. It automatically works with a role in python2, but crashes in python3.

[fviard]

Ok, because normally you should still create your configuration with --configure or manually creating a .s3cfg.
Anyway, iam looks like broken by the last change, i will look at that.

[fviard]

I have found another related issue that I have fixed, but I was testing with python 3.6 and trying with python 3.5 as you shows that there is another issue. :-s
So still looking at it.

[fviard]

Your issue should now be fixed.
I will try to create a new release tomorrow with these fixes.

[marcinkowalczyk]

hi,

looks it's still broken IAM Role

Problem: <class 'TypeError: must be str, not bytes
S3cmd:   2.0.2
python:   3.6.7 (default, Oct 22 2018, 11:32:17)
[GCC 8.2.0]
environment LANG=C.UTF-8

Traceback (most recent call last):
  File "/usr/local/bin/s3cmd", line 3092, in <module>
    rc = main()
  File "/usr/local/bin/s3cmd", line 3001, in main
    rc = cmd_func(args)
  File "/usr/local/bin/s3cmd", line 503, in cmd_object_get
    remote_list, exclude_list, remote_total_size = fetch_remote_list(args, require_attribs = False)
  File "/usr/local/lib/python3.6/dist-packages/S3/FileLists.py", line 476, in fetch_remote_list
    objectlist, tmp_total_size = _get_filelist_remote(S3Uri(prefix), recursive = need_recursion)
  File "/usr/local/lib/python3.6/dist-packages/S3/FileLists.py", line 397, in _get_filelist_remote
    recursive = recursive, uri_params = uri_params)
  File "/usr/local/lib/python3.6/dist-packages/S3/S3.py", line 309, in bucket_list
    for truncated, dirs, objects in self.bucket_list_streaming(bucket, prefix, recursive, uri_params, limit):
  File "/usr/local/lib/python3.6/dist-packages/S3/S3.py", line 343, in bucket_list_streaming
    response = self.bucket_list_noparse(bucket, prefix, recursive, uri_params, max_keys)
  File "/usr/local/lib/python3.6/dist-packages/S3/S3.py", line 374, in bucket_list_noparse
    response = self.send_request(request)
  File "/usr/local/lib/python3.6/dist-packages/S3/S3.py", line 1252, in send_request
    method_string, resource, headers = request.get_triplet()
  File "/usr/local/lib/python3.6/dist-packages/S3/S3.py", line 201, in get_triplet
    self.sign()
  File "/usr/local/lib/python3.6/dist-packages/S3/S3.py", line 197, in sign
    bucket_region, self.headers, self.body)
  File "/usr/local/lib/python3.6/dist-packages/S3/Crypto.py", line 196, in sign_request_v4
    signing_key = getSignatureKey(secret_key, datestamp, region, service)
  File "/usr/local/lib/python3.6/dist-packages/S3/Crypto.py", line 176, in getSignatureKey
    kDate = sign(encode_to_s3('AWS4' + key), dateStamp)
TypeError: must be str, not bytes

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
  Please try reproducing the error using
  the latest s3cmd code from the git master
  branch found at:
    https://github.com/s3tools/s3cmd
  and have a look at the known issues list:
    https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions
  If the error persists, please report the
  above lines (removing any private
  info as necessary) to:
   [email protected]
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

[serkanh]

Seeing the exact same issue when running it with AWS EC2 roles.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Invoked as: /usr/local/bin/s3cmd ls
Problem: <class 'TypeError: must be str, not bytes
S3cmd:   2.0.2
python:   3.6.8 (default, Mar 27 2019, 08:49:59) 
[GCC 6.3.0 20170516]
environment LANG=C.UTF-8

Traceback (most recent call last):
  File "/usr/local/bin/s3cmd", line 3092, in <module>
    rc = main()
  File "/usr/local/bin/s3cmd", line 3001, in main
    rc = cmd_func(args)
  File "/usr/local/bin/s3cmd", line 149, in cmd_ls
    subcmd_all_buckets_list(s3)
  File "/usr/local/bin/s3cmd", line 154, in subcmd_all_buckets_list
    response = s3.list_all_buckets()
  File "/usr/local/lib/python3.6/site-packages/S3/S3.py", line 302, in list_all_buckets
    response = self.send_request(request)
  File "/usr/local/lib/python3.6/site-packages/S3/S3.py", line 1252, in send_request
    method_string, resource, headers = request.get_triplet()
  File "/usr/local/lib/python3.6/site-packages/S3/S3.py", line 201, in get_triplet
    self.sign()
  File "/usr/local/lib/python3.6/site-packages/S3/S3.py", line 176, in sign
    self.headers = sign_request_v2(self.method_string, resource_uri, self.params, self.headers)
  File "/usr/local/lib/python3.6/site-packages/S3/Crypto.py", line 101, in sign_request_v2
    string_to_sign += header + ":" + cur_headers[header] + "\n"
TypeError: must be str, not bytes

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
  Please try reproducing the error using
  the latest s3cmd code from the git master
  branch found at:
    https://github.com/s3tools/s3cmd
  and have a look at the known issues list:
    https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions
  If the error persists, please report the
  above lines (removing any private
  info as necessary) to:
   [email protected]
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Including the Dockerfile to give more context as well.

FROM python:3.6

RUN pip install awscli --upgrade
RUN pip install s3cmd
RUN apt-get update && apt-get install -y postgresql postgresql-contrib

RUN mkdir /pipeline
RUN mkdir /work

ADD . /pipeline
WORKDIR /pipeline

ENTRYPOINT [ "/pipeline/pipeline.sh" ]

[serkanh]

update: solved by basically cloning and building it on the image. Looks like fixed merged only 3 days after the last release.

FROM python:3.6

RUN pip install awscli --upgrade
RUN apt-get update && apt-get install -y git
**RUN git clone https://github.com/s3tools/s3cmd.git
RUN cd s3cmd && pip install .**
RUN apt-get update && apt-get install -y postgresql postgresql-contrib

RUN mkdir /pipeline
RUN mkdir /work

ADD . /pipeline
WORKDIR /pipeline

ENTRYPOINT [ "/pipeline/pipeline.sh" ]